Skip to content

Querying Records with Paging

When working with large datasets in Mythradon via the REST API, it's important to implement paging to retrieve records in manageable batches. Paging prevents overwhelming the server and ensures efficient use of network resources and client memory.

The Mythradon API supports paging through the offset and maxSize query parameters on GET /{entityType} requests:

  • maxSize controls how many records are returned per page.
  • offset determines the starting position of records to return.
  • The response includes a total property that indicates the total number of matching records across all pages.

By incrementing the offset and continuing until the total number of records has been retrieved, you can page through all records for a given entity type.

Below are examples in PHP, JavaScript (Node.js), and Python demonstrating how to implement a full paging loop using the total value from the response to determine when to stop.

Examples

PHP Example – Paging Using total

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
<?php

$apiUrl = 'https://<your-Mythradon-domain>/api/v1/Account';
$apiKey = '<Your API Key>';

$maxSize = 100;
$offset = 0;
$allRecords = [];

do {
    $query = http_build_query([
        'offset' => $offset,
        'maxSize' => $maxSize
    ]);

    $url = "$apiUrl?$query";

    $opts = [
        'http' => [
            'method' => 'GET',
            'header' => [
                "X-Api-Key: $apiKey",
                "Content-Type: application/json"
            ]
        ]
    ];

    $context = stream_context_create($opts);
    $response = file_get_contents($url, false, $context);

    if ($response === false) {
        die("API request failed.\n");
    }

    $data = json_decode($response, true);

    $allRecords = array_merge($allRecords, $data['list']);
    $offset += $maxSize;

    echo "Fetched " . count($allRecords) . " of " . $data['total'] . " records...\n";

} while ($offset < $data['total']);

echo "Complete! Total records retrieved: " . count($allRecords) . "\n";

JavaScipt Example – Paging Using total

const fetch = require('node-fetch');

const apiUrl = 'https://<your-Mythradon-domain>/api/v1/Account';
const apiKey = '<Your API Key>';
const maxSize = 100;

let offset = 0;
let allRecords = [];

async function fetchAllRecords() {
    let total = Infinity;

    while (offset < total) {
        const url = `${apiUrl}?offset=${offset}&maxSize=${maxSize}`;

        const response = await fetch(url, {
            method: 'GET',
            headers: {
                'X-Api-Key': apiKey,
                'Content-Type': 'application/json'
            }
        });

        const data = await response.json();

        allRecords = allRecords.concat(data.list);
        total = data.total;
        offset += maxSize;

        console.log(`Fetched ${allRecords.length} of ${total} records...`);
    }

    console.log(`Complete! Total records retrieved: ${allRecords.length}`);
}

fetchAllRecords().catch(console.error);

Python Example – Paging Using total

import requests

api_url = 'https://<your-Mythradon-domain>/api/v1/Account'
api_key = '<Your API Key>'

max_size = 100
offset = 0
all_records = []

headers = {
    'X-Api-Key': api_key,
    'Content-Type': 'application/json'
}

while True:
    params = {
        'offset': offset,
        'maxSize': max_size
    }

    response = requests.get(api_url, headers=headers, params=params)
    response.raise_for_status()
    data = response.json()

    all_records.extend(data['list'])
    total = data['total']
    offset += max_size

    print(f"Fetched {len(all_records)} of {total} records...")

    if offset >= total:
        break

print(f"Complete! Total records retrieved: {len(all_records)}")

🔍 Notes

  • The maxSize should be ≤ recordListMaxSizeLimit (usually 200 or 1000).
  • If the X-No-Total: true header is used, the total field will be missing and this logic won't work — you'd then need to stop when the response list is empty.

See also