Stream Records From a Large XML File With xmltodict

A full tree is convenient for small XML files. For a large feed with independent records, xmltodict can hand each finished record to a callback that writes it out immediately.

Worked example

A small in-memory feed for the streaming callback

from io import BytesIO
import json
import xmltodict

source = BytesIO(
    b"<catalog><item><sku>003</sku></item>"
    b"<item><sku>014</sku></item></catalog>"
)

def write_item(path, item):
    print(json.dumps(item, ensure_ascii=False))
    return True

xmltodict.parse(source, item_depth=2, item_callback=write_item)

Steps

  1. Install xmltodict in a Python environment and save the example as stream.py. The synthetic feed has catalog at depth one and each item at depth two.
  2. Run python3 stream.py > items.ndjson. Confirm that there are two lines and that each line parses as one complete JSON object.
  3. For a trusted file of the same shape, replace the BytesIO source with a file opened in binary mode. Keep the callback streaming to its destination instead of collecting records into a list.

What to check

  • The sample writes {"sku": "003"} and {"sku": "014"} on separate lines. The strings keep their leading zeros.
  • If the real file has an extra container level, adjust item_depth to that structure. Treat NDJSON as a stream of individual records and handle incomplete output if an input error interrupts a real batch.

References