Stream Records From a Large XML File With xmltodict
A full tree is convenient for small XML files. For a large feed with independent records, xmltodict can hand each finished record to a callback that writes it out immediately.
Worked example
A small in-memory feed for the streaming callback
from io import BytesIO
import json
import xmltodict
source = BytesIO(
b"<catalog><item><sku>003</sku></item>"
b"<item><sku>014</sku></item></catalog>"
)
def write_item(path, item):
print(json.dumps(item, ensure_ascii=False))
return True
xmltodict.parse(source, item_depth=2, item_callback=write_item)Steps
- Install xmltodict in a Python environment and save the example as stream.py. The synthetic feed has catalog at depth one and each item at depth two.
- Run python3 stream.py > items.ndjson. Confirm that there are two lines and that each line parses as one complete JSON object.
- For a trusted file of the same shape, replace the BytesIO source with a file opened in binary mode. Keep the callback streaming to its destination instead of collecting records into a list.
What to check
- The sample writes {"sku": "003"} and {"sku": "014"} on separate lines. The strings keep their leading zeros.
- If the real file has an extra container level, adjust item_depth to that structure. Treat NDJSON as a stream of individual records and handle incomplete output if an input error interrupts a real batch.