data-formats

# Data Formats How to work with diverse and unknown data formats. ## Format Detection Always inspect before parsing: ```bash file <filename> # MIME type detection xxd <filename> | head -5 # hex dump (first bytes) head -3 <filename> # text preview python3 -c " with open('<filename>', 'rb') as f: h = f.read(16) print(h, h.hex()) " ``` ## Common Formats ### Binary - **Magic bytes**: Most binary formats start with a signature (ELF: `\x7fELF`, PNG: `\x89PNG`) - **Endianness**: Check if little-endian or big-endian (`struct.unpack('<I', ...)` vs `'>I'`) - **Alignment**: Fields are often aligned to 4 or 8 bytes - **Offsets**: Binary headers often contain offsets to other sections ### Structured text - **CSV/TSV**: Check delimiter (comma, tab, pipe), quoting, header row - **JSON**: `python3 -c "import json; json.load(open('f'))"` - **YAML**: Check indentation, anchors/aliases - **TOML**: `python3 -c "import tomllib; ..."` - **XML**: Check encoding declaration, namespaces ### Checkpoints / Model files - **PyTorch**: `.pt`, `.pth` → `torch.load(f, map_location='cpu')` - **TensorFlow**: `.ckpt` → index + data files, use `tf.train.load_checkpoint()` - **NumPy**: `.npy`, `.npz` → `numpy.load()` - **HuggingFace**: `config.json` + `model.safetensors` - **ONNX**: `onnx.load()` ### Database files - **SQLite**: `file` says "SQLite 3.x database" → `sqlite3 <file> ".tables"` - **WAL files**: SQLite write-ahead log — recover with `sqlit...