datalad

# DataLad ## Overview DataLad is a data management layer over Git and git-annex. Git tracks the dataset structure, small text files, and the history. git-annex tracks the *content* of large files, storing each file as a key and keeping the bytes somewhere that is not necessarily the local repository. That split is the single most important thing to internalise, because it means a freshly cloned dataset contains the full history and the full file listing while containing almost none of the data. A 100 TB dataset clones in seconds and occupies a few megabytes. The bytes arrive only when asked for, per file, with `datalad get`. The second thing DataLad adds is provenance. `datalad run` executes a command and commits the result together with a machine-readable record of the command, its inputs, and its outputs. `datalad rerun` reads that record back and re-executes it. This turns "how was this figure produced" from an archaeology problem into a command. ## When to use DataLad instead of plain Git Use DataLad when any of the following holds: - Files are too large for Git to handle comfortably, or the total exceeds what every collaborator wants on disk. - Data lives in more than one place (a lab server, a cluster scratch, S3, a supercomputer) and you need to know which copies exist. - The analysis must be re-executable, and a plain commit message is not enough evidence. - You are consuming published datasets from OpenNeuro, DANDI, or `datasets.datalad.org`, which are d...