TB: Reproducible Research Code and Data

Overview

How researchers who write code for a living — but were never trained as software engineers — should manage code and data. The canonical statement is Gentzkow & Shapiro's Code and Data for the Social Sciences: A Practitioner's Guide (2014), a ~40-page handbook that translates software-engineering and database practice into rules a small research group can sustain. Its founding heuristic: if multi-billion-dollar firms and whole university courses exist to solve your problem, find out what the experts do.

Core Tension

Discipline costs time now and pays later — and research (unlike product engineering) constantly tempts you to defer the payment because "this analysis is one-off." Gentzkow & Shapiro's empirical counterclaim: "you will end up running every step more times than you think." The rules are calibrated to what a two-professor-plus-RAs group can actually maintain, not to industrial best practice — hence anti-rules like "Otherwise, don't abstract" and "Don't write documentation you will not maintain."

Key Insights Across Sources

The eight rule-pairs (Gentzkow & Shapiro 2014)

  • Automation: Automate everything that can be automated; write a single script that executes all code from beginning to end. The run script is the only build documentation that "cannot be incomplete, ambiguous, or out of date." Replicability test: delete all outputs, run one script, get them back exactly.
  • Version control: Store code AND data under version control; run the whole directory before checking it back in (a commit is only valid after a full clean end-to-end run — CI discipline before CI was mainstream). "Not one piece of commercial software… was written with the 'date and initial' method."
  • Directories: Separate by function (build vs. analysis); separate inputs/code/output/temp; make directories portable — analysis code links to fixed revisions of shared datasets so upstream changes never silently break downstream results.
  • Keys: Store cleaned data in tables with unique, non-missing keys; keep data normalized as far into the pipeline as possible, denormalizing only at the last (estimation) step with zero manipulation during the merge. "The physical structure of a database should communicate its logical structure" — normalized data is self-documenting.
  • Abstraction: Abstract to eliminate redundancy and improve clarity; otherwise, don't — premature generality is a real cost.
  • Documentation: Don't write documentation you will not maintain; code should be self-documenting. Every comment is a second representation of the same information that will eventually contradict the first. The furnace-switch story: a "do not touch" sign fails, tape over the switch works — validation beats warnings.
  • Management: Manage tasks with a task management system; e-mail is not a task management system. Ambiguity of ownership/status grows "more than arithmetically" with team size.
  • Code style (appendix): short shy functions, descriptive names, set off the key algebra, unit tests ("economists… test their code anyway. They just do it manually" — same effort, test discarded), profile slow code, store "too much" output from slow code, separate slow code from fast code.

Transfer to experimental HEP practice (first-party assessment, 2026-07-27)

Most of the handbook is baseline hygiene a modern HEP workflow already exceeds (scripted pipelines, git, issue trackers). Three rules name real live gaps:

  • Provenance-complete reruns: HEP analyses rarely re-derive everything before a commit (compute cost), which is how "which ntuple version made this plot?" archaeology happens. Portable form: a committed provenance chain (code tag + dataset version + config) making any published plot reproducible by one command.
  • Pinned data revisions: shared skim/ntuple areas on group storage are exactly their "shared drive" failure mode — pin by dataset tag in config, upgrade deliberately, never point at "latest path."
  • Normalized analysis metadata: event data is already columnar/keyed, but cross-sections, SF versions, golden JSONs, and sample lists scattered across dicts at mixed aggregation levels are the denormalized county/state mess. One keyed samples-table, merged into job configs at the last step.

Key Quotes

  • "If you are trying to solve a problem, and there are multi-billion dollar firms whose entire business model depends on solving the same problem… you might want to figure out what the experts do."
  • "Version control is like an undo command for everything."
  • "If it's not worth maintaining a piece of documentation up to that standard, it probably isn't worth writing it in the first place."
  • "Write your code so it will not let those inputs in the door in the first place."

Related Concepts

  • Unit Tests — the handbook independently lands on unit testing as specification + regression detection; manual interactive testing is the same work with the test thrown away
  • It Pays to Get the Design Right — directory structure and keys are design tuned to the problem; structure replaces documentation

Cross-Topic Connections

Backlinks