How Scientific Reproducibility Works | From Data, Code and Method to a Result Another Researcher Can Reconstruct

How Scientific Reproducibility Works | From Data, Code and Method to a Result Another Researcher Can Reconstruct is a world-facing guide to scientific reproducibility, computational reproducibility, reproducible research, open data, open code, research software, data provenance, analysis pipelines, version control, computational environments, workflow capture, random seeds, containers, notebooks, research transparency, auditability and scientific rigor. A scientific result is reproducible when another researcher can take the same underlying data, the same stated methods and the same computational steps and recover a consistent result.

This page uses the National Academies convention for precision: reproducibility means obtaining consistent computational results using the same input data, computational steps, methods, code and conditions of analysis, while replicability means obtaining consistent results across new studies that collect new data to answer the same scientific question. Different fields sometimes reverse or blur these terms, so good scientific writing should define them explicitly. This guide uses the National Academies distinction throughout.

Within the eduKateYishun scientific-method library, How Scientific Replication Works focuses on the new-data test; How Scientific Evidence Works focuses on confidence from observations; How Scientific Measurement Works focuses on trustworthy quantities; How Scientific Experiments Work focuses on test architecture; and How Scientific Hypotheses Work focuses on candidate claims. This guide follows the reconstruction loop: raw data → documented preprocessing → code and software environment → analysis → figures and tables → reported result → independent rerun → discrepancy diagnosis → reproducible record.

---

Choose your route

• Basic route: what reproducibility is and why it matters.
• Data route: raw data, metadata, provenance, cleaning and versioning.
• Code route: scripts, notebooks, software versions, dependencies, seeds and containers.
• Workflow route: how one command or pipeline rebuilds tables, figures and results.
• Audit route: how another researcher checks a published result step by step.
• Open-science route: sharing, privacy, licensing, repositories and persistent identifiers.
• Advanced route: deterministic versus stochastic pipelines, environment drift, numerical instability, hardware differences and long-term computational preservation.

1. Reproducibility asks whether the reported result can be reconstructed

A published paper may show:

• a table;
• a figure;
• a regression coefficient;
• a model score;
• a simulation result;
• a statistical test.

Reproducibility asks:

If another researcher starts from the same input data and follows the stated computational method, can they recover a consistent result?

This is a minimum auditability standard for computational science.

2. Reproducibility is not new-data replication

If another researcher uses the original dataset and code to regenerate Figure 3, that is reproducibility under the convention used here.

If another researcher collects a new dataset and asks whether the finding appears again, that is replication.

The National Academies distinguishes these two because their failure modes are different. A result can be computationally reproducible yet fail to replicate in new data; it can also be empirically replicable while the original paper’s computational workflow is poorly documented.

3. Reproducibility is about the research object, not the researcher’s memory

Suppose the original analyst says:

“I know what I did.”

That is not enough.

A reproducible study should leave behind enough information that another competent researcher can reconstruct the analysis without relying on undocumented personal memory.

4. The reproducible research object

A useful reproducible package can include:

• raw or legally shareable source data;
• data dictionary;
• metadata;
• cleaning code;
• analysis code;
• software versions;
• dependency list;
• random seeds where relevant;
• workflow instructions;
• figure-generation code;
• table-generation code;
• README documentation;
• licensing and citation information.

The goal is not maximal paperwork.

The goal is enough structure to reconstruct the published result reliably.

5. Reproducibility is a chain

A result can fail to reproduce at many points:

data source → download → cleaning → coding → model → software → output → manuscript

If one link is undocumented, the final number may become impossible to verify.

6. Raw data matter

Analysts often work with cleaned datasets, but reproducibility improves when the path from original source to analytic dataset is preserved.

Why?

Because cleaning decisions can change:

• sample size;
• outliers;
• missing values;
• variable coding;
• units;
• time windows.

A clean dataset without cleaning history hides scientific decisions.

7. Raw does not mean perfect

Raw data can contain:

• errors;
• duplicate records;
• sensor artifacts;
• missing values;
• invalid entries.

Reproducibility does not require preserving errors forever.

It requires preserving how errors were detected and corrected.

8. Data provenance

Data provenance answers:

• Where did the data come from?
• Who collected them?
• When?
• With which instrument?
• Under which protocol?
• Which transformations occurred later?

Without provenance, identical-looking columns can represent very different scientific objects.

9. Provenance protects against silent substitution

Suppose a dataset is updated by the provider after the paper is published.

If the analyst later downloads the “same” file without recording the exact version, the numbers may change.

Reproducibility therefore needs:

• version identifiers;
• timestamps;
• checksums;
• persistent archived copies where permitted.

10. Checksums

A checksum is a digital fingerprint of a file.

If two files have different checksums, they differ somewhere.

Checksums help verify that the input data are exactly the intended version.

11. Data dictionaries

A data dictionary should explain:

• variable names;
• meaning;
• units;
• coding;
• missing-value symbols;
• valid ranges.

Without this, a column named “age” might mean:

• years;
• months;
• age at enrolment;
• age at follow-up.

Reproducibility depends on semantics, not only bytes.

12. Unit consistency

A classic failure is mixing:

• kilograms and pounds;
• metres and centimetres;
• seconds and milliseconds;
• local time and UTC.

Units should be explicit and machine-checkable where practical.

13. Time zones

Time-based data can fail reproducibility through:

• daylight saving changes;
• local versus UTC timestamps;
• ambiguous date formats;
• time-zone conversion.

Record time-zone assumptions explicitly.

14. Missing data codes

Values such as:

• -99;
• 999;
• NA;
• blank;

may mean missing.

If software interprets -99 as a real measurement, the analysis can become nonsense while still running successfully.

15. Data cleaning should be code, not memory

Manual spreadsheet edits are difficult to audit.

Prefer scripted cleaning where possible:

raw data + cleaning code → analytic data

This makes the transformation repeatable and inspectable.

16. Manual edits sometimes happen

If a manual correction is unavoidable, record:

• which cell or record;
• original value;
• new value;
• reason;
• date;
• person making the change.

Unlogged edits are reproducibility debt.

17. Reproducibility debt

Reproducibility debt accumulates when:

• code is undocumented;
• data versions are unknown;
• analysis occurs interactively without scripts;
• dependencies are unpinned;
• file paths live only on one laptop.

The study may work today and become impossible to reconstruct later.

18. Reproducibility should be built during the project

Trying to “make it reproducible” after publication is expensive.

Better:

• version from the beginning;
• script transformations;
• automate outputs;
• document dependencies.

---

19. Code is part of the method

In computational science, a paragraph saying:

“We analysed the data in Python”

is not a complete method.

Code can contain:

• filter rules;
• transformations;
• model choices;
• defaults;
• randomness;
• plotting decisions.

Sharing code exposes the actual implementation.

20. Analysis scripts

A strong analysis pipeline should make it clear:

• which script runs first;
• which files it reads;
• which files it writes;
• which parameters it uses.

Numbered scripts or a workflow manager can help.

21. One-button rebuild

An ideal reproducible project can rebuild major outputs with one documented command.

For example:

make all

or an equivalent workflow command.

The exact tool does not matter.

The principle does:

manuscript outputs should derive from recorded inputs and code, not hand-edited final files.

22. Literate programming

Notebooks and literate-programming systems combine:

• code;
• explanation;
• figures;
• results.

This can make the analysis easier to follow.

But notebooks can also hide state if cells are run out of order.

23. Notebook state problems

A notebook may show a result that cannot be reproduced from a clean restart because:

• cells were executed out of order;
• variables persisted;
• hidden files existed;
• manual edits occurred.

Before publication, restart the environment and run the notebook from top to bottom.

24. Scripts versus notebooks

Neither is universally superior.

Scripts are often better for:

• automation;
• testing;
• large pipelines.

Notebooks are often useful for:

• exploration;
• teaching;
• narrative analysis.

A reproducible project can combine both.

25. Random seeds

Stochastic algorithms can produce different outputs each run.

Examples include:

• random sampling;
• machine-learning initialisation;
• simulation;
• bootstrap procedures.

Record seeds when exact reruns matter.

26. Seeds are not the whole story

Even with the same seed, results can differ because of:

• library versions;
• parallel computation;
• GPU nondeterminism;
• hardware;
• algorithm implementation.

Exact determinism must be tested, not assumed.

27. Software versions

A function can change between library versions.

Defaults can change.

Bugs can be fixed.

Numerical solvers can change.

Record exact versions.

28. Dependency pinning

Dependency files can record exact package versions.

Examples include environment files, lockfiles or package manifests.

The goal is not loyalty to one tool.

It is an explicit computational environment.

29. Operating system

Some analyses behave differently on:

• Linux;
• macOS;
• Windows.

Differences can arise from:

• file paths;
• system libraries;
• compiler versions;
• locale settings.

30. Locale

Locale can change:

• decimal separators;
• date parsing;
• sorting;
• character encoding.

These details can silently alter analysis.

31. Character encoding

UTF-8 versus other encodings can change:

• text parsing;
• category labels;
• string matching.

Record encoding explicitly when working with text data.

32. Containers

Containers package:

• software;
• libraries;
• system dependencies;
• configuration.

They improve portability by recreating a computational environment more faithfully.

33. Containers are not permanent by themselves

A container image can disappear from a registry.

Long-term reproducibility benefits from:

• archived images;
• Dockerfiles or recipes;
• version tags;
• checksums.

34. Virtual machines

Virtual machines can preserve a fuller operating-system environment.

They can be heavier than containers but useful for complex legacy software.

35. Workflow managers

Workflow systems can record dependencies:

raw data → cleaned data → model → figure → manuscript

If one input changes, the workflow knows what must be rebuilt.

36. Make, Snakemake, Nextflow and similar tools

Different fields use different workflow systems.

The important principles are:

• explicit dependencies;
• automation;
• logged steps;
• failure detection.

37. Continuous integration

Continuous-integration systems can rerun:

• tests;
• analysis;
• documentation;
• figures.

when code changes.

This catches reproducibility breaks early.

38. Unit tests for science code

Scientific code should test important functions.

Examples:

• unit conversion;
• data filtering;
• equation implementation;
• known toy cases.

Tests do not prove scientific validity, but they reduce software error.

39. Regression tests

A regression test checks whether code changes unexpectedly alter known outputs.

This is especially useful for long-lived research software.

40. Scientific code is software

Research code may begin as a quick script.

If major conclusions depend on it, software quality becomes scientific quality.

---

41. Reproducibility begins before analysis

The most common mistake is treating reproducibility as a publication-stage task.

By then:

• file names have drifted;
• temporary scripts have disappeared;
• manual edits are forgotten;
• software versions have changed;
• the analyst no longer remembers why one threshold was chosen.

Reproducibility should be designed from project start.

42. Project structure

A clear project structure can separate:

• raw data;
• processed data;
• code;
• figures;
• tables;
• documentation;
• results.

The exact folder names do not matter.

What matters is predictable organisation.

43. Raw data should be protected from accidental overwrite

One strong practice is to treat raw data as read-only.

Cleaning code should write new processed files rather than modify the source file in place.

This preserves the original evidence object.

44. Intermediate data

Complex analyses can produce multiple intermediate datasets.

Document:

• which script created each file;
• which source file it used;
• which filters were applied;
• which variables were derived.

Intermediate files without provenance become dead ends.

45. Data lineage

Data lineage traces a value from:

source measurement → transformation → analytic variable → model → published result

This makes it possible to answer:

“Where did this number in Table 2 come from?”

46. Every published number should have a path

An ideal reproducible paper can answer:

• Which dataset created this number?
• Which script produced it?
• Which parameters were used?
• Which software version ran it?

That path is the computational provenance of the result.

47. File naming matters

Names such as:

• final.csv;
• final2.csv;
• final_really_final.csv;

are signs of manual versioning.

Use meaningful names and version-control systems instead.

48. Version control

Version control records:

• what changed;
• when;
• who changed it;
• why, through commit messages.

Git is widely used, but the principle matters more than the tool.

49. Commit small logical changes

A commit called:

“Fix everything”

is less useful than:

• “Correct unit conversion in temperature preprocessing”;
• “Add exclusion rule from preregistration”;
• “Pin package versions.”

Commit history becomes a scientific audit trail.

50. Tag releases

When a paper is submitted or published, tag the exact code version used.

This prevents later development from blurring the historical analysis.

51. Persistent identifiers

Repositories can assign persistent identifiers such as DOIs to archived releases.

This lets readers cite the exact research artifact.

52. Repositories

Depending on field and restrictions, research artifacts can be archived in:

• institutional repositories;
• general scientific repositories;
• discipline-specific repositories;
• code archives.

A repository should provide durable access beyond one personal website.

53. Persistent access matters

A link to:

“~/Desktop/project/data.csv”

works only for one computer.

A reproducible project should avoid:

• local absolute paths;
• personal network drives;
• undocumented cloud folders.

54. Relative paths

Relative paths help projects run on another computer.

Instead of:

C:\Users\Name\Desktop\study\data.csv

use a project-relative path where appropriate.

55. Configuration files

Analysis parameters should be separated from code when possible.

A configuration file can record:

• data locations;
• thresholds;
• model options;
• seed;
• output directory.

This reduces hidden constants inside scripts.

56. Magic numbers

A magic number is an unexplained constant inside code.

Example:

if score > 17.3:

Why 17.3?

Reproducibility requires a reason:

• pre-specified threshold;
• instrument limit;
• published standard.

57. Defaults should be explicit

Statistical software often has defaults for:

• missing-data handling;
• optimisation;
• regularisation;
• confidence intervals;
• random seeds.

If defaults change between versions, results can change.

Record important settings explicitly.

58. Warnings are scientific information

Warnings such as:

• model failed to converge;
• matrix is singular;
• missing values removed;
• overflow occurred;

should not be ignored automatically.

Capture logs.

59. Logging

A reproducible pipeline can log:

• start time;
• data version;
• software version;
• parameters;
• warnings;
• runtime;
• output paths.

Logs make future debugging much easier.

60. Error handling

Code should fail loudly when critical assumptions are violated.

Examples:

• unexpected missing columns;
• impossible units;
• duplicate identifiers;
• wrong data version.

Silent failure creates false reproducibility because the pipeline completes while producing the wrong result.

---

61. Environment capture

The same code can behave differently in another environment.

Capture:

• language version;
• package versions;
• system libraries;
• compiler;
• operating system;
• hardware when relevant.

62. Lockfiles

Lockfiles record exact dependency versions and sometimes cryptographic hashes.

They help another researcher install the same environment.

63. Environment files

Examples include:

• requirements files;
• Conda environments;
• R renv lockfiles;
• Julia manifests;
• JavaScript lockfiles.

Different languages use different tools.

64. System libraries matter

A statistical package may depend on:

• BLAS;
• LAPACK;
• C/C++ libraries;
• graphics libraries.

Differences can affect numerical output.

65. Compiler differences

Scientific software compiled with different settings can show small numerical differences.

Usually these are negligible.

Sometimes they affect unstable algorithms.

66. Floating-point arithmetic

Computers represent real numbers approximately.

Operations such as:

• addition;
• subtraction;
• matrix inversion;
• iterative optimisation;

can accumulate rounding error.

67. Order of operations

Floating-point addition is not perfectly associative.

Large parallel computations may sum values in different orders and produce tiny differences.

Exact bitwise identity is therefore not always realistic across hardware.

68. Scientific reproducibility can tolerate numerical tolerance

If two runs differ at the 12th decimal place but yield the same scientific conclusion, they may be considered computationally consistent.

The tolerance should match the scientific precision.

69. Numerical instability

An algorithm is numerically unstable when tiny input or rounding differences create large output changes.

Reproducibility testing can expose this.

70. Condition numbers

In numerical linear algebra, a high condition number can indicate that a solution is sensitive to small input errors.

Scientific analysts should know when their computations are ill-conditioned.

71. Solver tolerances

Optimisation and differential-equation solvers use stopping tolerances.

Different tolerances can change:

• parameter estimates;
• simulation trajectories;
• convergence status.

Record them when scientifically relevant.

72. Stochastic optimisation

Machine learning often uses stochastic optimisation.

Reproducibility should report:

• seed;
• initialisation;
• batch order;
• number of runs.

73. One seed is not enough for robust science

Fixing one seed can reproduce one run.

It does not show the result is stable across random initialisations.

Report distributions across seeds where randomness affects performance materially.

74. Deterministic modes

Some machine-learning frameworks provide deterministic settings.

These can reduce run-to-run variation but may:

• slow computation;
• change algorithms;
• remain hardware-dependent.

75. GPU nondeterminism

Parallel GPU operations may execute in variable order.

Exact reruns can therefore differ slightly even with the same seed.

Document expectations honestly.

76. Hardware recording

For hardware-sensitive analyses, record:

• CPU/GPU model;
• accelerator libraries;
• driver versions;
• precision mode.

77. Mixed precision

Machine-learning systems may use:

• 32-bit;
• 16-bit;
• brain floating point;
• mixed precision.

Precision choice can affect training and inference.

78. Distributed computing

Large analyses can depend on:

• worker count;
• task ordering;
• network timing;
• checkpoint recovery.

Distributed reproducibility requires additional metadata.

79. Cloud computing

Cloud environments can be ephemeral.

Record infrastructure through code or configuration where possible.

Infrastructure-as-code can help reconstruct:

• machine type;
• storage;
• network;
• software image.

80. Containers plus workflow plus archive

A durable computational record often combines:

data snapshot + code version + environment recipe + workflow + archived outputs

No single artifact is enough.

---

81. Research software engineering

Research software can be central to scientific claims.

Good practices include:

• modular code;
• tests;
• documentation;
• version control;
• code review;
• issue tracking.

82. Code review

Another researcher reading code can catch:

• wrong units;
• incorrect indexing;
• off-by-one errors;
• data leakage;
• inverted conditions.

Code review is a scientific quality-control process.

83. Pair programming

For high-stakes analysis, two researchers can work together or independently implement the same calculation.

Agreement increases confidence that one coding mistake is not driving the result.

84. Independent implementation

If two teams implement the same mathematical method independently and obtain the same result from the same data, that is a strong computational cross-check.

It remains reproducibility, not new-data replication.

85. Reference implementations

A simple, transparent implementation can serve as a reference against which optimised code is checked.

Speed should not outrun verification.

86. Synthetic test data

Build small datasets where the expected answer is known.

Use them to test:

• data cleaning;
• model code;
• edge cases;
• missing values.

87. Simulation-based calibration

In some Bayesian workflows, simulate data from known parameters, fit the model and check whether inference recovers calibrated uncertainty across repeated simulations.

This tests the computational inference pipeline.

88. Posterior predictive checks are not reproducibility tests

They evaluate model fit.

Reproducibility asks whether the same analysis can be reconstructed.

Different scientific jobs should remain distinct.

89. Tests for data pipelines

A data pipeline can test:

• expected number of rows;
• unique identifiers;
• valid category values;
• date ranges;
• unit bounds.

These are scientific assertions about data.

90. Schema validation

A schema defines:

• columns;
• types;
• required fields;
• allowed values.

Schema validation catches unexpected data changes before analysis.

91. Data contracts

When one team supplies data to another, a data contract can define:

• field names;
• units;
• update frequency;
• missingness rules.

This reduces pipeline drift.

92. Reproducibility across time

A project reproducible today may fail ten years later because:

• software is unavailable;
• web APIs changed;
• operating systems disappeared;
• data links broke.

Long-term reproducibility requires preservation.

93. Bit rot

Digital files can become corrupted.

Archives use:

• checksums;
• redundant storage;
• integrity audits.

94. Link rot

Web resources disappear.

Archive critical artifacts in persistent repositories rather than relying on ordinary website links.

95. API drift

An analysis that downloads live data from an API may produce a different dataset later.

Archive the exact input snapshot where licensing permits.

96. Dynamic databases

Databases can be corrected or updated.

Record:

• query;
• access date;
• database version;
• exported snapshot.

97. Web scraping

Scraped web content is especially fragile.

Sites change.

Document:

• URL;
• date;
• scraper version;
• archived raw HTML where lawful.

98. External services

Cloud APIs can change model versions or output behaviour.

If scientific results depend on an external service, record:

• provider;
• model/version identifier;
• date;
• parameters;
• request format;
• response archive where permitted.

99. Proprietary software

Proprietary tools can still support reproducible science, but access barriers matter.

Record:

• software name;
• version;
• license requirements;
• settings;
• export formats.

100. Proprietary algorithms

If a crucial algorithm is closed and cannot be inspected, independent reproducibility is limited.

This does not automatically invalidate the result.

It limits auditability and should be disclosed.

---

101. Reproducibility and privacy

Open data are not always ethical or legal.

Human-subject data can contain:

• medical information;
• location;
• genetics;
• identity;
• sensitive behaviour.

Reproducibility must respect privacy.

102. De-identification

Removing names is not always sufficient.

People can sometimes be reidentified through combinations of:

• age;
• location;
• dates;
• rare traits.

103. Controlled access

Some data should be shared through:

• data-use agreements;
• secure enclaves;
• approved researchers;
• audited environments.

Reproducibility does not require public download by everyone.

104. Synthetic data

Synthetic datasets can help others understand code and data structure without exposing real records.

They cannot always reproduce the exact scientific result because they are not the original data.

Use them as documentation, not substitute evidence.

105. Differential privacy

Differential privacy adds controlled noise to protect individuals.

This can alter exact outputs.

A reproducibility package should document the privacy mechanism and randomness.

106. Secure computation

Some projects allow researchers to send code to secure data environments.

The code runs near the protected data; only approved outputs leave.

This can support audit without raw-data release.

107. Data-use agreements

Agreements can specify:

• who may access data;
• what analyses are allowed;
• how outputs are reviewed;
• retention period.

108. Ethical reproducibility

The goal is:

maximum auditability consistent with participant rights, consent, law and community governance.

Open science is not permission to ignore ethics.

109. Indigenous and community data governance

Some datasets involve communities with collective rights and governance expectations.

Reproducibility plans should respect:

• community consent;
• data sovereignty;
• cultural protocols;
• benefit sharing.

110. Restricted data can still be reproducible

A study can be reproducible if:

• qualified researchers can request access;
• analysis code is shared;
• environment is documented;
• access procedure is clear.

---

111. Licensing

Data and code need legal reuse terms.

Without a license, others may be technically able to download artifacts but not know whether reuse is permitted.

112. Code licenses

Open-source licenses specify reuse rights and obligations.

Choose a license appropriate to the project and institution.

113. Data licenses

Data may require different licenses than software.

Licensing should respect:

• source agreements;
• participant consent;
• database rights;
• institutional rules.

114. Citation files

A citation file can tell users how to cite:

• code;
• dataset;
• software release.

This improves credit for reproducibility work.

115. Credit matters

Cleaning data, writing software and maintaining workflows are scientific contributions.

Fields should credit:

• data curators;
• research software engineers;
• method developers;
• reproducibility auditors.

116. Reproducibility as infrastructure

Reliable research depends on work that may not look like discovery:

• documentation;
• software maintenance;
• archiving;
• testing;
• curation.

Infrastructure deserves institutional support.

117. NIH rigor and transparency

NIH currently emphasises rigor, transparent methods and reproducible biomedical research as core elements of trustworthy science. Its guidance highlights rigorous design, methodology, analysis, interpretation and reporting, and its newer replication/reproducibility initiatives aim to build infrastructure for “gold standard” science. NIH — Enhancing Reproducibility through Rigor and Transparency and NIH — Strengthening Replication and Reproducibility provide current institutional context.

118. Reproducibility is broader than open code

A GitHub repository can exist and still be unusable.

Reproducibility requires:

• correct code;
• correct data;
• correct environment;
• clear instructions;
• complete workflow.

119. Open but unreproducible

Common pattern:

• code shared;
• data missing;
• dependencies unpinned;
• instructions incomplete.

Transparency is necessary but not sufficient.

120. Closed but reproducible under controlled access

A sensitive medical study can be reproducible if another authorized researcher can access:

• same data;
• same code;
• same secure environment.

Public openness and scientific reproducibility overlap but are not identical.

---

121. Cross-Domain Reproducibility Casebook

Case 1 — Spreadsheet analysis

A researcher manually filters rows, copies values and builds a chart.

The paper is published.

Months later, nobody remembers:

• which rows were removed;
• which formula was typed;
• whether hidden cells existed.

Reproducibility repair:

• preserve original workbook;
• record filtering rules;
• convert key transformations to scripts;
• document formulas.

Case 2 — Statistical software menu clicks

An analyst uses a graphical interface and reports a regression.

Another researcher cannot tell:

• which options were selected;
• which missing-data rule was used;
• whether interactions were included.

Repair:

export syntax or analysis script and save the exact software version.

Case 3 — R analysis

The script runs today but fails two years later because package functions changed.

Repair:

• lock package versions;
• archive environment;
• tag code release.

Case 4 — Python notebook

A notebook displays the final figure but fails after “Restart and Run All”.

Cause:

a variable was created in a cell executed earlier but later moved.

Repair:

require clean-state execution before release.

Case 5 — MATLAB simulation

A simulation depends on a proprietary toolbox.

Another lab has a newer toolbox and gets slightly different results.

Repair:

• record exact version;
• archive model files;
• specify solver and tolerances;
• report acceptable numerical tolerance.

Case 6 — Climate model

A large climate simulation may require:

• specific initial conditions;
• forcing datasets;
• compiler;
• supercomputer architecture;
• checkpoint files.

Exact bitwise reproduction may be difficult.

Scientific reproducibility can instead require consistent climatological outputs within defined tolerances.

Case 7 — Molecular dynamics

Trajectory diverges after tiny numerical changes.

That does not automatically mean the scientific result is unreproducible.

Compare reproducible summary statistics:

• energy distributions;
• structural ensembles;
• diffusion constants.

Case 8 — Monte Carlo simulation

Different seeds produce different individual draws.

Reproducibility standard:

• same seed can reproduce one run;
• distribution across seeds remains statistically consistent.

Case 9 — Bayesian MCMC

Posterior summaries differ because chains did not converge.

Repair:

• save seed;
• report diagnostics;
• increase effective sample size;
• use multiple chains.

Case 10 — Machine-learning benchmark

A paper reports one best accuracy.

Another team reruns the code and gets lower values.

Possible reasons:

• different random seed;
• GPU kernel;
• data split;
• library version;
• hidden preprocessing.

Reproducibility should include seed distribution and exact split definition.

Case 11 — Deep-learning training

Training is nondeterministic across GPUs.

Instead of demanding exact weights, define reproducibility at:

• validation metric;
• calibration;
• prediction distribution;
• scientific conclusion.

Case 12 — Foundation-model API

A study calls an external model API.

The provider later updates the model.

Reproducibility requires:

• model identifier;
• date;
• temperature and parameters;
• prompt;
• response archive where terms allow.

Case 13 — Web search study

Search rankings change daily and by location.

Archive:

• query;
• date/time;
• region;
• results;
• browser state where relevant.

Case 14 — Social-media dataset

Platform API later removes fields.

Reproducibility may depend on an archived lawful snapshot and data dictionary.

Case 15 — Genomics pipeline

Results depend on:

• reference genome version;
• aligner version;
• variant caller;
• filter thresholds.

Every component should be versioned.

Case 16 — RNA-seq analysis

Gene identifiers change across annotation releases.

Record:

• reference build;
• annotation database;
• normalisation method;
• count filtering.

Case 17 — Proteomics

Results depend on:

• instrument settings;
• search database;
• mass tolerances;
• software;
• FDR procedure.

Reproducibility spans wet lab and computation.

Case 18 — Medical imaging

Image analysis can change with:

• scanner;
• reconstruction algorithm;
• segmentation software;
• normalisation.

Archive acquisition and analysis parameters.

Case 19 — fMRI

Results can depend on:

• preprocessing;
• motion correction;
• smoothing;
• statistical threshold.

Reproducibility improves through complete workflow sharing and standard data formats.

Case 20 — Epidemiological registry

Database updates can change diagnoses and follow-up.

Record extraction date and query.

Prefer archived analytic extracts where permitted.

Case 21 — Electronic health records

Clinical data pipelines depend on:

• coding systems;
• hospital workflows;
• ETL logic;
• de-identification.

Reproducibility requires more than sharing the final CSV.

Case 22 — Laboratory instrument export

Instrument software performs hidden preprocessing before export.

Archive:

• raw instrument file;
• firmware/software version;
• processing settings;
• export format.

Case 23 — Microscopy

Image quantification depends on:

• exposure;
• gain;
• segmentation threshold;
• image processing.

Store raw images, not only processed figures.

Case 24 — Flow cytometry

Gating decisions can change results.

Reproducibility requires:

• raw FCS files;
• compensation settings;
• gating strategy;
• software version.

Case 25 — Mass spectrometry

Different search engines and databases can produce different identifications.

Archive raw spectra and search configuration.

Case 26 — Telescope data

Astronomical analysis can depend on calibration pipelines and reference catalogs.

Record exact data release and calibration version.

Case 27 — GIS analysis

Spatial results can change with:

• projection;
• coordinate system;
• boundary file;
• raster resolution.

Record all spatial reference information.

Case 28 — Survey weighting

Survey results depend on:

• weights;
• strata;
• clusters;
• finite-population corrections.

Final percentages are not reproducible without design metadata.

Case 29 — Text analysis

Results can change with:

• tokenisation;
• stemming;
• stop-word list;
• language model;
• encoding.

Archive preprocessing resources.

Case 30 — Qualitative coding software

Reproducibility in qualitative analysis does not mean one machine regenerates an interpretation automatically.

Auditability can still improve through:

• coding framework;
• decision trail;
• software export;
• reflexive notes.

Terminology should match the field’s epistemic goals.

---

121. Reproducibility Failure Atlas

Failure 1 — Missing raw data

Pattern: only cleaned dataset survives.

Repair: archive source data or document controlled-access path.

Failure 2 — Unknown data version

Pattern: dynamic database changed.

Repair: record snapshot, date, query and checksum.

Failure 3 — Manual spreadsheet edits

Pattern: transformations cannot be reconstructed.

Repair: script or log edits.

Failure 4 — Hidden Excel formulas

Pattern: copied values lose formula history.

Repair: preserve workbook and formula documentation.

Failure 5 — Absolute paths

Pattern: code only runs on original laptop.

Repair: use project-relative paths.

Failure 6 — Missing dependency file

Pattern: package versions unknown.

Repair: lock environment.

Failure 7 — Package update

Pattern: default changed between versions.

Repair: pin package and explicit options.

Failure 8 — Random seed missing

Pattern: stochastic output changes.

Repair: record seed and distribution across seeds.

Failure 9 — GPU nondeterminism

Pattern: same seed, different weights.

Repair: document nondeterminism and define metric tolerance.

Failure 10 — Notebook out of order

Pattern: displayed output depends on hidden state.

Repair: restart and run all before release.

Failure 11 — Missing environment variable

Pattern: secret or path expected implicitly.

Repair: document required environment configuration without exposing secrets.

Failure 12 — External API changed

Pattern: same request returns different schema.

Repair: archive inputs/outputs and API version.

Failure 13 — Data link rotted

Pattern: URL returns 404.

Repair: use persistent repository.

Failure 14 — Proprietary software unavailable

Pattern: old license no longer obtainable.

Repair: archive environment where legal and export open intermediate formats.

Failure 15 — Operating system unavailable

Pattern: legacy program will not run.

Repair: VM or container preservation.

Failure 16 — Locale mismatch

Pattern: comma decimal parsed as text.

Repair: explicit locale.

Failure 17 — Date parsing mismatch

Pattern: 03/04 interpreted differently.

Repair: ISO dates and explicit parser.

Failure 18 — Time-zone mismatch

Pattern: events shift dates.

Repair: store timezone-aware timestamps.

Failure 19 — Character encoding mismatch

Pattern: categories become corrupted.

Repair: standard encoding such as UTF-8 and validation.

Failure 20 — Software warning ignored

Pattern: non-converged model reported.

Repair: fail pipeline on critical warnings.

Failure 21 — Numerical tolerance unstated

Pattern: tiny floating differences called failure.

Repair: define scientifically meaningful tolerance.

Failure 22 — Floating-point instability

Pattern: small differences explode.

Repair: use stable algorithm and sensitivity checks.

Failure 23 — Hidden preprocessing

Pattern: data were manually normalised before script.

Repair: move all steps into pipeline.

Failure 24 — Missing intermediate file generation

Pattern: result depends on orphan file.

Repair: regenerate intermediates from raw inputs.

Failure 25 — Hard-coded threshold

Pattern: unexplained constant changes result.

Repair: configuration + rationale.

Failure 26 — Hidden exclusion list

Pattern: certain IDs removed manually.

Repair: version the exclusion rule.

Failure 27 — Different reference database

Pattern: genomic/proteomic IDs change.

Repair: archive database release.

Failure 28 — Data schema drift

Pattern: column meaning changes over time.

Repair: schema validation and versioning.

Failure 29 — Missing random split

Pattern: train/test partition cannot be reconstructed.

Repair: archive split indices or seed.

Failure 30 — Benchmark contamination

Pattern: test set influenced development.

Repair: fresh locked test set; this is validity plus reproducibility.

Failure 31 — Human-in-the-loop step undocumented

Pattern: analyst visually chooses threshold.

Repair: document decision rule and reviewer.

Failure 32 — Multiple versions of one figure

Pattern: manuscript contains manually edited plot.

Repair: generate final figure from code.

Failure 33 — Manuscript numbers typed manually

Pattern: text disagrees with table after update.

Repair: programmatically insert results or run consistency checks.

Failure 34 — Results folder contains stale outputs

Pattern: old figure accidentally submitted.

Repair: clean build directory before workflow runs.

Failure 35 — Hidden network dependency

Pattern: code fetches live web resource silently.

Repair: pin and archive resource.

Failure 36 — Secret credentials required

Pattern: pipeline cannot run for another researcher.

Repair: document credential acquisition and use environment variables.

Failure 37 — Source data legally restricted

Pattern: no public download.

Repair: controlled access and synthetic example data.

Failure 38 — Researcher left institution

Pattern: only person who knew workflow is gone.

Repair: institutional repositories and team documentation.

Failure 39 — Hardware no longer exists

Pattern: instrument/software obsolete.

Repair: preserve raw outputs and emulated environment where feasible.

Failure 40 — Result reproducible only by author

Pattern: undocumented tacit steps.

Repair: independent audit and complete documentation.

---

122. Reproducibility and AI-assisted science

AI tools introduce new reproducibility questions.

Prompt versioning

Record the exact prompt where scientifically relevant.

Model versioning

Record provider and model identifier.

Sampling parameters

Record temperature, top-p or equivalent settings.

Tool calls

If the model used external tools, record:

• tool version;
• input;
• output.

Non-determinism

Language models can produce different outputs for the same prompt.

Scientific use should analyse distributions or archive actual outputs.

Model drift

Hosted models can change without a new public name.

Archive date and response.

123. AI-generated code needs audit

Generated code can contain:

• subtle bugs;
• fabricated functions;
• wrong assumptions;
• security problems.

Human review and tests remain essential.

124. AI-generated analysis plans

An AI can suggest:

• models;
• transformations;
• visualisations.

Record which suggestions were adopted and why.

125. AI should not erase provenance

If an AI transforms data or code, preserve:

• original input;
• prompt;
• generated output;
• human modifications.

This keeps the research chain auditable.

---

126. The Reproducibility Notebook

A reproducibility notebook records enough technical and scientific context that another researcher can rebuild the analysis without guessing.

Notebook field 1 — Research question

State the scientific question and target outputs.

Notebook field 2 — Data source

Record provider, acquisition date, version, query and access conditions.

Notebook field 3 — Raw-data checksum

Record file checksums or repository identifiers so the exact inputs can be verified.

Notebook field 4 — Data dictionary

Define variables, units, coding and missing values.

Notebook field 5 — Cleaning pipeline

List the script or workflow that converts raw data into analytic data.

Notebook field 6 — Manual interventions

Record any unavoidable manual edits explicitly.

Notebook field 7 — Software environment

Record language, package, operating-system and system-library versions.

Notebook field 8 — Hardware

Record CPU/GPU or instrument hardware when output can depend materially on it.

Notebook field 9 — Randomness

Record seeds and stochastic settings.

Notebook field 10 — Analysis command

State the exact command or workflow entry point used to generate results.

Notebook field 11 — Primary output

Map every published table and figure to the script that creates it.

Notebook field 12 — Warnings

Archive convergence warnings, data issues and exceptions.

Notebook field 13 — Release tag

Tag the exact code version associated with submission or publication.

Notebook field 14 — Repository

Provide the persistent archive location.

Notebook field 15 — Access restrictions

Explain how another qualified researcher can access controlled data.

Notebook field 16 — License

State reuse permissions for code and data where applicable.

Notebook field 17 — Reproduction test

Record who reran the project from a clean environment and whether outputs matched.

Notebook field 18 — Known limitations

State hardware dependence, stochastic variation or unavailable proprietary components.

---

127. Reproducibility Audit: 30 questions

1. Can the exact source data be identified?
2. Is the source-data version known?
3. Are raw data preserved?
4. Is the cleaning process scripted?
5. Are manual edits logged?
6. Is there a data dictionary?
7. Are units explicit?
8. Are missing-value codes explicit?
9. Are time zones explicit?
10. Are data checksums available?
11. Is the code version archived?
12. Are dependencies pinned?
13. Is the operating system documented?
14. Is hardware documented where relevant?
15. Are random seeds recorded?
16. Can the workflow run from a clean environment?
17. Are notebooks executable top-to-bottom?
18. Are warnings captured?
19. Are figures generated by code?
20. Are table values generated by code?
21. Are manuscript numbers consistent with outputs?
22. Are external APIs versioned or archived?
23. Are proprietary dependencies disclosed?
24. Is restricted-data access documented?
25. Are licenses clear?
26. Is there a README?
27. Can a second researcher run the workflow?
28. Are expected numerical tolerances defined?
29. Are stochastic results tested across seeds?
30. Is the exact publication release preserved?

128. Clean-room reproduction test

One of the strongest checks is to give the project to someone who did not build it.

Ask them to start from:

• README;
• archived data;
• environment file;
• workflow command.

If they cannot reproduce the result without repeated private help, documentation is incomplete.

129. The author is a poor final reproducibility tester

Original authors carry hidden knowledge.

They know:

• which file is correct;
• which warning is harmless;
• which step was manual.

An independent reproducer exposes tacit assumptions.

130. Reproducibility badges and checklists

Some journals and conferences use:

• open-data badges;
• open-materials badges;
• artifact evaluation;
• reproducibility checklists.

Badges can encourage good practice, but they should represent actual accessibility and auditability rather than box-ticking.

---

131. Reproducibility Practice Bank: 30 reusable tasks

Practice 1 — Rebuild a figure

Start from raw data and regenerate one published figure without manual editing.

Practice 2 — Rebuild a table

Trace every number in one table to a script and source dataset.

Practice 3 — Remove local paths

Move the project to another folder and test whether it still runs.

Practice 4 — Fresh environment

Create a new virtual environment and install only declared dependencies.

Practice 5 — Pin versions

Record exact package versions and rerun.

Practice 6 — Seed test

Run the analysis under five seeds and inspect result stability.

Practice 7 — Restart notebook

Restart kernel and run every cell in order.

Practice 8 — Delete intermediates

Delete derived files and confirm the workflow rebuilds them from raw data.

Practice 9 — Schema test

Change one column name and confirm the pipeline fails clearly rather than silently.

Practice 10 — Unit test

Feed a known toy input to a scientific function and confirm the expected output.

Practice 11 — Missing-value test

Insert a missing value and verify the analysis handles it as documented.

Practice 12 — Locale test

Run under another locale and check dates/decimals.

Practice 13 — Operating-system test

Run on another supported OS or container.

Practice 14 — Package-update test

Upgrade one dependency and compare outputs.

Practice 15 — Numerical tolerance

Define a tolerance and test whether platform differences stay within it.

Practice 16 — Clean install

Give the project to a colleague who has never run it.

Practice 17 — Checksum verification

Verify downloaded data match the archived hash.

Practice 18 — Data provenance trace

Pick one result and trace it back to the raw measurement.

Practice 19 — Figure provenance trace

Pick one figure and identify every script, input and parameter involved.

Practice 20 — External API freeze

Archive API response and rerun analysis offline.

Practice 21 — Container build

Build the analysis environment from a container recipe.

Practice 22 — VM archive

For legacy software, boot an archived virtual machine and regenerate output.

Practice 23 — Dependency graph

Draw the pipeline from raw data to final manuscript.

Practice 24 — Warning audit

Run the pipeline and classify every warning.

Practice 25 — Manual-step hunt

Search the workflow for steps that require human clicking or copying.

Practice 26 — Rebuild after six months

Archive the project, wait, then attempt a fresh rebuild.

Practice 27 — Restricted-data audit

Ask whether another authorized researcher can obtain the same data legally.

Practice 28 — Reproducibility README

Write setup, data, run and expected-output instructions.

Practice 29 — Manuscript consistency

Automatically compare manuscript numbers with generated outputs.

Practice 30 — Full reproducibility cycle

Data snapshot → environment → clean build → output comparison → discrepancy diagnosis → archived release.

---

132. Reproducibility Maturity Model

Stage 0 — Personal analysis

Files live on one computer; method depends on memory.

Stage 1 — Scripted analysis

Major transformations are in code, but environment and data versions remain informal.

Stage 2 — Versioned project

Code, dependencies and data provenance are recorded.

Stage 3 — Clean rebuild

The project can regenerate results from a fresh environment.

Stage 4 — Independent reproduction

Another researcher can rebuild the result using shared artifacts.

Stage 5 — Archived reproducible object

Data/code/environment are persistently archived with identifiers and documentation.

Stage 6 — Maintained reproducibility

Automated tests, dependency monitoring and long-term preservation protect the result across time.

Maturity is not prestige. It is risk reduction.

---

133. Reproducibility by field: physics

Physics workflows may include:

• detector calibration;
• event reconstruction;
• simulation;
• statistical analysis.

Reproducibility requires versioning each layer.

134. Reproducibility by field: chemistry

Chemistry includes wet-lab details and computation.

Record:

• instrument method;
• reagent source;
• analysis software;
• spectral processing.

135. Reproducibility by field: biology

Biological data may be expensive or irreplaceable.

Archive:

• raw instrument outputs;
• sample metadata;
• reference database versions;
• analysis pipeline.

136. Reproducibility by field: medicine

Patient data are often restricted.

Reproducibility can use:

• controlled data enclaves;
• shared code;
• synthetic examples;
• data dictionaries.

137. Reproducibility by field: epidemiology

Registry-based analyses depend heavily on:

• cohort definitions;
• code lists;
• time windows;
• exclusion rules.

Phenotype algorithms should be archived.

138. Reproducibility by field: psychology

Share:

• stimuli;
• survey instruments;
• scoring code;
• analysis plan.

Participant data may need de-identification or controlled access.

139. Reproducibility by field: education

Education research may depend on:

• assessment scoring;
• school identifiers;
• hierarchical models;
• missing-data handling.

Student privacy must be protected.

140. Reproducibility by field: economics

Economic research often uses:

• administrative data;
• public surveys;
• custom code;
• complex cleaning.

Replication packages should include complete code and data-access instructions.

141. Reproducibility by field: ecology

Ecological studies may combine:

• field observations;
• remote sensing;
• GIS;
• climate data.

Spatial and temporal data versions matter.

142. Reproducibility by field: climate science

Climate workflows can depend on petabytes of data.

Full copying may be impractical.

Reproducibility can rely on:

• persistent datasets;
• versioned model code;
• workflow descriptions;
• derived-data archives.

143. Reproducibility by field: astronomy

Astronomy benefits from large public archives.

Still record:

• data release;
• calibration pipeline;
• catalog version;
• query.

144. Reproducibility by field: computer science

Computer-science results can depend on:

• hardware;
• compiler;
• dataset split;
• benchmark;
• seed.

Artifact evaluation is especially valuable.

145. Reproducibility by field: machine learning

ML requires separating:

• data reproducibility;
• training reproducibility;
• evaluation reproducibility;
• benchmark reproducibility.

Report distributions, not only best runs.

146. Reproducibility by field: qualitative research

Exact computational reproduction may not be the field’s central goal.

Auditability can still be improved through:

• transparent coding frameworks;
• decision trails;
• reflexivity;
• archived de-identified excerpts where ethical.

Use field-appropriate terminology.

---

147. Reproducibility and manuscript writing

A reproducible manuscript should minimize manual copying between analysis and prose.

Common failure:

table updated, text still contains old number.

Possible solution:

• dynamic documents;
• automated checks;
• programmatic table insertion.

148. Dynamic documents

Systems such as R Markdown, Quarto, Jupyter Book or other literate workflows can combine code and narrative.

The specific tool does not matter.

The benefit is reducing separation between computation and report.

149. Dynamic documents still need clean state

A dynamic document can fail if it reads:

• stale files;
• hidden cache;
• local environment variables.

Clean builds remain essential.

150. Caching

Caching can speed expensive analyses.

But stale caches can hide changes.

Record cache invalidation rules and allow a full clean rebuild.

151. Build artifacts

Figures and tables are build artifacts.

They should be regenerated from source rather than manually modified whenever practical.

152. Visual edits

If figures require manual annotation, document it or encode the annotation in code.

Manual image editing can alter data appearance and break provenance.

153. Image provenance

For scientific images, retain:

• raw image;
• processing steps;
• contrast settings;
• cropping;
• annotations.

154. Scientific image integrity

Image adjustments should not selectively alter evidence.

Reproducibility supports image-integrity audit.

155. Tables should be data products

A table pasted manually from a spreadsheet can drift.

Generate tables programmatically from the analytic dataset where possible.

156. Manuscript-level checks

Before publication:

• rebuild all outputs;
• check manuscript values;
• verify data/code links;
• tag release;
• archive artifacts.

---

157. Twenty-five myths about reproducibility

Myth 1 — Reproducible means true

False. A perfectly reproducible analysis can reproduce a biased design, invalid measurement or wrong scientific assumption.

Myth 2 — Irreproducible means fraudulent

False. Missing dependencies, documentation gaps, data-version drift or software differences can cause failure without misconduct.

Myth 3 — Sharing code is enough

False. Code without data, environment, instructions or dependencies may be unusable.

Myth 4 — Sharing data is enough

False. Raw data alone do not reveal preprocessing, exclusions or model settings.

Myth 5 — Open science and reproducibility are identical

False. Openness helps, but controlled-access data can still support reproducible work.

Myth 6 — Reproducibility means exact bitwise identity

Not always. Numerical and stochastic systems can require scientifically defined tolerances.

Myth 7 — Same random seed guarantees same result

False. Hardware, parallelism and library versions can still change outputs.

Myth 8 — Containers solve reproducibility forever

False. Container images themselves must be archived and documented.

Myth 9 — Notebooks are automatically reproducible

False. Hidden state and execution order can make notebooks fragile.

Myth 10 — Scripts are automatically reproducible

False. Scripts can contain hard-coded paths, undocumented assumptions and missing dependencies.

Myth 11 — The author can always reproduce their own work

False. Time, software drift and forgotten steps can make even self-reproduction difficult.

Myth 12 — A result reproducible by the author is enough

False. Independent reproduction is a stronger audit.

Myth 13 — Proprietary software makes reproducibility impossible

Not necessarily. Exact version, settings and archived environment can support reproducibility, though access barriers remain.

Myth 14 — Restricted data make reproducibility impossible

No. Controlled access can preserve privacy while allowing qualified researchers to rerun analysis.

Myth 15 — Reproducibility is only for computational science

Computational reproducibility is especially important because computation is widespread, but laboratory protocols, instrument settings and data provenance also matter.

Myth 16 — Reproducibility replaces replication

False. Reproducibility checks the same data/workflow; replication checks new data.

Myth 17 — Replication replaces reproducibility

False. A result can replicate while the original analysis remains unauditable.

Myth 18 — Reproducibility is clerical work

False. It is scientific quality control and often reveals substantive analytical errors.

Myth 19 — Version control is only for software engineers

False. Scientific claims depend on evolving code, data and methods.

Myth 20 — A README is optional

False for serious sharing. Without setup and workflow instructions, artifacts can be effectively unusable.

Myth 21 — Raw data should be edited to fix errors

Usually false. Preserve raw data and create corrected derivatives through documented transformations.

Myth 22 — Reproducibility is guaranteed if the paper lists software

False. Exact versions, parameters and workflows may still be missing.

Myth 23 — Numerical differences always mean scientific disagreement

False. Tiny floating-point differences can be scientifically irrelevant.

Myth 24 — Reproducibility can be added cheaply at the end

Usually false. Retrofitting documentation and provenance is expensive.

Myth 25 — Once reproducible, always reproducible

False. Digital environments decay. Long-term preservation requires maintenance.

---

158. Reproducibility Glossary

Artifact

A research object such as data, code, model, workflow, notebook or environment needed to reconstruct a result.

Bitwise reproducibility

Exact binary identity of computational output across runs.

Build artifact

An output generated from source inputs, such as a figure, table or compiled report.

Checksum

A digital fingerprint used to verify file identity and detect changes.

Clean build

Running a workflow from a fresh environment without relying on stale intermediate outputs or hidden state.

Code provenance

The history and version lineage of code used to produce a result.

Computational environment

The combination of operating system, software, libraries, configuration and hardware relevant to an analysis.

Computational reproducibility

Obtaining consistent computational results using the same input data, code, methods and analysis conditions.

Container

A packaged software environment containing applications and dependencies designed for portable execution.

Data dictionary

Documentation defining variables, units, codes, valid values and missingness conventions.

Data lineage

The trace of data from original source through transformations to final output.

Data provenance

Information about the origin, history and processing of data.

Dependency

A software package, library, service or resource required by code.

Dependency pinning

Recording exact dependency versions to recreate an environment.

Determinism

The property that the same inputs and environment produce the same output.

Environment drift

Change in software, hardware or dependencies that can alter computational behaviour over time.

Infrastructure as code

Machine-readable configuration that defines computational infrastructure reproducibly.

Literate programming

A workflow combining code, results and explanatory narrative in one document or linked system.

Lockfile

A file recording exact dependency versions and sometimes hashes.

Metadata

Data describing data: source, units, timing, instrument, context or structure.

Persistent identifier

A stable identifier such as a DOI used to locate a research artifact over time.

Provenance

The documented origin and transformation history of a research object.

Random seed

An initial value controlling a pseudo-random number generator to reproduce a stochastic sequence.

README

A human-readable guide explaining project purpose, setup, data, execution and expected outputs.

Replicability

Under the convention used here, consistent results across new studies that collect new data to answer the same scientific question.

Repository

A managed location for storing, versioning and sharing research artifacts.

Reproducibility

Under the National Academies convention used here, obtaining consistent results using the same input data, computational steps, methods, code and conditions of analysis.

Research compendium

A bundled collection of data, code, documentation and other artifacts supporting a research result.

Schema

A formal description of data structure, types and constraints.

Software environment

The specific versions and configuration of programs and libraries used in analysis.

Source data

The original data used as inputs before analysis transformations.

Stochastic computation

Computation containing randomness such that individual runs may differ.

Version control

A system recording changes to files over time.

Virtual machine

An emulated computer environment including a full operating system.

Workflow

The ordered set of computational steps linking inputs to outputs.

Workflow manager

Software that encodes dependencies and execution order among computational tasks.

---

159. Advanced FAQ: Scientific Reproducibility

1. Why reproduce an analysis if the authors already ran it?

Because independent rerunning can detect undocumented steps, coding errors, environment dependence and mismatches between reported and actual methods.

2. Is reproducibility a minimum standard?

For computation-heavy results, it is a basic auditability standard. It does not establish scientific validity on its own.

3. Can a wrong result be reproducible?

Yes. A wrong formula can reproduce perfectly.

4. Can a correct result be irreproducible?

Yes. The scientific conclusion may be true, but poor documentation can prevent reconstruction.

5. Does reproducibility require public data?

No. Controlled-access data can support reproduction by authorized researchers.

6. Does reproducibility require open-source software?

No, but closed software limits auditability and long-term access.

7. What should be shared when data cannot be shared?

Share code, data dictionaries, synthetic examples, access instructions and environment details.

8. Should raw data always be public?

No. Privacy, consent, intellectual property and community governance can restrict release.

9. What is the difference between reproducibility and repeatability?

Terminology varies. In the convention used here, reproducibility refers to same-data computational reconstruction. Repeatability often refers to repeated measurement under similar conditions.

10. Why does terminology vary?

Different scientific and engineering communities developed conventions independently. This is why definitions should be stated explicitly.

11. Can a notebook alone be a reproducibility package?

Only if it also has accessible data, a defined environment and reliable clean-state execution.

12. Should we share intermediate data?

Share or regenerate them. If generation is expensive, archiving can save time.

13. Should generated figures be stored?

They can be archived, but the authoritative route should regenerate them from source.

14. What if the analysis takes weeks of supercomputer time?

Archive derived outputs, checkpoints, environment and a smaller verification test where full rerun is impractical.

15. What if the raw data are petabytes?

Use persistent institutional archives, dataset identifiers, exact queries and derived-data snapshots.

16. What if a cloud service disappears?

Archive critical artifacts and environment recipes before dependence becomes permanent.

17. What if the software is proprietary?

Record exact version, settings and license path; export interoperable intermediate data where possible.

18. What if the software licence expires?

Long-term reproducibility is threatened. Consider containers/VMs if licensing permits or open alternative implementations.

19. Does Docker guarantee exact reproduction?

No. Hardware, external data and container availability can still differ.

20. What is a reproducibility tolerance?

A pre-defined acceptable numerical difference when exact bitwise output is not scientifically necessary or technically possible.

21. How do I choose a tolerance?

Base it on scientific precision and numerical stability, not convenience.

22. What if two computers produce results differing in the tenth decimal?

If the scientific conclusion and meaningful digits agree, the result may be reproducible within tolerance.

23. What if two computers produce different p-values around .05?

The analysis is fragile. Report effect estimates and investigate numerical/model sensitivity rather than relying on the threshold.

24. Can random-seed variation change scientific conclusions?

Yes. If so, report the distribution across seeds and improve stability.

25. What if only one seed works well?

The result is not robust. Reproducibility of that seed does not establish model reliability.

26. Should AI prompts be archived?

If AI output materially affects the scientific result, prompt, model version, parameters and output should be recorded where permitted.

27. Can hosted AI models be reproducible?

They can be partially reproducible if versions and outputs are archived, but provider-side changes can limit long-term reconstruction.

28. Is code review part of reproducibility?

It supports reproducibility and software correctness, though rerunning the pipeline is still necessary.

29. Is automated testing part of reproducibility?

Yes. Tests detect changes that would silently alter results.

30. What is the final reproducibility question?

Can another competent researcher take the documented artifacts and reconstruct a scientifically consistent result without relying on hidden knowledge?

---

160. Publication Artifact Package

Before a computational paper is published, prepare a release containing:

1. README;
2. data access instructions;
3. data dictionary;
4. code;
5. environment specification;
6. workflow command;
7. seed information;
8. expected outputs;
9. license;
10. citation information;
11. persistent archive identifier.

161. README structure

A useful README can include:

Purpose

What scientific result does this project reproduce?

Requirements

What hardware/software is needed?

Data

Where do inputs come from?

Setup

How is the environment installed?

Run

What command rebuilds the analysis?

Outputs

Which files should appear?

Expected differences

Are small stochastic/numerical differences normal?

Contact

How can users report problems?

162. Artifact evaluation

Some conferences and journals ask independent reviewers to evaluate code/data artifacts.

Review can test:

• availability;
• functionality;
• reusability;
• reproducibility.

This is a powerful quality-control step.

163. Functional artifact versus reproducible artifact

Code that runs is functional.

Code that recreates the reported result is reproducible.

These are different standards.

164. Reusable artifact

A reusable artifact has:

• clear API;
• documentation;
• tests;
• licensing.

Reusability goes beyond reproducibility.

165. Publication archives should be immutable

Once published, preserve an immutable release.

Future bug fixes can create new releases.

Do not overwrite the historical version silently.

166. Corrections should point to new versions

If a bug changes results:

• issue correction;
• archive corrected code;
• preserve original;
• explain scientific impact.

---

167. Long-Term Preservation

Reproducibility today is easier than reproducibility in twenty years.

Long-term strategy should consider:

• format longevity;
• software obsolescence;
• storage integrity;
• repository funding.

168. Prefer open durable formats

Where possible, archive data in well-documented formats rather than obscure proprietary binaries only.

Examples include:

• CSV for simple tables;
• TIFF for scientific images;
• NetCDF/HDF for structured scientific arrays where appropriate;
• plain-text source code.

Format choice depends on field.

169. Preserve raw proprietary files too

If an instrument produces a proprietary raw format, keep it.

Also export open derivatives when possible.

The raw file may contain information lost in the derivative.

170. Emulation

Legacy environments can sometimes be preserved through:

• virtual machines;
• emulators;
• container archives.

171. Migration

Data can be migrated to newer formats over time.

Migration should preserve:

• content;
• metadata;
• checksums;
• conversion history.

172. Preservation metadata

Archives should record:

• file format;
• software;
• creation date;
• checksum;
• rights;
• dependencies.

173. Institutional responsibility

Long-term reproducibility should not depend on one researcher’s laptop or personal cloud account.

Institutions and repositories need preservation plans.

174. Research software citation

Software should be citable so maintainers receive academic credit.

Credit supports long-term maintenance.

175. Data citation

Datasets should be citable as research outputs.

This encourages curation and provenance.

---

176. The Result-Reconstruction Protocol

A reproducibility audit should begin with one concrete published result.

Choose:

• one figure;
• one table;
• one model coefficient;
• one summary statistic.

Then reconstruct it from the bottom up.

Step 1 — Identify the published target

Record exactly what must be reproduced.

Example:

“Figure 4, panel B, blue line.”

Step 2 — Find the generating code

There should be a clear mapping from output to script, function or notebook cell.

Step 3 — Find the input data

Identify:

• file;
• version;
• checksum;
• data dictionary.

Step 4 — Reconstruct preprocessing

Run:

• filtering;
• unit conversion;
• missing-data handling;
• feature construction.

Do not use a mystery cleaned file unless its provenance is known.

Step 5 — Reconstruct the environment

Install the declared software and dependencies.

Step 6 — Run from clean state

Delete cached intermediates and run the workflow from the beginning.

Step 7 — Compare outputs

Compare:

• numbers;
• figures;
• statistical conclusions;
• logs.

Step 8 — Classify differences

Differences can be:

• bitwise;
• numerically tiny;
• scientifically meaningful;
• qualitatively contradictory.

Step 9 — Diagnose the discrepancy

Trace backward through:

• code;
• environment;
• data;
• documentation.

Step 10 — Archive the reproducible release

Once reconstructed, preserve the tested artifact package.

---

177. Reproduction Difference Categories

Category A — Exact match

Outputs are bitwise identical or textually identical where exact identity is expected.

Category B — Numerically equivalent

Small floating-point differences occur, but scientifically meaningful digits agree.

Category C — Stochastically equivalent

Individual outputs differ, but distributions and summary metrics agree within expected random variation.

Category D — Analytically equivalent

Different implementation produces the same scientific estimate within tolerance.

Category E — Partially reproducible

Some outputs match while others cannot be reconstructed.

Category F — Non-reproducible

The published result cannot be recovered from the stated data/method/environment.

Category G — Unverifiable

Required data, code or environment are unavailable, so reproducibility cannot be assessed.

“Unverifiable” is different from “wrong”.

178. Exact match is not always necessary

Examples where exact match may be unnecessary:

• Monte Carlo simulation;
• GPU training;
• parallel floating-point reduction;
• chaotic dynamical simulations.

Define a scientifically meaningful reproducibility criterion.

179. Exact match can be necessary

Examples:

• deterministic summary statistics;
• data cleaning row counts;
• reported sample size;
• categorical exclusions.

If sample size changes unexpectedly, the discrepancy matters.

180. Scientific tolerance versus engineering tolerance

Scientific tolerance asks:

“Would the difference change the interpretation?”

Engineering tolerance can ask:

“Does the system remain within operational limits?”

Both should be explicit.

---

181. Reproducibility and data cleaning

Data cleaning is often the least visible and most consequential stage.

Common decisions include:

• duplicate removal;
• outlier rules;
• date windows;
• unit conversions;
• category recoding;
• missingness.

Each decision can change the result.

182. Never overwrite raw identifiers silently

If identifiers are corrected or merged, preserve a mapping table.

183. Outlier handling

Outlier decisions should be:

• predefined where possible;
• coded;
• reported;
• sensitivity-tested.

Manual deletion after plotting is difficult to reproduce.

184. Missing-data handling

Document whether missing data are:

• dropped;
• imputed;
• modelled;
• coded as category.

Different methods can change estimates substantially.

185. Multiple imputation

Reproducibility requires:

• imputation model;
• variables included;
• number of imputations;
• seed;
• software version.

186. Derived variables

Variables such as BMI, risk scores or composite indices should be generated from code and documented formulas.

187. Clinical code lists

Epidemiological studies often define diseases using code lists.

Archive the exact code set and vocabulary version.

188. Natural-language labels

Category labels can be renamed over time.

Reproducibility benefits from stable codes behind human-readable labels.

189. Data filtering order

Filter order can matter when operations interact.

Code should make the sequence explicit.

190. Analytic sample flow

A reproducible paper should be able to report:

raw N → exclusions → missingness → final analytic N

Flow diagrams should derive from the actual pipeline where possible.

---

191. Reproducibility and statistical analysis

Model formula

Record the exact mathematical/statistical formula.

Reference categories

Reference levels can change coefficients.

Contrast coding

Categorical variable coding should be explicit.

Weights

Survey, sampling or inverse-probability weights should be archived.

Clustered standard errors

Record cluster variable and method.

Robust standard errors

Different implementations exist; specify the version.

Optimiser

Record optimiser and convergence criteria.

Initial values

Complex models can converge to different local optima from different starts.

Model convergence

Do not report unconverged estimates as final without explanation.

Multiple testing correction

Record method and family of tests.

Confidence intervals

Specify:

• Wald;
• profile likelihood;
• bootstrap;
• Bayesian credible interval.

192. Statistical defaults are not scientific constants

Software can default to:

• different degrees of freedom;
• different variance estimators;
• different optimisation routines.

Do not rely on undocumented defaults for important results.

193. Reproducibility and bootstrap

Record:

• seed;
• number of resamples;
• resampling unit;
• confidence-interval method.

194. Reproducibility and permutation tests

Record:

• permutation scheme;
• number of permutations;
• seed;
• exchangeability assumptions.

195. Reproducibility and Bayesian analysis

Record:

• priors;
• sampler;
• chains;
• iterations;
• warmup;
• seed;
• convergence diagnostics.

196. Priors are part of the method

“We used a Bayesian model” is incomplete without prior specification.

197. Reproducibility and model selection

Record whether model choice was:

• pre-specified;
• AIC/BIC based;
• cross-validated;
• stepwise;
• manual.

198. Model-selection uncertainty

If many models were tried and one chosen, reproducibility of the final model does not represent the uncertainty of the selection process.

Archive the candidate set or selection rule.

---

199. Reproducibility and visualization

Figures are part of the scientific argument.

Record:

• data subset;
• transformations;
• smoothing;
• axis limits;
• colour scale;
• aggregation.

200. Axis limits can alter interpretation

A truncated y-axis can exaggerate differences.

Reproducible figure code makes such choices visible.

201. Smoothing

Loess, moving averages and kernel smoothers depend on parameters.

Record them.

202. Binning

Histogram bin width and boundaries can change apparent patterns.

Use documented rules.

203. Colour maps

Colour choices can affect perception and accessibility.

Reproducibility includes the mapping from values to colours.

204. Image cropping

Cropping should not hide relevant evidence.

Preserve full raw image and crop coordinates.

205. Image contrast

Scientific image adjustments should be applied consistently and documented.

206. Composite figures

Record:

• which source image created each panel;
• what processing occurred;
• how panels were assembled.

---

207. Reproducibility in Primary and Secondary Science

School Science can teach reproducibility without requiring programming.

Primary 3 — Record what you did

Students write:

• materials;
• steps;
• observations.

Another student should be able to understand the procedure.

Primary 4 — Use enough detail for another group

Record:

• amounts;
• timing;
• measurement units.

The method becomes executable, not just descriptive.

Primary 5 — Distinguish repeatability and reproduction

Students can learn that repeating their own measurement differs from another group following their recorded method.

Primary 6 — Audit the method

Ask:

• Could another group repeat these steps?
• Which detail is missing?
• Would they get comparable measurements?

Secondary — Data and spreadsheet reproducibility

Students can:

• save raw data;
• record formulas;
• label units;
• keep analysis steps.

JC — Reproducible practical work

Students can submit:

• raw data;
• processing steps;
• graphs;
• calculation formulas.

The skill is scientific transparency.

208. Reproducibility as communication

A method section is a message to a future researcher.

The question is:

Have we communicated enough for someone else to reconstruct what we actually did?

Good reproducibility is therefore part of scientific writing.

---

209. Reproducibility Governance

Institutions can support reproducibility through:

• data management plans;
• software support;
• repositories;
• training;
• research software engineers;
• artifact review.

210. Data management plans

A plan can specify:

• storage;
• backup;
• sharing;
• privacy;
• retention;
• preservation.

211. Reproducibility training

Researchers should learn:

• version control;
• workflow automation;
• testing;
• data provenance;
• documentation.

These are scientific skills, not optional technical extras.

212. Research software engineers

Research software engineers bring software-engineering expertise into science.

They can improve:

• testing;
• architecture;
• performance;
• documentation;
• maintenance.

213. Core facilities

Institutions can maintain shared:

• compute environments;
• data archives;
• containers;
• instrument pipelines.

214. Journal policies

Journals can require:

• data availability statement;
• code availability statement;
• reporting checklist;
• artifact review.

215. Funding agencies

Funding bodies can support reproducibility by:

• requiring rigor plans;
• funding software maintenance;
• supporting data repositories;
• valuing replication and reproducibility work.

216. NIH current direction

NIH’s current reproducibility guidance emphasises rigorous, transparent research design, methodology, analysis, interpretation and reporting. Its 2026 replication/reproducibility initiative also describes building sustained infrastructure for replication and reproducibility as a scientific discipline. NIH rigor and reproducibility guidance and NIH replication and reproducibility initiative provide the current institutional reference.

217. Reproducibility incentives

Researchers need credit for:

• datasets;
• software;
• workflows;
• documentation;
• artifact evaluation.

Otherwise the labour remains invisible.

218. Maintenance funding

A reproducible pipeline can become unreproducible if no one maintains dependencies or archives.

Maintenance is part of the research lifecycle.

---

219. Reproducibility and scientific integrity

Reproducibility can reveal honest mistakes.

Examples:

• wrong sign;
• mislabelled group;
• incorrect denominator;
• duplicated data.

Finding and correcting these mistakes is scientific integrity.

220. Reproducibility is not misconduct detection alone

Most reproducibility problems are not fraud.

They often arise from:

• complex workflows;
• poor documentation;
• software drift;
• human error.

221. Corrections strengthen science

If reproduction reveals a material error:

• correct the paper;
• update the archive;
• preserve the history;
• state whether conclusions changed.

222. Reproducibility and peer review

Traditional peer review often does not rerun code.

Artifact review adds a technical layer.

Both can complement one another.

223. Reproducibility before peer review

Authors should not rely on reviewers to discover missing artifacts.

Run an internal clean-room audit first.

---

224. Reproducibility, Validity, Robustness and Replication: four different questions

Scientific reliability is multidimensional.

Dimension	Main question
Reproducibility	Can the same data/method/workflow reconstruct the result?
Robustness	Does the conclusion survive reasonable analytic choices?
Replicability	Does the result survive new data?
Validity	Does the study actually support the scientific claim?

A study can score differently on each dimension.

225. Reproducible but invalid

Example:

A dataset contains a biased measure.

The analysis reproduces perfectly.

The scientific conclusion is still invalid.

226. Valid but hard to reproduce

Example:

The scientific result may be correct, but the original analysis depends on undocumented software.

The evidence can still be valid while the research object is poorly auditable.

227. Robust but irreproducible

One dataset can show the same conclusion under many models, yet the exact computational workflow may be unavailable.

Robustness does not replace reproducibility.

228. Reproducible but non-replicable

The original code/data reconstruct perfectly, but new studies fail.

This suggests the original result may have been sample-specific or context-bound.

229. Replicable but not reproducible

New studies find the same phenomenon, but the original paper’s code is lost.

The scientific phenomenon may be real even though the historical analysis is not reconstructable.

230. Mature science needs all four

The strongest result is:

• computationally reproducible;
• analytically robust;
• empirically replicable;
• scientifically valid.

---

231. Reproducibility Decision Manual

Decision 1 — Do we need exact bitwise reproduction?

Use exact identity when computation is deterministic and outputs are discrete or exact.

Decision 2 — Do we need numerical tolerance?

Use tolerance when floating-point or hardware variation is scientifically irrelevant.

Decision 3 — Do we need stochastic equivalence?

Use distributional checks for Monte Carlo, simulation and machine learning.

Decision 4 — Should raw data be public?

Only when consent, privacy and licensing permit.

Decision 5 — Should restricted data be archived?

Yes, with controlled access and clear request procedures.

Decision 6 — Is a container enough?

No. Also archive data, workflow and container recipe.

Decision 7 — Should we save intermediate outputs?

Save expensive or difficult intermediates, but preserve the ability to regenerate them.

Decision 8 — Should we freeze package versions?

Yes for the publication artifact. Future development can use newer versions separately.

Decision 9 — Should we share notebooks?

Yes when useful, but clean-state execution must be verified.

Decision 10 — Should we share scripts too?

For complex pipelines, scripts/workflows often provide more stable automation.

Decision 11 — Should we archive the database extract?

Yes when lawful and practical; otherwise archive the exact query and data version.

Decision 12 — Should we archive API responses?

Yes when external APIs are dynamic and terms permit.

Decision 13 — Should we record hardware?

When results can materially depend on hardware or accelerator behaviour.

Decision 14 — Should we record all warnings?

Critical analysis warnings should be preserved and resolved or explained.

Decision 15 — Should we fail on warnings?

For known critical warnings such as non-convergence, yes.

Decision 16 — Should we use a workflow manager?

When dependencies are complex enough that manual ordering becomes risky.

Decision 17 — Should we use continuous integration?

When the project evolves and automated checks can catch reproducibility breaks.

Decision 18 — Should we preserve a VM?

For legacy or proprietary environments where lighter environment capture is insufficient.

Decision 19 — Should we write tests?

If important scientific conclusions depend on code, yes.

Decision 20 — Who should test the package?

Someone other than the primary author, ideally from a clean environment.

---

232. Artifact Matrix

Artifact	Purpose	Minimum metadata
Raw data	source evidence	version, source, checksum
Processed data	analysis input	generating script, source version
Code	transform/analysis	version, language, license
Environment	software context	dependency lock, OS
Workflow	execution order	entry command, dependencies
Notebook	narrative analysis	clean-state execution
Figure	visual result	source script/data
Table	numeric result	source script/data
Model	trained/fitted object	parameters, seed, code version
README	human instructions	setup, run, outputs

233. Artifact completeness

A project can fail reproducibility because one artifact is missing.

Examples:

• code exists, data do not;
• data exist, environment does not;
• environment exists, run order does not.

Think in systems.

234. Minimal reproducible package

For a simple deterministic analysis, a minimal package may be:

• data;
• script;
• dependency list;
• README.

235. Complex reproducible package

For a large project, add:

• workflow manager;
• containers;
• tests;
• checkpoints;
• hardware metadata;
• access scripts.

236. Artifact quality should match claim importance

A high-stakes medical model deserves a stronger reproducibility package than a preliminary exploratory classroom exercise.

Standards can scale with risk.

---

237. Reproducibility Handover Standard

A project should survive the departure of its original analyst.

The handover should include:

• project map;
• data-access instructions;
• credentials procedure;
• environment setup;
• workflow entry point;
• known failures;
• contact information.

238. Bus factor

The “bus factor” asks how many people must become unavailable before the project can no longer be maintained.

A bus factor of one is a reproducibility risk.

239. Team redundancy

At least two people should understand critical workflows in high-value projects.

240. Handover rehearsal

Before a student or staff member leaves, have another team member:

• install environment;
• run workflow;
• reproduce main result.

241. Institutional ownership

Key artifacts should live in institutional or durable repositories, not only personal accounts.

242. Secrets management

Passwords and API keys should not be stored in public code.

Document how authorized users obtain credentials.

243. Reproducibility without exposing secrets

Use:

• environment variables;
• secret managers;
• template configuration files.

244. Key rotation

Credentials change over time.

The project should not depend on one permanent secret.

---

245. Reproducibility Stop Rule

When is a project reproducible enough?

A practical stop rule is met when:

• required artifacts are archived;
• a clean environment can rebuild outputs;
• an independent researcher can follow instructions;
• expected numerical differences are defined;
• privacy/access constraints are documented.

246. Reproducibility Reopen Rule

Reaudit when:

• dependencies change;
• data source updates;
• new hardware is used;
• paper is corrected;
• repository moves;
• external service changes.

247. Reproducibility maintenance schedule

Long-lived projects can rerun a clean build:

• at publication;
• after major dependency updates;
• annually for critical infrastructure;
• before important reuse.

248. Reproducibility health check

Check:

• links work;
• containers build;
• dependencies install;
• data remain accessible;
• workflow still completes.

249. Reproducibility retirement

A computational environment can be retired if:

• superseded by a maintained version;
• security risks prevent continued use;
• artifacts are migrated successfully.

Preserve the historical release.

---

250. Reproducibility and Peer Review

Peer review traditionally evaluates:

• question;
• design;
• analysis;
• interpretation.

Artifact review evaluates:

• availability;
• functionality;
• reconstruction.

Both are valuable.

251. Reviewer time is limited

Not every reviewer can rebuild a complex analysis.

Specialist artifact reviewers can reduce burden.

252. Automated reproducibility reports

A project can automatically generate:

• dependency list;
• session information;
• seed;
• data hashes;
• test status.

This helps reviewers assess completeness quickly.

253. Reproducibility check before submission

Run:

1. clean checkout;
2. fresh environment;
3. data verification;
4. full workflow;
5. output comparison;
6. archive release.

This should be part of submission workflow.

---

254. Reproducibility and Scientific Education

Students should learn that a scientific result is not only an answer.

It is also a record of how the answer was produced.

Primary level

Record materials, units, steps and observations clearly.

Secondary level

Preserve raw data and calculations.

JC level

Use spreadsheets or code with visible formulas and versioned datasets.

University level

Use version control, environment capture and reproducible reports.

255. School reproducibility challenge

Student A writes an experimental method.

Student B, without speaking to A, follows it.

Then compare:

• what was clear;
• what was missing;
• which details mattered.

This is an age-appropriate clean-room reproducibility test.

256. Spreadsheet provenance exercise

Give students raw data and ask them to produce a graph.

Then require:

• raw sheet preserved;
• formulas visible;
• graph linked to data;
• units labelled.

257. Reproducibility teaches responsibility

Students learn:

• do not hide mistakes;
• preserve evidence;
• show calculations;
• let others check.

This is scientific integrity in practical form.

---

258. Reproducibility and Automated Laboratories

Robotic laboratories can improve reproducibility by logging:

• volumes;
• timing;
• temperature;
• instrument commands;
• sample IDs.

But automation can replicate bugs perfectly.

259. Machine-readable protocols

Protocols encoded in machine-readable form can reduce ambiguity.

They can specify:

• steps;
• quantities;
• branching;
• instrument commands.

260. Laboratory information management systems

LIMS can track:

• samples;
• reagents;
• instrument runs;
• results;
• provenance.

Good configuration supports reproducibility.

261. Electronic laboratory notebooks

ELNs can preserve:

• protocol versions;
• attachments;
• timestamps;
• notes.

Export and long-term accessibility should be considered.

262. Reagent provenance

For experiments, record:

• supplier;
• catalog number;
• lot number;
• expiry;
• storage.

Wet-lab reproducibility intersects with computational reproducibility at the data-generation step.

263. Instrument calibration provenance

A reproducible analysis is only as good as the measurement system that created the data.

Preserve calibration records.

---

264. Reproducibility and Model Cards

Machine-learning artifacts can include model cards documenting:

• training data;
• architecture;
• hyperparameters;
• intended use;
• limitations;
• evaluation.

This supports reconstruction and responsible reuse.

265. Dataset documentation

Datasheets or data cards can document:

• collection;
• population;
• consent;
• known biases;
• licensing.

266. Benchmark documentation

Record:

• exact test set;
• evaluation script;
• metric implementation;
• preprocessing.

267. Model checkpoints

Archive:

• weights;
• architecture;
• tokenizer;
• configuration.

A checkpoint without preprocessing can still be unusable.

268. Training-data disclosure

Where possible and lawful, document:

• source datasets;
• version;
• filtering;
• deduplication.

If data cannot be disclosed, state the limitation.

269. Reproducibility of evaluation versus training

Sometimes training is too costly to reproduce fully.

Evaluation can still be reproducible using archived checkpoints and test sets.

Be explicit about which layer is reproducible.

270. Reproducibility statements should be scoped

Instead of:

“Our model is reproducible.”

say:

• training reproducible within hardware tolerance;
• evaluation exactly reproducible;
• data processing reproducible;
• proprietary data unavailable.

Scope makes claims honest.

---

271. Advanced Reconstruction Casebook

Case 31 — The missing denominator

A published rate cannot be reproduced.

The numerator matches.

The denominator does not.

Possible causes:

• different eligibility window;
• duplicate removal;
• missing-data exclusion;
• population denominator update.

Reproducibility audit should trace denominator construction explicitly.

Case 32 — The invisible unit conversion

An analyst converted milligrams to grams manually before importing data.

No script records the change.

Another researcher uses the raw values and obtains a 1,000-fold difference.

Lesson:

unit conversions belong in code and metadata.

Case 33 — The moving time window

A study reports events “within one month”.

Original code used 30 days.

Reproducer uses calendar months.

Counts differ.

Temporal definitions must be operational, not colloquial.

Case 34 — The hidden timezone

Events near midnight are assigned to different dates.

Original timestamps were local time; reproducer assumes UTC.

Lesson:

timezone is part of the data-generating method.

Case 35 — The category recode

A survey category “Prefer not to answer” was manually merged with missing values.

The decision was not documented.

Another analyst retains the category and gets a different regression.

Recode rules should be explicit.

Case 36 — The silently dropped rows

Statistical software automatically drops observations with any missing model variable.

The manuscript reports the original sample size instead of the model sample size.

Reproduction reveals the mismatch.

Case 37 — The changing reference category

A categorical predictor uses “Control” as reference in one environment and alphabetically first category in another.

Coefficient signs appear different.

The model is mathematically equivalent but presentation changed.

Explicit reference categories prevent confusion.

Case 38 — The optimizer switch

A package update changes the default optimiser.

The model converges to a different local optimum.

Record optimizer and initial values.

Case 39 — The convergence warning

Original paper reports parameter estimate.

Reproducer sees:

“Model failed to converge.”

The number may exist but not be trustworthy.

Reproducibility includes warnings, not only values.

Case 40 — The bootstrap mismatch

Two analysts both use 1,000 bootstrap samples but different resampling units.

One resamples individuals.

Another resamples clusters.

Uncertainty differs materially.

The resampling unit is part of the method.

Case 41 — The train/test leak

A machine-learning pipeline standardises features using the full dataset before splitting.

The code reproduces perfectly.

The reported accuracy is biased.

This is reproducible invalidity.

Case 42 — The undocumented augmentation

An image model used random crop and colour augmentation.

Replication code omits it.

Training performance changes.

Data augmentation is part of the method.

Case 43 — The changing tokenizer

A language model evaluation uses a tokenizer whose version changes.

Token counts and truncation change.

Archive tokenizer version.

Case 44 — The hidden caching layer

A notebook appears to rerun but actually loads an old precomputed object.

Delete cache and rebuild.

Case 45 — The stale figure

Analysis changes, but the manuscript includes a manually saved old plot.

Dynamic generation prevents stale artifacts.

Case 46 — The inaccessible random split

The paper says “80/20 split”.

No split indices or seed are saved.

Another split yields a different model score.

Archive exact split.

Case 47 — The model checkpoint mismatch

Evaluation script loads “best_model.pt”.

Several files with that name exist.

Use content hashes and immutable identifiers.

Case 48 — The floating database

Clinical database continuously receives corrections.

Reproducer queries the database one year later and gets different records.

Archive the analytic snapshot or data release.

Case 49 — The software licence disappears

A proprietary package is discontinued.

Scientific archive contains only project file, not export.

Preserve interoperable outputs while software is available.

Case 50 — The interactive dashboard

A paper reports numbers seen on an online dashboard.

The dashboard later updates.

Archive downloaded source data and dashboard state.

---

272. Reproducibility Risk Matrix

Project feature	Main risk	Protection
manual spreadsheets	hidden edits	script or edit log
dynamic database	input drift	snapshot + query
many dependencies	environment drift	lockfile/container
GPU training	nondeterminism	seed distribution + environment
web APIs	provider change	version + archived responses
restricted data	access barrier	controlled-access instructions
legacy instrument	software obsolescence	raw export + VM
notebooks	hidden state	restart/run-all test
long pipeline	manual ordering errors	workflow manager
high-stakes result	undetected code error	independent implementation/code review

273. Risk-based reproducibility

Not every study needs the same infrastructure.

A low-stakes descriptive analysis may need:

• script;
• data;
• README.

A high-stakes clinical algorithm may need:

• tests;
• container;
• model archive;
• independent reproduction;
• security review;
• long-term preservation.

Reproducibility effort should scale with scientific consequence.

---

274. Reproducibility and Robustness Workflow

After reproducing the exact analysis, test robustness.

Step 1 — Exact reconstruction

Can the original result be regenerated?

Step 2 — Reasonable alternate preprocessing

Does the conclusion survive defensible cleaning choices?

Step 3 — Alternate model

Does the conclusion depend on one specification?

Step 4 — Sensitivity to thresholds

Do nearby thresholds change conclusions dramatically?

Step 5 — Subsample checks

Does one small subgroup drive the result?

Step 6 — Stochastic variation

Does the conclusion survive seeds?

Reproducibility comes first.

Robustness follows.

275. Reproducibility as the baseline for critique

It is difficult to debate scientific interpretation when the original result cannot be reconstructed.

A reproducible analysis gives critics and supporters a common starting point.

276. Reproducibility enables adversarial analysis

Researchers who disagree can:

• rerun the same baseline;
• change one assumption;
• compare outputs.

This makes scientific disagreement more precise.

---

277. Reproducibility and Pre-registration

Preregistration records what researchers planned before outcomes were known.

Reproducibility records how the analysis was actually implemented.

Both matter.

278. Plan versus implementation

A preregistration might say:

“Exclude participants who fail Criterion C.”

The code implements Criterion C incorrectly.

Reproducibility audit exposes the mismatch.

279. Deviations

If analysis deviates from preregistration:

• record deviation;
• give reason;
• label exploratory if appropriate.

Code should match the reported deviation.

280. Registered reports

Registered reports reduce result-based publication bias.

Reproducible artifacts make the accepted analysis auditable after completion.

---

281. Reproducibility and Data Quality

A perfectly reproducible pipeline can process poor data perfectly.

Data quality needs its own checks.

Accuracy

Does the recorded value reflect reality?

Completeness

Are expected records present?

Consistency

Do related fields agree?

Timeliness

Are data current enough for the question?

Validity

Do values obey defined constraints?

Uniqueness

Are duplicate records handled appropriately?

282. Data quality tests belong in pipelines

Automate checks such as:

• age >= 0;
• end date after start date;
• identifier unique;
• percentage between 0 and 100.

283. Source-data verification

When possible, compare digital records with:

• instrument logs;
• source documents;
• reference standards.

Reproducibility begins with trustworthy input.

---

284. Reproducibility and Scientific Images

Scientific images can contain substantial hidden processing.

A reproducible image workflow should record:

• raw image file;
• acquisition settings;
• cropping;
• contrast;
• filtering;
• segmentation;
• quantification.

285. Do not use screenshots as primary scientific data

When possible, preserve the underlying numeric or raw image data.

Screenshots are difficult to reanalyse.

286. Image-analysis masks

If segmentation produces masks, save them or the code that regenerates them.

287. Manual annotation

Human-labelled images should record:

• annotator;
• guidelines;
• blinding;
• adjudication.

288. Inter-rater reliability

If human coding affects results, report agreement and disagreement rules.

This is part of reproducible method.

---

289. Reproducibility and Databases

SQL queries

Archive exact queries.

Database schema

Record table/column definitions.

Data release

Record version or extraction timestamp.

Joins

Document keys and join type.

Deduplication

Record duplicate resolution.

Derived cohorts

Generate cohort membership from code.

290. Query reproducibility

Two analysts can write different SQL queries that look logically similar but produce different datasets.

Query code is part of the method.

291. Database permissions can alter views

Different users may see different records or tables.

Record access role where relevant.

292. Data warehouse transformations

An analytic table may already have undergone ETL transformations upstream.

Document those dependencies or reference a versioned data product.

---

293. Reproducibility and Simulations

Simulation reproducibility needs:

• initial conditions;
• parameters;
• random seed;
• solver;
• time step;
• stopping conditions.

294. Chaotic simulations

Exact trajectories can diverge.

Define reproducibility using:

• ensemble statistics;
• invariants;
• long-run distributions;
• scientific summary metrics.

295. Agent-based models

Record:

• agent rules;
• update ordering;
• network;
• random seed;
• initial population.

296. Simulation checkpoints

For expensive simulations, archive checkpoint states so others can verify later stages without rerunning everything.

297. Simulation provenance

Link each output to:

• parameter file;
• code commit;
• input data;
• hardware job.

---

298. Reproducibility and Real-Time Systems

Real-time systems use data that change continuously.

Examples:

• weather;
• markets;
• traffic;
• epidemics.

Archive input snapshots at decision time.

299. Streaming data

Record:

• stream offsets;
• window definitions;
• late-arriving data rules;
• processing version.

300. Live dashboards

Dashboard numbers are transient.

Archive the data and code behind published claims.

301. Forecast archives

For forecasting studies, preserve the actual forecast as issued, not only the later recomputed model.

This separates reproducibility from hindsight.

---

302. Reproducibility and Decision Systems

If a model influences real decisions, reproducibility should include:

• model version;
• input snapshot;
• decision threshold;
• post-processing;
• output shown to user.

303. Model registry

A registry can record:

• production model;
• version;
• training data;
• evaluation;
• deployment date.

304. Audit logs

For high-stakes systems, log which model made which decision with which inputs.

This supports later scientific and operational audit.

305. Reproducibility of a decision

The question becomes:

Given the archived state at that time, can we reconstruct why this decision was produced?

This is reproducibility applied to deployed science.

---

306. Reproducibility Result Classification

When a reproduction attempt finishes, classify the outcome carefully.

Class 1 — Exactly reproducible

The same inputs and environment regenerate identical outputs where exact identity is expected.

Class 2 — Numerically reproducible

Outputs differ slightly because of floating-point or hardware effects but agree within scientific tolerance.

Class 3 — Stochastically reproducible

Individual random outcomes differ but distributions, metrics and conclusions match expectation.

Class 4 — Reproducible after documented correction

The original package had a fixable error, and the corrected release reconstructs the intended result.

Class 5 — Partially reproducible

Some outputs can be reconstructed but others cannot.

Class 6 — Environment-limited

The analysis depends on unavailable software, hardware or service.

Class 7 — Data-limited

The required original data are unavailable or access cannot be reproduced.

Class 8 — Documentation-limited

Key processing or parameter choices are missing.

Class 9 — Scientifically inconsistent

The same data and stated method produce a materially different scientific result.

Class 10 — Unassessable

Too many essential artifacts are missing to judge.

This classification avoids calling every failure “irreproducible” for the same reason.

---

307. What to do when a result does not reproduce

Step 1 — Freeze the evidence

Preserve the original package, reproducer environment and discrepancy.

Step 2 — Compare inputs

Verify data hashes and versions.

Step 3 — Compare environments

Check software, libraries, OS and hardware.

Step 4 — Compare workflow

Check run order, parameters and caches.

Step 5 — Compare outputs stage by stage

Find the first point where values diverge.

Step 6 — Decide whether difference is scientific

Ignore meaningless decimal noise; investigate material differences.

Step 7 — Correct transparently

If an error is found, release a new version and state whether conclusions change.

308. First-divergence debugging

Do not compare only final figures.

Compare:

1. row counts;
2. summary statistics;
3. processed datasets;
4. model inputs;
5. model parameters;
6. final outputs.

The first divergence often reveals the root cause.

309. Binary search through a pipeline

For long workflows, compare checkpoints halfway through.

If they match, the problem is later.

If they differ, the problem is earlier.

This is a computational version of scientific narrowing.

310. Contacting original authors

When something does not reproduce:

• send exact error;
• show environment;
• show data checksum;
• ask a specific question.

This is more productive than “your paper doesn’t work”.

311. Author response

Original authors should provide:

• missing detail;
• correct artifact;
• known issue;
• correction if necessary.

Transparent response strengthens trust.

312. Correction threshold

A correction is appropriate when:

• published number is wrong;
• code contains a material bug;
• method description omitted a critical step.

State whether the main conclusions change.

313. Retraction threshold

Retraction may be appropriate when errors or misconduct make the results unreliable at a fundamental level.

Non-reproducibility by itself does not automatically imply retraction.

314. Version history should remain visible

Preserve:

• v1 original;
• v2 corrected;
• change log.

Scientific history should not be silently rewritten.

---

315. Reproducibility Scorecard

Dimension	0	1	2
Data	unavailable	partial/access unclear	versioned access
Code	missing	partial	complete versioned code
Environment	unknown	versions listed	lock/container
Workflow	manual	documented	automated clean build
Documentation	minimal	README	full data/method docs
Testing	none	some checks	automated tests/CI
Independent audit	none	internal	external clean-room reproduction
Archiving	personal	shared	persistent versioned archive

This scorecard is diagnostic, not a universal certification.

316. Avoid false precision in scoring

A project scoring 13 is not scientifically “better” than one scoring 12 in every respect.

Use the scorecard to find weak links.

---

317. Reproducibility Standards by Project Type

Simple statistical paper

• data;
• analysis script;
• dependency versions;
• README.

Large observational study

• cohort-building code;
• data dictionary;
• code lists;
• secure data access;
• analysis environment.

Machine-learning paper

• training/evaluation split;
• code;
• config;
• seeds;
• environment;
• checkpoints where feasible.

Simulation paper

• model code;
• initial conditions;
• parameters;
• solver;
• seed;
• summary reproducibility criteria.

Image-analysis paper

• raw images;
• acquisition metadata;
• processing code;
• segmentation rules.

API-dependent paper

• API version;
• date;
• requests;
• responses where permitted;
• fallback archive.

Restricted-data paper

• access process;
• synthetic data;
• code;
• secure environment instructions.

---

318. Reproducibility Verification Matrix

Question	Evidence
Same data?	checksum/version
Same code?	commit/tag
Same environment?	lockfile/container
Same parameters?	config/log
Same workflow?	workflow definition
Same output?	hash/tolerance comparison
Independent check?	audit report

319. Automated verification

A pipeline can automatically check:

• input hashes;
• test status;
• environment version;
• output hashes;
• expected files.

320. Reproducibility report

At the end of a run, generate a machine-readable or human-readable report containing:

• timestamp;
• commit;
• environment;
• input hashes;
• seed;
• test results;
• output hashes.

This is the computational equivalent of a lab log.

---

321. Reproducibility and Collaboration

Collaborative projects need shared conventions.

Branch strategy

Use a clear version-control workflow.

Code review

Review important changes before merging.

Data ownership

Define who can modify source data.

Release management

Tag versions used for figures and papers.

Issue tracking

Record bugs and methodological questions.

322. Collaboration can reduce hidden knowledge

When several people must understand the workflow, undocumented assumptions are exposed earlier.

323. Onboarding as reproducibility test

A new team member should be able to:

• clone project;
• obtain data;
• create environment;
• run analysis.

If onboarding requires oral folklore, documentation is incomplete.

---

324. Reproducibility and FAIR Principles

FAIR data principles emphasise that data should be:

• Findable;
• Accessible;
• Interoperable;
• Reusable.

FAIR and reproducibility overlap but are not identical.

A dataset can be FAIR yet not enough to reproduce a specific analysis.

325. Findable

Persistent identifiers and searchable metadata help others locate artifacts.

326. Accessible

Access protocol should be clear, including controlled access.

327. Interoperable

Open standards and metadata support use across systems.

328. Reusable

Licensing, provenance and documentation support future analysis.

329. Reproducibility needs executable context too

FAIR data alone do not capture code, environment and workflow.

Research compendia connect the pieces.

---

330. Reproducibility and Semantic Versioning

Software projects often use version numbers such as:

MAJOR.MINOR.PATCH

Scientific projects can adopt similar principles:

• major: analysis changes scientifically;
• minor: new feature/output;
• patch: bug fix without changed conclusion.

The exact scheme is optional.

The principle is explicit change history.

331. Data versioning

Datasets also need versions.

Document:

• added records;
• corrected values;
• changed definitions.

332. Model versioning

A model artifact should identify:

• training data version;
• code version;
• hyperparameters;
• checkpoint.

333. Manuscript versioning

Preprint, accepted version and corrected version may contain different results.

Link each to the corresponding artifact release.

---

334. Reproducibility scope and boundaries

This guide focuses on same-data, same-method computational reconstruction.

Related scientific-method topics are treated separately:

• Replication — new data testing the same scientific question.
• Evidence — how observations change confidence.
• Measurement — how signals become trustworthy quantities.
• Experiments — how tests are designed.
• Uncertainty — how ranges and unknowns are represented.
• Scientific communication — how findings are explained publicly.

The durable reproducibility question is:

Can another researcher reconstruct what we actually did from the artifacts we preserved?

335. Where this page sits in the eduKateYishun scientific-method system

• How Scientific Replication Works — new-data confirmation.
• How Scientific Measurement Works — data quality at acquisition.
• How Scientific Evidence Works — confidence from observations.
• How Scientific Experiments Work — test design.
• How Scientific Hypotheses Work — testable claims.
• How Scientific Causation Works — counterfactual intervention effects.
• How Scientific Theories Work — explanatory frameworks.

336. Authoritative further reading

• National Academies — Definitions of Reproducibility and Replicability
• National Academies — Reproducibility
• NIH — Enhancing Reproducibility through Rigor and Transparency
• NIH — Strengthening Replication and Reproducibility

---

337. Reproducibility Result Classification

When a reproduction attempt finishes, classify the outcome carefully.

Class 1 — Exactly reproducible

The same inputs and environment regenerate identical outputs where exact identity is expected.

Class 2 — Numerically reproducible

Outputs differ slightly because of floating-point or hardware effects but agree within scientific tolerance.

Class 3 — Stochastically reproducible

Individual random outputs differ, but distributions and summary metrics agree within expected variation.

Class 4 — Reproducible after documented correction

The original package had a fixable error and the corrected release reconstructs the intended result.

Class 5 — Partially reproducible

Some outputs can be reconstructed while others cannot.

Class 6 — Environment-limited

The analysis depends on unavailable software, hardware or service.

Class 7 — Data-limited

Required input data are unavailable or access cannot be reconstructed.

Class 8 — Documentation-limited

Key processing or parameter choices are missing.

Class 9 — Scientifically inconsistent

The same data and stated method produce a materially different scientific result.

Class 10 — Unassessable

Too many essential artifacts are missing to judge.

338. What to do when a result does not reproduce

1. Preserve the original package and the reproducer’s environment.
2. Compare data hashes and versions.
3. Compare software and dependencies.
4. Compare workflow order, parameters and caches.
5. Compare intermediate outputs stage by stage.
6. Identify the first material divergence.
7. Correct transparently and archive a new version.

The first-divergence method is efficient because it narrows the problem before the final output becomes confusing.

339. Reproducibility Scorecard

Dimension	Weak	Strong
Data	version unknown	versioned source + checksum
Code	partial or missing	complete tagged release
Environment	undocumented	lockfile/container
Workflow	manual	automated clean build
Documentation	minimal	README + data/method docs
Testing	none	automated tests
Audit	author only	independent clean-room rebuild
Archive	personal storage	persistent versioned repository

The scorecard is diagnostic rather than a universal ranking. Its value is locating the weakest reproducibility link.

340. Reproducibility Handover Standard

A project should survive the departure of its original analyst.

The handover should include:

• project map;
• data-access instructions;
• environment setup;
• workflow entry command;
• known limitations;
• release tag;
• contact path for access issues.

A high-value project should have more than one person who understands the critical workflow.

341. Secrets without hidden science

Passwords and API keys should not be committed to public code.

Use:

• environment variables;
• secret managers;
• template configuration files.

Document how authorized users obtain credentials without exposing them.

342. Reproducibility Stop Rule

A project is ready for publication when:

• required artifacts are archived;
• a clean environment can rebuild the main result;
• an independent person can follow the instructions;
• expected numerical differences are defined;
• privacy and access limits are documented.

343. Reproducibility Reopen Rule

Reaudit when:

• dependencies change;
• data source updates;
• hardware changes;
• paper is corrected;
• repository moves;
• external service changes.

344. Reproducibility and peer review

Traditional peer review asks whether the science is plausible and well designed.

Artifact review asks whether the computational record actually works.

The two forms of review complement one another.

345. Artifact-review questions

• Are the artifacts accessible?
• Do they install?
• Does the workflow run?
• Do the reported outputs regenerate?
• Are deviations documented?

346. Reproducibility and FAIR principles

FAIR data are Findable, Accessible, Interoperable and Reusable.

These principles overlap with reproducibility but do not replace it.

A dataset can be FAIR while the analysis code or environment is missing.

347. Reproducibility requires executable context

The full computational record connects:

findable data + executable code + captured environment + documented workflow.

348. Reproducibility and collaboration

Shared conventions improve scientific continuity.

Useful practices include:

• code review;
• issue tracking;
• release tagging;
• shared responsibility for critical pipelines.

349. Onboarding as an audit

A new team member should be able to:

• obtain the project;
• obtain or request the data;
• build the environment;
• run the main analysis.

If onboarding requires oral folklore, documentation is incomplete.

350. Reproducibility and deployed models

When models make real decisions, archive:

• model version;
• input snapshot;
• threshold;
• post-processing;
• output shown to the user.

The operational question becomes:

Can we reconstruct why this decision was produced at that time?

---

351. Clean-Room Rebuild Laboratory

The strongest practical test of computational reproducibility is a clean-room rebuild by someone who did not create the original analysis.

Stage 1 — Start with a clean environment

Use a new computer, fresh virtual machine or clean container host.

The reproducer should not inherit:

• old caches;
• global packages;
• hidden files;
• personal environment variables.

Stage 2 — Obtain the archived release

Use the exact tagged or archived publication version.

Verify the archive hash.

Stage 3 — Obtain or request the data

Follow the public or controlled-access route exactly as described.

Verify the data version and checksum.

Stage 4 — Build the declared environment

Install only the dependencies documented by the project.

If extra packages are required, the package specification was incomplete.

Stage 5 — Run tests before analysis

Run:

• unit tests;
• data-integrity tests;
• schema checks;
• toy examples.

A failed test should stop the pipeline.

Stage 6 — Run the main workflow

Use the documented entry command.

The reproducer should not have to infer script order from file names.

Stage 7 — Compare checkpoints

Compare:

• row counts;
• summary statistics;
• processed-data hashes;
• model-input dimensions.

Stage 8 — Compare final outputs

Compare:

• estimates;
• uncertainty intervals;
• figures;
• tables;
• model metrics.

Stage 9 — Apply the correct tolerance

Exact identity may be expected for deterministic outputs.

Numerical or stochastic tolerance may be appropriate for:

• floating-point computation;
• Monte Carlo;
• parallel systems;
• GPU training.

Stage 10 — Write the audit report

Record:

• auditor;
• date;
• environment;
• successes;
• differences;
• resolved issues;
• remaining limitations.

The audit report is evidence that the published computational result can actually be reconstructed.

---

352. Reproducibility Casebook: final reconstruction failures

Case A — Same code, different sample size

Possible causes:

• data version changed;
• missing values handled differently;
• duplicate removal changed;
• date window parsed differently.

Start upstream.

Case B — Same sample size, different model estimate

Inspect:

• reference category;
• weights;
• model formula;
• standard-error method;
• software default.

Case C — Same estimate, different significance

Often caused by different:

• standard errors;
• degrees of freedom;
• cluster correction;
• confidence-interval method.

The scientific estimate may reproduce even if threshold language changes.

Case D — Same table, different plot

Check:

• axis limits;
• transformation;
• smoothing;
• aggregation;
• colour mapping.

Case E — Same model, different machine-learning score

Check:

• split indices;
• seed;
• checkpoint;
• preprocessing;
• evaluation script.

Case F — Old software reproduces, new software does not

If the new package corrected a bug:

1. preserve historical result;
2. run corrected analysis;
3. state whether scientific conclusions change.

Case G — Hosted AI model changed

If the provider updated the model and no archived output exists, exact reproduction may be impossible.

This is why hosted-model studies should archive:

• date;
• model identifier;
• prompt;
• parameters;
• actual responses where permitted.

Case H — Restricted dataset no longer accessible

If access policy changes, the project can become practically unreproducible even if the code is perfect.

Durable access planning is part of the artifact package.

Case I — One output manually edited

Analysis reproduces, but the published figure contains manual labels or altered values.

Scientific image provenance is broken.

Final figures should derive from code or documented edit steps.

Case J — Author-only success

The author can run the project because they know three undocumented steps.

An independent reproducer cannot.

Classification:

not yet independently reproducible.

Repair the documentation, not the auditor.

---

353. Reproducibility and Scientific Scope

A reproducibility statement should name what is actually reconstructable.

Examples:

Strong scoped statement:

“All reported tables and figures can be regenerated from the archived analytic dataset using the supplied container and workflow.”

Weak over-broad statement:

“The study is fully reproducible.”

Why scope matters:

• raw instrument data may be unavailable;
• training may be too costly to repeat;
• one proprietary dependency may remain;
• some human coding may not be reconstructable exactly.

354. Layered reproducibility claims

A project can report:

• data-processing reproducibility;
• statistical-analysis reproducibility;
• figure reproducibility;
• model-evaluation reproducibility;
• training reproducibility.

Each layer can have a different status.

355. Reproducibility uncertainty

Reproducibility itself can contain uncertainty.

Examples:

• stochastic algorithms;
• chaotic simulation;
• hardware-dependent arithmetic.

Report the expected range of variation.

356. Reproducibility boundaries

A result may be reproducible only on:

• specific hardware;
• specific software;
• specific data release.

Boundary knowledge is useful.

It tells future researchers what must be preserved.

---

357. Reproducibility and Scientific Memory

Science is cumulative only if future researchers can recover what earlier researchers actually did.

Without preserved artifacts, a paper becomes a compressed narrative with missing machinery.

358. Papers are not complete computational records

A paper must compress.

It cannot contain:

• every line of code;
• every dependency;
• every raw file.

The publication therefore needs linked artifacts.

359. Research compendium as scientific memory

A research compendium can preserve:

• paper;
• data;
• code;
• environment;
• workflow;
• documentation.

It is the expanded memory behind the compressed paper.

360. Persistent identifiers connect the memory

DOIs and immutable release IDs link a publication to the exact artifacts used.

361. Citation preserves credit and discoverability

Data and software should be cited so:

• future users can locate them;
• maintainers receive credit;
• versions remain identifiable.

362. Reproducibility is intergenerational

A doctoral student may graduate.

A lab may close.

A software package may disappear.

The scientific result should not disappear with them.

---

363. Reproducibility Integrity Standard

A reproducible study should preserve the distinction between:

• what was planned;
• what was implemented;
• what was corrected;
• what was published.

364. Do not rewrite the historical analysis silently

If code changes after publication:

• tag new version;
• keep old version;
• write change log.

365. Do not hide failed clean builds

A failed rebuild is useful diagnostic information.

Fix it and document the issue.

366. Do not call unavailable artifacts “available on request” indefinitely

If a paper promises data/code on request, the request process should actually work.

Persistent repositories are generally more durable.

367. Do not expose private data for the sake of openness

Privacy and consent remain scientific obligations.

368. Do not treat reproducibility badges as the goal

The goal is a result that can be reconstructed.

The badge is only a signal.

---

369. Final Reproducibility Audit

Before the owner is complete, the reproducibility framework should answer all of these:

1. What exact result is being reconstructed?
2. Which exact data version was used?
3. Can those data be accessed or requested?
4. Is raw-to-analytic transformation documented?
5. Is the code version archived?
6. Are dependencies captured?
7. Are software and OS versions known?
8. Is hardware recorded where relevant?
9. Are random seeds and stochastic settings recorded?
10. Can the project run from a clean state?
11. Can intermediate files be regenerated?
12. Can figures and tables be regenerated?
13. Are warnings and convergence status visible?
14. Are external services versioned or archived?
15. Are restricted-data rules clear?
16. Are licenses and citation instructions present?
17. Can another researcher follow the README?
18. Are acceptable numerical tolerances defined?
19. Is the publication release immutable?
20. Has someone independent actually tried the rebuild?

370. Reproducibility acceptance standard

A serious computational result is ready for publication when:

another competent researcher can obtain the same input data, reconstruct the declared environment, run the documented workflow and recover a scientifically consistent result without relying on undocumented personal knowledge.

371. Reproducibility versus replication: final boundary

This owner stops at reconstruction.

The moment a researcher collects genuinely new data to ask whether the scientific finding holds again, the job moves to How Scientific Replication Works.

That separation keeps the estate collision-free.

372. Final source standard

The central definition follows the National Academies:

• Reproducibility and Replicability in Science — Definitions
• Reproducibility
• NIH — Enhancing Reproducibility through Rigor and Transparency
• NIH — Strengthening Replication and Reproducibility

The National Academies notes that terminology varies across communities, which is why the article states the convention explicitly rather than pretending there is universal language.

373. The Reproducibility Contract

A scientific computation makes a contract with the future:

I will preserve the data version, code, environment, parameters and workflow that created my result. I will state where exact reconstruction is impossible. I will protect sensitive data appropriately. I will preserve corrections and version history. And I will leave enough of the path visible that another researcher can reconstruct what I actually did.

374. How Scientific Reproducibility Works: final synthesis

Modern science often compresses enormous computational processes into one line of a paper.

Behind that line may be:

• millions of records;
• hundreds of code steps;
• dozens of dependencies;
• stochastic models;
• external data services.

Reproducibility expands that line back into an inspectable process.

The durable sequence is:

versioned data → documented transformations → versioned code → captured environment → explicit workflow → regenerated outputs → independent reconstruction.

Reproducibility does not prove the theory.

It does not replace replication.

It does not guarantee valid measurement.

It does something more basic and indispensable:

it makes the computation part of the scientific record rather than a private event that happened once on one machine.

---

375. Future-Proof Reproducibility

A reproducible project should not be designed only for the computer on which it was created. Scientific results can outlive laptops, software companies, operating systems and research groups. Future-proofing is the effort to keep the computational path reconstructable after the surrounding technology changes.

Preserve source, not only executable output

A compiled binary may stop running on future hardware.

Source code gives future researchers a better chance of rebuilding the environment or porting the method.

Preserve environment recipes, not only environment images

A container image is valuable.

A container recipe is also valuable because it explains how the image was built.

Keep both when feasible.

Preserve human-readable documentation

Machine-readable manifests are powerful, but future researchers may need plain explanations of:

• why a step exists;
• which file is authoritative;
• which approximation was intentional.

Preserve open intermediate formats

Even if the raw instrument format is proprietary, export documented open derivatives where scientifically appropriate.

This reduces dependence on one vendor.

Preserve representative test cases

Small test data with known expected outputs help future maintainers verify a ported environment.

Preserve benchmark outputs

Archive:

• key summary statistics;
• expected figure hashes where exact;
• tolerance ranges where approximate.

Future rebuilds need something to compare against.

376. Reproducibility Under Software Obsolescence

Software disappears in several ways:

• project abandoned;
• website removed;
• licence server shut down;
• old OS unsupported;
• security policy blocks installation.

A preservation plan can include:

• source archive;
• binary archive where lawful;
• virtual machine;
• container image;
• exported open data;
• migration documentation.

377. Emulation versus migration

Emulation preserves the old environment.

Migration moves the workflow to a new environment.

Both can support future reproducibility.

Migration should be validated against archived benchmark outputs.

378. Reproducibility Under Data Evolution

Data sources evolve through:

• corrections;
• new records;
• changed coding;
• changed schema.

The publication should reference a frozen version.

Later analyses can use updated data as a new study version.

379. Data correction does not invalidate reproducibility

Two questions can coexist:

1. Can we reproduce what the original paper reported from the original data version?
2. What does the corrected data version show now?

Preserve both.

380. Reproducibility Under Algorithm Evolution

A scientific algorithm can improve after publication.

Examples:

• bug fix;
• faster solver;
• better optimiser;
• new approximation.

The old version should remain archived for historical reproduction.

The new version should be evaluated scientifically rather than silently substituted.

---

381. Reproducibility Verification Levels

Level 1 — Artifact availability

Data/code exist and can be accessed.

Level 2 — Artifact readability

Another researcher can understand file roles and documentation.

Level 3 — Environment reconstruction

Dependencies can be installed or restored.

Level 4 — Workflow execution

The analysis runs end to end.

Level 5 — Output reconstruction

Main figures, tables and statistics are recovered.

Level 6 — Independent verification

A researcher not involved in the original analysis completes the rebuild.

Level 7 — Temporal verification

The project still rebuilds after substantial time or technology change.

These levels describe progressively stronger evidence of computational durability.

382. Why Level 1 is not enough

A public code repository can be:

• uninstallable;
• undocumented;
• missing data;
• missing run order.

Availability is only the first step.

383. Why Level 6 matters

Independent verification tests hidden knowledge.

The original analyst already knows how to navigate ambiguity.

The independent analyst does not.

If they succeed, documentation has crossed an important threshold.

---

384. Reproducibility Failure Escalation

Minor issue

One package version missing, easily identified.

Response: update documentation.

Moderate issue

One figure cannot be rebuilt because a preprocessing script is missing.

Response: reconstruct script and archive corrected release.

Major issue

Main statistical result changes materially under the documented workflow.

Response: audit urgently and consider correction.

Critical issue

Data, code or reported analysis cannot support the main scientific conclusion.

Response: correction, expression of concern or retraction may be needed depending on the facts.

The severity depends on scientific consequence, not embarrassment.

385. Reproducibility Incident Report

A reproducibility incident report can contain:

• reported problem;
• affected output;
• root cause;
• scientific impact;
• corrective action;
• new release;
• preventive action.

This borrows a useful principle from engineering: learn from failure systematically.

386. Root-Cause Categories for Computational Science

• data provenance;
• data cleaning;
• software bug;
• dependency drift;
• hardware/numerical issue;
• randomness;
• documentation;
• manual intervention;
• external service;
• access failure.

Classification helps institutions see recurring problems across projects.

---

387. Reproducibility Quality Assurance

Quality assurance can be designed into the research lifecycle.

At project start

• create repository;
• define data folders;
• set environment;
• write initial README.

During data collection

• preserve raw files;
• record metadata;
• version protocols.

During analysis

• script transformations;
• test code;
• commit changes;
• capture seeds.

Before submission

• clean rebuild;
• independent audit;
• archive release.

After publication

• monitor issue reports;
• preserve links;
• release corrections transparently.

388. Reproducibility QA should be proportional

A small exploratory pilot does not require the same infrastructure as:

• clinical-decision software;
• national policy analysis;
• foundational scientific database.

Scale quality assurance with consequence.

---

389. Reproducibility and Institutional Memory

Research institutions often lose knowledge when:

• students graduate;
• staff leave;
• labs move;
• servers are replaced.

Institutional memory requires:

• repositories;
• access governance;
• documentation;
• maintenance ownership.

390. Ownership should be explicit

After publication, who maintains:

• repository;
• data access;
• software issues;
• security updates?

A project without an owner can decay rapidly.

391. Succession planning

Long-lived research infrastructure should name:

• primary maintainer;
• backup maintainer;
• institutional contact.

392. Repository sustainability

Choose repositories with:

• persistent identifiers;
• preservation policy;
• institutional backing.

Personal cloud links are not preservation plans.

---

393. Reproducibility and Method Transparency

A method should expose decisions that materially affect results.

This includes:

• thresholds;
• transformations;
• exclusions;
• model selection;
• stopping criteria.

394. Transparent does not mean verbose

Good documentation is:

• complete;
• structured;
• searchable.

A thousand pages of disorganised notes can be less useful than a clear workflow diagram and precise README.

395. Reproducibility diagrams

A workflow diagram can show:

source data → cleaning → feature construction → model → outputs.

Each node should link to code or artifact.

396. Machine-readable provenance

Advanced workflows can store provenance in structured metadata.

This allows automated tools to trace dependencies.

397. Provenance graphs

A provenance graph represents:

• artifacts;
• processes;
• versions;
• dependencies.

This is especially useful for large computational pipelines.

---

398. Reproducibility and Scientific Decision-Making

Decision-makers should ask whether the analysis behind a recommendation is reproducible before treating it as operational evidence.

Examples:

• clinical risk score;
• budget forecast;
• environmental model;
• education-policy analysis.

399. Decision reproducibility

For a historical decision, preserve:

• input data;
• model version;
• threshold;
• generated recommendation.

Later audit should be able to reconstruct the decision.

400. Reproducibility supports accountability

When a scientific model affects people, reproducibility helps answer:

• which model acted?
• which data were used?
• which rule produced the output?

This is both scientific and governance infrastructure.

---

401. Final Reproducibility Checklist

• Exact research result identified.
• Exact input data version preserved.
• Raw-to-analytic transformations scripted.
• Data dictionary present.
• Code version tagged.
• Dependencies locked.
• Operating system documented.
• Hardware documented where relevant.
• Random seeds and stochastic settings preserved.
• Clean build tested.
• Notebooks run top-to-bottom.
• Warnings logged.
• Figures generated from code.
• Tables generated from code.
• External APIs versioned or archived.
• Restricted-data path documented.
• Licensing clear.
• Persistent archive created.
• Independent auditor completed rebuild.
• Numerical tolerances defined.
• Known limitations stated.
• Correction/version history preserved.

402. Final acceptance question

Could another competent researcher, years from now, understand which data we used, recreate the computational environment, run the same scientific method and recover a result consistent with what we published?

If the answer is yes, the computational result has become part of the durable scientific record rather than a private event on one machine.

403. Final reproducibility integrity note

Reproducibility should never be used to make a weak study look stronger than it is.

A reproducible bias remains bias.

A reproducible confounded association remains confounded.

A reproducible coding mistake remains a mistake.

The guide therefore preserves one boundary above all:

reproducibility makes the computation inspectable; evidence, validity, causation, replication and theory decide what the computation means scientifically.

---

404. Reproducibility Certification Appendix

There is no single universal certification that fits every scientific field, but a project can document a practical reproducibility status using explicit evidence rather than a vague badge.

Status A — Artifact available

The project exposes or provides controlled access to:

• data;
• code;
• documentation.

This is the first level only.

Status B — Environment reconstructable

Another researcher can create the declared software environment from:

• lockfile;
• container;
• virtual machine;
• environment recipe.

Status C — Workflow executable

The analysis can run end to end from the documented entry point.

Status D — Main outputs reconstructed

The reported main tables, figures and estimates are recovered within stated tolerance.

Status E — Independently verified

A researcher outside the original analysis team completes the rebuild successfully.

Status F — Temporally durable

The project has survived later rebuilding after software, hardware or personnel changed.

These statuses describe evidence of reproducibility rather than prestige.

405. Reproducibility Certification Record

A certification record can include:

Field	Record
Project release	commit/tag/DOI
Input data	version/checksum
Environment	container/lockfile
Auditor	name/team
Audit date	timestamp
Main result status	exact / numeric / stochastic
Known deviations	documented list
Limitations	data/software/hardware constraints

This makes the reproducibility claim itself reproducible.

406. Reproducibility Maintenance

After publication, a project can deteriorate silently.

Maintenance can include:

• checking repository links;
• rebuilding containers;
• testing dependency installation;
• verifying data access;
• running automated tests;
• archiving new corrected releases.

407. Dependency monitoring

Security or compatibility updates can affect packages.

A maintained project should distinguish:

• historical publication environment;
• current maintained environment.

The historical environment remains the reference for exact reconstruction.

408. Security versus reproducibility

Old software can contain vulnerabilities.

Long-term reproducibility should not require exposing active systems to unsafe packages.

Use:

• isolated VMs;
• offline containers;
• archived environments;
• migration to maintained code.

409. Reproducibility after migration

If a project moves from an obsolete environment to a modern one:

1. run old environment;
2. record benchmark outputs;
3. port the code;
4. run the same inputs;
5. compare outputs;
6. document any scientifically meaningful differences.

This creates continuity across technology generations.

410. Reproducibility and deprecation

A project can deprecate:

• old APIs;
• old file formats;
• old scripts.

Deprecation should point users to:

• replacement;
• migration guide;
• historical archive.

411. Reproducibility Issue Tracking

Users should have a place to report:

• installation failure;
• data mismatch;
• numerical divergence;
• documentation gap.

Issue trackers create a public maintenance record where appropriate.

412. Reproducibility response time

High-stakes scientific software should define ownership for critical reproducibility failures.

If the main published result stops rebuilding, the issue deserves priority.

---

413. Reproducibility and Scientific Trust

Scientific trust should not require readers to trust the analyst’s memory.

It should rest on inspectable artifacts.

Reproducibility supports trust because it allows another person to check:

• what data entered;
• what code ran;
• what environment existed;
• what result emerged.

414. Transparency reduces authority dependence

Without reproducibility, readers may have to say:

“I trust the authors.”

With reproducibility, they can say:

“I can inspect the route from input to result.”

Science becomes less dependent on personal authority.

415. Reproducibility is not distrust

Making research checkable is not an accusation.

It is normal scientific architecture.

Important claims deserve transparent computational records precisely because they matter.

416. Reproducibility and correction culture

A healthy research culture should make it easy to say:

• we found a bug;
• we corrected it;
• here is the old version;
• here is the new result.

Correction is evidence that the system can learn.

---

417. Reproducibility in One Page

If all details are forgotten, preserve this minimal operating system:

1. Freeze the input. Know exactly which data version was used.
2. Script the transformation. Raw data should become analytic data through documented code.
3. Version the code. Preserve the exact analysis release.
4. Capture the environment. Record software, dependencies and hardware when relevant.
5. Control randomness. Record seeds and define stochastic tolerances.
6. Automate the workflow. Make the build order explicit.
7. Generate outputs from source. Figures and tables should not depend on hidden manual edits.
8. Archive artifacts. Use persistent storage and identifiers.
9. Run a clean rebuild. Test from a fresh environment.
10. Let someone else try. Independent reconstruction is the strongest documentation test.

418. Reproducibility Final Handover

A mature project should hand the next researcher enough information to answer five questions immediately:

1. Which exact data did the paper use?
2. Which exact code version produced the result?
3. Which environment is required?
4. Which command rebuilds the analysis?
5. Which outputs should I expect?

If those questions are easy to answer, the computational record is healthy.

419. Reproducibility scope

This guide remains focused on reconstruction rather than new empirical confirmation.

Its durable responsibilities are:

• data provenance;
• code provenance;
• environment capture;
• workflow execution;
• artifact archiving;
• independent rerunning;
• long-term maintenance.

The adjacent replication guide focuses on collecting new data and asking whether the scientific finding holds again.

420. Final integrity close

A scientific computation should be able to survive the disappearance of the analyst who created it.

That is the deepest reproducibility test.

The result should remain reconstructable because the project preserved:

• the evidence;
• the transformations;
• the software;
• the environment;
• the workflow;
• the version history.

Reproducibility is the discipline of leaving enough scientific machinery behind that another researcher can rebuild the result without needing the original researcher’s memory.

When that machinery is preserved, computational science becomes durable enough to inspect, correct, teach, reuse and trust.

---

421. Final Reproducibility Verification Appendix

A reproducibility claim should be treated like any other scientific claim: it needs evidence. Saying “our analysis is reproducible” is weaker than documenting that another person actually rebuilt it from the archived artifacts.

Verification evidence 1 — Clean checkout

The auditor begins from the archived publication release rather than the author’s working directory.

Verification evidence 2 — Fresh environment

The auditor constructs the environment using only the supplied dependency information.

Verification evidence 3 — Input verification

The data checksum, version or controlled-access extract matches the publication record.

Verification evidence 4 — Full workflow execution

The auditor runs the documented entry command without private instructions from the analyst.

Verification evidence 5 — Output comparison

Main figures, tables and estimates match exactly or within pre-specified scientific tolerance.

Verification evidence 6 — Discrepancy log

Any differences are recorded with cause and resolution rather than silently repaired.

Verification evidence 7 — Independent sign-off

The final audit identifies who reproduced the result, when, and under which environment.

422. Reproducibility Verification Report

A short verification report can contain:

• Project: title and release ID.
• Auditor: independent researcher or team.
• Date: audit date.
• Data: version and checksum.
• Environment: container, VM or lockfile.
• Workflow: command executed.
• Result: exact, numeric, stochastic or partial reproduction.
• Differences: documented discrepancies.
• Conclusion: which published outputs were reconstructed successfully.

This report converts reproducibility from an aspiration into an observed property of the research artifact.

423. Reproducibility and Future Reuse

A project that can be reproduced is easier to reuse scientifically because future researchers can distinguish:

• the historical published analysis;
• their new modifications;
• the resulting change in output.

This supports cumulative work without erasing provenance.

424. Reuse should create a new version

When another team extends the project, they should preserve the original release and create a new version containing:

• new data;
• new analysis;
• new environment changes;
• new outputs.

The lineage between old and new should remain visible.

425. Reproducibility as scientific continuity

The strongest computational projects behave like durable scientific instruments. They can be inspected, calibrated, rerun, corrected and handed to another researcher.

The final standard is therefore not “the code is online”. It is:

the scientific result can be reconstructed from preserved inputs, explicit transformations, a captured environment and a documented workflow by someone other than the original analyst.

That is the point at which reproducibility becomes part of the scientific evidence system rather than an administrative extra.

---

426. Reproducibility continuity

This reproducibility guide should remain useful even as software tools change. The specific technologies used today—containers, notebooks, workflow managers, cloud services, package lockfiles—may eventually be replaced. The durable scientific obligations underneath them are more important than the tools themselves.

Future updates should preserve five core requirements:

1. Input identity: another researcher can identify the exact evidence used.
2. Transformation identity: every material data-processing and analysis step is explicit.
3. Environment identity: the computational conditions are captured well enough to reconstruct behaviour.
4. Output identity: published figures, tables and estimates map to executable analysis.
5. Audit identity: an independent researcher can verify what happened without undocumented personal knowledge.

The guide should also preserve its scope boundary. Reproducibility ends with reconstruction of the original computational result. If a new study collects new data and asks whether the scientific claim holds again, that belongs to replication. If the question is whether the evidence is trustworthy, continue to evidence and measurement. If the question is what the finding means mechanistically, continue to explanation and theory.

The final integrity rule is therefore:

Preserve enough of the research process that the result remains inspectable after the original analyst, software session and computer are gone.

When that condition is met, computational work becomes a durable scientific object rather than a transient execution.

427. Final Verification Close

A reproducible project should make one thing possible without interpretation-by-memory: a second researcher can start from the archived inputs, reconstruct the declared environment, execute the workflow, inspect warnings, and recover a scientifically consistent version of the published result. If a material discrepancy remains, the project should preserve that discrepancy and the correction path rather than overwrite history.

This final requirement is what separates a merely documented project from a genuinely reproducible one. Documentation describes what should happen. Independent reconstruction demonstrates what actually happens.