How Scientific Replication Works | From One Result to a Finding That Survives Independent Repetition

How Scientific Replication Works | From One Result to a Finding That Survives Independent Repetition is a world-facing guide to scientific replication, replicability, reproducibility, repeatability, independent replication, direct replication, conceptual replication, replication crisis, research rigor, preregistration, registered reports, publication bias, effect sizes, meta-analysis, robustness and scientific self-correction. A single study can be careful, statistically significant and still be wrong, incomplete, context-bound or exaggerated. Replication asks whether the finding survives when science gets another chance to test it with new data.

This page uses the National Academies distinction for clarity: replicability means obtaining consistent results across new studies aimed at the same scientific question, while reproducibility means obtaining consistent computational results using the same data, code, methods and conditions of analysis. Different fields sometimes use these terms differently, so good scientific writing should always define its usage. This guide focuses on new-data repetition; reproducibility is treated separately as data/code/workflow reconstruction.

Within the eduKateYishun scientific-method library, How Scientific Hypotheses Work focuses on candidate claims; How Scientific Evidence Works focuses on how observations change confidence; How Scientific Experiments Work focuses on test architecture; How Scientific Causation Works focuses on intervention and counterfactuals; and How Scientific Theories Work focuses on broad explanatory frameworks. This guide follows the loop original claim → replication design → new data → comparison → explanation of agreement/disagreement → synthesis → updated confidence.

---

Choose your route

• Basic route: why one study is not enough.
• Terminology route: replication vs reproducibility vs repeatability vs generalisability.
• Design route: direct replication, conceptual replication, multi-lab replication and adversarial collaboration.
• Statistics route: effect sizes, uncertainty, power, equivalence, prediction intervals and why exact duplication is not expected.
• Research-integrity route: preregistration, publication bias, selective reporting and replication incentives.
• Synthesis route: meta-analysis, heterogeneity, moderator analysis and the cumulative evidence base.
• Advanced route: replication failure diagnosis, hidden moderators, measurement non-invariance and theory revision.

1. Replication is science asking the same question again with new evidence

A study reports a result.

The first question is:

“What did this study find?”

The replication question is:

If we investigate the same scientific question again using new data, do we obtain a result consistent with the original finding, given the uncertainty and variation inherent in the system?

This second question is essential because any single study can be influenced by:

• sampling variation;
• measurement noise;
• researcher decisions;
• local context;
• unrecognized bias;
• data quality;
• chance.

2. Replication is not about proving the original study “right”

A replication can:

• support the original result;
• find a smaller effect;
• find a larger effect;
• find no detectable effect;
• find an effect only under certain conditions;
• reveal that the original result was context-specific;
• reveal that one method was biased.

Each outcome can teach us something.

3. Replication is not binary

The National Academies emphasises that replication should often be understood as a matter of degree rather than a simple “success/failure” label.

Two studies can differ numerically yet still be statistically and scientifically compatible.

Ask:

• Do effect estimates point in the same direction?
• Are confidence intervals compatible?
• Is the magnitude reasonably similar?
• Do differences match expected context variation?
• Does the replication test the same scientific question?

4. Exact numerical duplication is not expected with new data

New samples contain new random variation.

If the original study estimated an effect of 0.50, a replication might estimate 0.38 or 0.62 and still be compatible with the same underlying effect.

Replication is not a photocopy.

It is a new empirical test.

5. Why one study is fragile

Even a well-designed study can produce an unusual estimate by chance.

If many laboratories study the same effect, the first published result may be unusually large simply because large results are more noticeable and easier to publish.

Replication helps move from:

one estimate → distribution of estimates → better understanding of the underlying effect

6. The first estimate can regress toward the mean

Suppose an early study reports a very large effect.

Later studies find smaller effects.

This can happen without misconduct.

The first estimate may have been:

• noisy;
• selected because it was striking;
• conducted in a context where the effect was genuinely larger.

Replication tells us whether the phenomenon is stable and how large it usually is.

7. Replication increases confidence by testing independence

Evidence becomes stronger when another study uses:

• new participants;
• new samples;
• new measurement occasions;
• new researchers;
• new laboratories.

Why?

Because the original study’s accidental quirks are less likely to repeat identically.

8. Independence has levels

A replication conducted by the original lab with new data is valuable.

A replication by an independent lab can be more informative about:

• hidden procedural assumptions;
• experimenter effects;
• local equipment;
• analysis idiosyncrasies.

Both kinds matter.

9. Direct replication

A direct replication attempts to match the original study’s essential design as closely as practical while collecting new data.

The question is:

Does the reported effect reappear when the same scientific test is repeated?

10. Direct does not mean literally identical

No two biological, social or field studies are perfectly identical.

Participants differ.

Time differs.

Researchers differ.

Equipment may differ.

A direct replication aims to preserve the theoretically relevant features.

11. Theoretical fidelity matters more than cosmetic identity

Suppose the original experiment depends on:

• stimulus timing;
• dosage;
• temperature;
• measurement sensitivity.

A replication that matches the room colour but changes the dosage has preserved the wrong detail.

Replication design requires theory.

12. Conceptual replication

A conceptual replication tests the same underlying scientific claim using a different method, manipulation, measure or context.

This asks:

Does the effect survive when the surface form changes but the theoretical relationship remains?

13. Conceptual replication tests generality

Examples:

• different measurement instrument;
• different population;
• different operational definition;
• different experimental manipulation;
• different setting.

If the effect persists, confidence grows that the original finding was not an artifact of one procedure.

14. Direct and conceptual replication answer different questions

Direct replication asks:

“Can we get this finding again?”

Conceptual replication asks:

“Does the underlying scientific idea travel?”

A mature research programme needs both.

15. Replication and generalisability are different

Replicability asks whether a finding is consistent when the same scientific question is tested again with new data.

Generalisability asks whether the result applies in other contexts or populations.

Conceptual replication can contribute to generalisability, but the goals should be distinguished.

16. Reproducibility is a neighbouring job

Using the National Academies terminology:

Reproducibility: same input data and computational steps produce consistent results.

Replicability: new study, new data, same scientific question, consistent result.

A result can be computationally reproducible but fail to replicate.

A result can also be scientifically replicable even if the original computational workflow was poorly documented.

These are separate quality dimensions.

17. Repeatability

Some disciplines use “repeatability” for repeated measurements under the same conditions.

Terminology varies internationally and across standards bodies.

Always define what is meant.

18. Terminology conflicts are real

The National Academies specifically notes that different communities have used “reproducibility” and “replicability” inconsistently, sometimes with opposite definitions.

This article uses one explicit convention so readers know what is being discussed.

---

19. Replication as error correction

Science does not assume researchers are perfect.

Instead, science builds systems where errors can be found later.

Replication is one such system.

It can expose:

• false positives;
• overestimated effects;
• hidden moderators;
• measurement problems;
• analysis fragility;
• context dependence.

20. Replication protects against overconfidence

A dramatic first result can become a headline.

Replication asks whether the headline survives ordinary reality.

This is especially important when:

• sample is small;
• effect is surprising;
• many outcomes were tested;
• measurement is noisy;
• the finding has large practical consequences.

21. Replication is not an attack on researchers

Replication should be normal scientific maintenance.

Important claims deserve more testing precisely because they are important.

A replication attempt is not automatically an accusation of misconduct.

22. Non-replication is not automatically misconduct

Replication differences can arise from:

• sampling;
• method differences;
• context variation;
• measurement;
• statistical noise;
• publication bias;
• genuine false positives.

Misconduct is a separate hypothesis requiring evidence.

23. Replication can strengthen the original researchers

If independent teams reproduce the result, confidence increases beyond what the original authors could establish alone.

Replication distributes ownership of the finding across the scientific community.

24. Replication converts local knowledge into public knowledge

A result known only inside one laboratory is fragile.

A result that survives independent repetition becomes more transportable.

This is part of how science turns individual observation into community knowledge.

25. Replication is cumulative, not ceremonial

The goal is not to repeat every study mechanically.

Replication resources should be directed toward findings that are:

• important;
• uncertain;
• widely used;
• surprising;
• high-impact;
• foundational.

26. Which studies should be replicated?

Prioritise studies where:

• one result drives major policy;
• one finding anchors a theory;
• effect size seems unexpectedly large;
• sample was small;
• methods have many analytic choices;
• consequences of error are high.

27. Replication has opportunity cost

Every replication consumes:

• time;
• funding;
• participants;
• equipment;
• research effort.

Science therefore needs replication strategy, not only replication advocacy.

28. Replication portfolio

A field can maintain a portfolio containing:

• direct replications of key effects;
• conceptual replications;
• multi-lab tests;
• high-risk/high-impact findings;
• random audits.

This balances certainty-building and new discovery.

29. NIH and replication culture

NIH currently frames replication and reproducibility as foundational to “gold standard” science and is investing in replication infrastructure, incentives, data sharing and independent replication efforts.

This matters because replication quality depends not only on individual scientists but also on funding and institutional incentives.

30. Incentives shape what gets replicated

Discovery is often rewarded more visibly than confirmation.

Researchers may gain more career credit for:

• novel positive findings;
• new theories;
• new methods.

Replication can therefore be socially undervalued even when scientifically essential.

31. Scientific institutions can correct incentives

Possible strategies include:

• replication grants;
• registered reports;
• journals accepting null replications;
• shared protocols;
• multi-lab collaborations;
• replication awards.

Replication culture is part of scientific infrastructure.

---

32. Multi-lab replication

A multi-lab replication tests one scientific question across several independent laboratories using a coordinated protocol.

This design can reveal:

• between-lab variation;
• hidden procedural assumptions;
• population differences;
• experimenter effects;
• equipment differences.

One lab can be unusual.

Ten labs create a distribution.

33. Why multi-lab designs are powerful

They separate:

• effect that appears only locally;
• effect that generalises across labs;
• effect whose size varies systematically.

They also make protocol interpretation more explicit because several teams must implement the same scientific idea.

34. Standardisation versus variation

Replication design faces a tension.

If every lab uses exactly the same materials, population and procedure, the test is close to direct replication.

If labs vary naturally, the result can reveal robustness and generalisability.

A strong programme can use both:

1. standardised core protocol;
2. planned variation in theoretically relevant features.

35. Systematic heterogeneity

Differences across replications are not always noise.

They can reveal moderators.

Example:

• effect stronger in younger participants;
• effect disappears at low dose;
• effect depends on measurement instrument;
• effect exists only under one environmental condition.

Replication can discover where a claim applies.

36. Replication is a boundary-mapping tool

The first study says:

“The effect exists here.”

Replications ask:

• here again?
• somewhere else?
• with another method?
• at another dose?
• in another population?

The research programme turns one point into a map.

37. Adversarial collaboration

Researchers who disagree about a finding can jointly design a replication.

They pre-specify:

• protocol;
• analysis;
• success criteria;
• interpretation rules.

This reduces arguments after the result because both sides helped define the test beforehand.

38. Replication teams should include original expertise when useful

The original authors may know procedural details not obvious from the paper.

Consulting them can improve fidelity.

But the replication should remain independently auditable.

39. Original-author involvement is not required for scientific legitimacy

A well-documented method should be usable by others.

If a finding can be produced only when one specific team performs it, that limitation itself is scientifically important.

40. Registered replication reports

A replication can be preregistered before data collection.

Registered reports go further by reviewing methods before results are known.

This reduces:

• outcome switching;
• analysis flexibility;
• publication bias based on the result.

---

41. What counts as a successful replication?

There is no one universal rule.

Possible criteria include:

• same direction of effect;
• effect estimate within original confidence interval;
• replication confidence interval overlapping original estimate;
• meta-analytic combined effect;
• equivalence within a meaningful range;
• Bayesian evidence;
• successful prediction of replication data.

Each criterion answers a different question.

42. P-value matching is a poor replication standard

Original:

p = .03

Replication:

p = .08

Calling this automatically “failed” can be misleading.

If both studies estimate similar effects but the replication is noisier, the results may be compatible.

43. Significance is sensitive to sample size

The same effect size can produce:

• significant p-value in a large sample;
• non-significant p-value in a small sample.

Replication should compare effect estimates and uncertainty, not only threshold labels.

44. Compare effect sizes directly

Suppose:

• original effect = 0.60;
• replication effect = 0.35.

The questions are:

• are these statistically compatible?
• is the replication closer to the true effect because the original was inflated?
• is there real context heterogeneity?

Effect-size shrinkage can be informative.

45. Original effect sizes are often overestimated

Reasons include:

• small samples;
• publication selection;
• multiple analyses;
• winner’s curse;
• stopping after significance.

Replications often provide more conservative estimates.

46. Winner’s curse

When many noisy studies are conducted, the most extreme positive estimate is more likely to be noticed or published.

The selected study can therefore overestimate the underlying effect.

Replication pulls the estimate toward a more realistic value.

47. Power in replication

A replication should be large enough to answer the intended question.

If the original effect estimate is inflated, powering the replication only for that large effect can produce another underpowered study.

Better planning can use:

• smallest effect size of interest;
• meta-analytic estimate;
• precision target;
• equivalence margin.

48. Precision-based planning

Instead of asking:

“How many participants do we need to get p < .05?”

ask:

“How many participants do we need for the confidence interval to be narrow enough to distinguish scientifically important effects?”

This aligns replication with estimation.

49. Smallest effect size of interest

A replication can define the smallest effect that would matter scientifically or practically.

Then the analysis can distinguish:

• meaningful positive effect;
• effect too small to matter;
• uncertain result.

50. Equivalence testing

If the scientific question is:

“Is the effect effectively absent or smaller than a meaningful threshold?”

equivalence testing can be more appropriate than simply failing to reject zero.

51. Replication interval

A prediction interval based on the original evidence can describe the range of effect estimates expected in a new study.

If the replication falls far outside that range, the disagreement is more surprising.

52. Meta-analytic replication assessment

Instead of treating original and replication as opponents, combine them.

A meta-analysis estimates the effect across studies and quantifies heterogeneity.

This turns replication into cumulative science.

53. Fixed-effect meta-analysis

A fixed-effect model assumes studies estimate one common underlying effect, with differences arising from sampling error.

This can be appropriate when studies are very similar and the common-effect assumption is defensible.

54. Random-effects meta-analysis

A random-effects model allows underlying effects to differ across studies.

This is useful when populations, implementations or contexts vary.

The average effect and between-study heterogeneity both matter.

55. Heterogeneity

Heterogeneity asks:

“How much do the underlying study effects vary?”

High heterogeneity can mean:

• real moderators;
• method differences;
• measurement differences;
• bias;
• random variation.

It is a question, not a nuisance statistic to ignore.

56. I²

I² is commonly used to describe the proportion of observed variation attributable to heterogeneity rather than sampling error under a meta-analytic framework.

It should not be interpreted mechanically.

Its value depends on precision and number of studies.

57. Prediction intervals in meta-analysis

A random-effects prediction interval estimates where the effect of a new similar study might plausibly lie.

This can be more decision-relevant than the average effect alone.

58. A positive average can coexist with negative effects in some contexts

If heterogeneity is large, an intervention can be helpful on average but ineffective or harmful in some settings.

Replication across contexts reveals this structure.

59. Moderator analysis

When replications differ, ask whether effect size varies with:

• age;
• dose;
• country;
• measurement;
• lab;
• protocol version.

Moderator analysis turns replication disagreement into a new scientific hypothesis.

60. Meta-regression

Meta-regression relates study-level characteristics to effect estimates.

It can suggest moderators.

But study-level associations can be confounded and underpowered.

Meta-regression is exploratory unless supported by stronger evidence.

---

61. Replication failure diagnosis

When a replication differs, do not jump directly to:

“The original was false.”

Ask which layer changed.

Layer 1 — Sampling

Could ordinary random variation explain the difference?

Layer 2 — Measurement

Was the same construct measured with comparable validity?

Layer 3 — Protocol

Were theoretically important details preserved?

Layer 4 — Population

Did the replication test a different group?

Layer 5 — Context

Did environment, culture or time change the effect?

Layer 6 — Analysis

Were statistical methods comparable and pre-specified?

Layer 7 — Original bias

Was the original effect inflated by publication or analysis selection?

Layer 8 — True null

Was the original result a false positive?

Replication failure is a diagnostic tree.

62. Hidden moderators

A replication difference can reveal an unmeasured moderator.

Example:

• effect exists only after prior exposure;
• effect depends on baseline severity;
• effect requires a specific environmental range.

The next study should test the moderator prospectively.

63. Measurement non-invariance

A scale or instrument may not measure the same construct identically across populations.

If measurement meaning changes, replication can appear to fail even when the underlying phenomenon is similar.

Test measurement invariance before interpreting group differences.

64. Implementation fidelity

A treatment replication can fail because the intervention was not delivered in the same meaningful way.

Record:

• dose;
• duration;
• training;
• adherence;
• quality.

65. Context is not an excuse unless it predicts

After a failed replication, saying:

“The context was different”

is too vague.

A scientifically useful context explanation should predict:

• which context variable matters;
• how it changes the effect;
• what a new study should observe.

66. Replication can improve theory

If an effect appears only under Condition C, theory should be revised:

Old:

“X causes Y.”

New:

“X causes Y when C is present.”

The replication did not merely “fail”.

It found a boundary.

---

67. The replication crisis

Several fields have confronted evidence that some published findings do not reproduce or replicate as reliably as expected.

The phrase “replication crisis” is often used to describe this problem.

But the lesson should not be:

“Science cannot be trusted.”

The deeper lesson is:

science is strongest when it can detect where its own evidence is weaker than expected and build better systems to correct it.

68. Non-replication has multiple causes

Possible causes include:

• low statistical power;
• small samples;
• publication bias;
• selective reporting;
• p-hacking;
• flexible analysis choices;
• measurement problems;
• context dependence;
• genuine false positives;
• incorrect replication implementation.

No one explanation accounts for every failed replication.

69. Low power creates unstable estimates

Small studies produce noisy effect estimates.

If only statistically significant small studies are published, the literature can overrepresent unusually large estimates.

Replication with larger samples can correct this distortion.

70. Publication bias

Positive, novel or statistically significant results may be more likely to be published.

Null or contradictory findings may remain unpublished.

The visible literature can therefore exaggerate the true strength of evidence.

71. The file-drawer problem

Imagine 20 laboratories test the same null effect.

By chance, one obtains p < .05.

If only that one study is published, readers see a “positive” literature even though the full research record does not support the effect.

Replication and study registration help reveal the missing denominator.

72. Selective outcome reporting

A study can measure:

• ten outcomes;
• five time points;
• three subgroups.

If only the statistically significant combination is reported, the result appears more confirmatory than it was.

Replication can fail because it tests the declared claim rather than the hidden search process.

73. P-hacking

P-hacking refers to using analytic flexibility to search for a statistically significant result and then reporting the selected analysis as if it were the planned test.

Possible routes include:

• trying many covariates;
• trying many exclusion rules;
• trying many outcome transformations;
• stopping when significance appears;
• testing many subgroups.

Replication is less likely to reproduce a result that was selected from many hidden possibilities.

74. Researcher degrees of freedom

Even without deliberate p-hacking, reasonable choices can multiply.

Two researchers can analyse the same question differently.

If only one favourable analysis is published, the reported result understates uncertainty.

75. Garden of forking paths

Analytic decisions can branch based on what the data look like.

Each branch may appear reasonable.

But the final path is partly data-driven.

Preregistration makes the intended confirmatory path visible before outcomes are known.

76. HARKing

Hypothesizing After Results are Known can make an exploratory pattern appear as if it had been predicted.

Replication tests whether the post-hoc hypothesis predicts new data.

That is scientifically healthy.

77. Preregistration

Preregistration can record:

• hypotheses;
• sample plan;
• primary outcomes;
• analysis plan;
• exclusion rules;
• stopping rule.

It does not forbid exploration.

It distinguishes exploration from confirmation.

78. Registered reports

Registered reports review a study’s question and method before results are known.

In-principle acceptance can be based on methodological quality rather than whether the result is exciting.

This reduces publication incentives that favour positive results.

79. Replication registered reports

Replication studies are especially suited to registered reports because:

• scientific question already exists;
• protocol can be reviewed prospectively;
• analysis criteria can be agreed before outcomes;
• null replication results remain publishable.

80. Open materials

Replication improves when original researchers share:

• stimuli;
• protocols;
• instruments;
• code;
• analysis scripts;
• data dictionaries.

Transparency lowers ambiguity.

81. Open data

Where ethics and privacy allow, sharing data supports computational reproducibility and secondary checks.

It can reveal:

• coding errors;
• analysis differences;
• alternative interpretations.

But replication still requires new data.

82. Open code

Code helps others reconstruct:

• preprocessing;
• statistical models;
• figures;
• tables;
• exclusions.

A replication team can then separate:

• analytic reproducibility;
• empirical replicability.

83. Version control

Research code and protocols should be versioned.

Without version history, a replication may unknowingly use a revised method.

Version control supports:

• audit;
• comparison;
• rollback;
• transparent updates.

84. Data provenance

Replication depends on knowing:

• where data came from;
• how samples were selected;
• how variables were transformed;
• how missing values were handled.

Data provenance is scientific infrastructure.

85. Methods sections are often not enough

Published methods can omit:

• exact timing;
• equipment settings;
• training details;
• implementation quirks;
• analysis defaults.

Replication can reveal which omitted details mattered.

86. Protocols should identify theoretically essential details

A replication protocol should distinguish:

• essential features;
• optional features;
• allowed variation.

This makes replication scientifically interpretable.

87. Hidden tacit knowledge

Some laboratory techniques depend on skills learned informally.

Replication can fail because tacit procedural knowledge was not documented.

This is not evidence that the effect is imaginary.

It is evidence that the method is less portable than assumed.

88. Protocol videos and training

Complex procedures can be documented with:

• videos;
• checklists;
• shared training;
• standard reference materials.

These reduce implementation ambiguity.

89. Replication can improve methods, not only findings

A difficult-to-replicate protocol can motivate:

• better measurement;
• automation;
• standardisation;
• more robust controls.

Replication pressure can improve experimental technology.

---

90. Bias from stopping rules

If data collection stops as soon as p < .05, the resulting effect estimate can be biased upward.

A replication with a fixed larger sample can find a smaller effect.

This is not necessarily contradiction.

91. Optional stopping and replication

Sequential designs are legitimate when planned with appropriate methods.

The problem is undisclosed stopping based on favourable results.

92. Publication selection and winner’s curse interact

The most publishable early study may be:

• one of the largest effects;
• one of the smallest p-values;
• one of the most surprising results.

Replication helps estimate the typical effect rather than the most extreme visible estimate.

93. Small-study effects

In some literatures, smaller studies report larger effects.

This can reflect:

• publication bias;
• method differences;
• sampling variation;
• selective reporting.

Funnel plots and meta-regression can investigate the pattern.

94. Funnel plots

A funnel plot displays effect size against precision.

Asymmetry can suggest publication bias or small-study effects.

But asymmetry has multiple causes.

It is not a definitive proof of bias.

95. P-curve and related tools

Methods based on distributions of significant p-values have been proposed to assess evidential value and selective reporting.

They depend on assumptions and should not be interpreted mechanically.

96. Selection models

Meta-analytic selection models attempt to model how publication probability depends on study results.

They can estimate how much the literature may be distorted by selective publication.

97. Robustness through specification curves

A specification curve evaluates many reasonable analytic choices.

If a finding survives across choices, robustness increases.

If it appears only under one narrow analysis, replication risk is higher.

98. Multiverse analysis

A multiverse analysis examines the range of results across defensible analytic pathways.

This exposes uncertainty from researcher choices.

Replication can then pre-specify which pathways matter most.

99. Robustness is not the same as replication

A finding can be robust to many analyses in one dataset yet fail in new data.

Robustness and replication are complementary.

100. Replication requires genuinely new evidence

Re-analysing the original dataset is useful.

But it does not answer the same question as collecting new observations.

New data are the defining feature of replication under the convention used here.

---

101. Replication in school Science

School experiments often teach replication poorly.

Students may repeat one measurement three times and call that “replication”.

That is closer to repeated measurement or repeatability.

A true replication at classroom scale would involve a new run, new sample or another group asking the same scientific question again.

102. Primary Science: repeat and compare

Young students can learn:

• one result can be unusual;
• repeating a test helps check consistency;
• other groups may get slightly different numbers;
• differences should be investigated, not hidden.

103. Primary 3–4: measurement repeatability

At younger levels, repeating measurements teaches:

• care;
• variation;
• averaging;
• detecting obvious mistakes.

This is a foundation for later replication thinking.

104. Primary 5–6: independent groups

Different groups can run the same experiment.

Compare:

• same pattern?
• similar magnitude?
• different conditions?
• measurement differences?

This introduces the idea that science should survive beyond one group.

105. Secondary Science: replication versus repeated trials

Students can explicitly distinguish:

Repeated trial: same team repeats measurements within one experiment.

Replication: a new study or independent run collects new data to test the same scientific question.

106. School replication should not demand identical numbers

Students often think different results mean someone “did it wrong”.

Teach:

• measurement variation is normal;
• results should be compared with uncertainty;
• large disagreement deserves diagnosis.

107. Classroom replication notebook

Record:

• question;
• method;
• sample;
• conditions;
• result;
• difference from other groups;
• possible explanation.

This creates early scientific transparency.

---

108. Cross-Domain Replication Casebook

Replication looks different across scientific fields because the systems, measurements and interventions differ. The underlying question remains the same: does the scientific finding survive a new encounter with reality?

Case 1 — Chemical reaction rate

Original study:

Reaction rate increases with temperature over the tested range.

Direct replication:

A second laboratory repeats the experiment with fresh reagents and new measurements.

Conceptual replication:

A different reaction system tests whether the same kinetic principle holds.

Boundary result:

At extreme temperature, catalyst degradation changes the relationship.

The replication programme maps the domain.

Case 2 — Material tensile strength

Original finding:

Material A has higher tensile strength than B.

Replication should vary:

• new specimens;
• new batch;
• new testing machine;
• independent lab.

If only one manufacturing lot shows the effect, the claim becomes batch-specific.

Case 3 — Battery degradation

Original result:

High temperature accelerates capacity loss.

Replication across:

• cell manufacturers;
• charge protocols;
• temperature ranges;
• laboratories.

can reveal which conditions govern the effect.

Case 4 — Drug efficacy

A randomized trial finds benefit.

Replication questions include:

• does another trial reproduce efficacy?
• does a different population respond similarly?
• does real-world effectiveness match trial efficacy?
• does the mechanism replicate?

No one replication answers all four.

Case 5 — Rare adverse effect

A trial is too small to detect a rare harm.

Post-market databases find a signal.

Replication occurs through:

• another country;
• another healthcare system;
• different observational design;
• mechanistic evidence.

Converging new data can establish the harm more strongly.

Case 6 — Biomarker study

Original study reports that Biomarker B predicts disease.

Direct replication:

same assay, new cohort.

Conceptual replication:

different assay measuring the same biological construct.

External replication:

new hospital and different patient population.

Each layer tests a different dependency.

Case 7 — Gene association

Genetic association studies historically produced many false positives when samples were small.

Large independent cohorts and consortia improved replication by increasing power and standardising analysis.

This illustrates how infrastructure can transform replicability.

Case 8 — Gene knockout phenotype

One laboratory knocks out Gene G and observes Phenotype P.

Replication should test:

• another line;
• another background strain;
• independent editing method;
• rescue experiment.

This helps separate genuine gene effect from off-target or background effects.

Case 9 — Animal model

An intervention works in one mouse strain.

A replication in another strain fails.

Possible conclusion:

the effect is genotype-dependent rather than universally absent.

Replication finds the moderator.

Case 10 — Psychology priming effect

A small study reports a large behavioural priming effect.

Larger preregistered replications find smaller or absent effects.

Scientific update:

• effect may be weaker than first reported;
• boundary conditions may matter;
• original estimate may have been inflated.

The correct response is synthesis, not slogan.

Case 11 — Memory effect

A retrieval-practice study finds improved delayed recall.

Replications test:

• different materials;
• different ages;
• different delay intervals;
• transfer tasks.

Repeated success across contexts strengthens the theoretical mechanism.

Case 12 — Education intervention

A tutoring programme improves outcomes in one school.

Replication across schools can fail if:

• teacher training differs;
• student population differs;
• dosage differs;
• curriculum implementation differs.

Implementation fidelity must be measured.

Case 13 — Class-size experiment

Original trial shows smaller classes improve outcomes.

Replication in another system should preserve the causal contrast while recognising differences in:

• teacher supply;
• baseline class size;
• curriculum;
• school resources.

Replication tests transport as well as existence.

Case 14 — Social-science survey finding

One national survey finds an association.

Replication with:

• new survey year;
• another country;
• different questionnaire;

can reveal whether the pattern depends on measurement or context.

Case 15 — Policy evaluation

A difference-in-differences study estimates a policy effect.

Replication can use:

• another jurisdiction;
• synthetic control;
• regression discontinuity;
• another period.

Convergence across causal designs is stronger than repeating one model specification.

Case 16 — Economics field experiment

An incentive increases behaviour in one city.

Another city shows no effect.

Possible moderator:

• baseline prices;
• culture;
• market structure;
• existing incentives.

Replication reveals where the intervention generalises.

Case 17 — Conservation intervention

A protected area improves wildlife abundance.

Replication can compare:

• other habitats;
• other species;
• different enforcement levels;
• longer time horizons.

Ecological replication often requires planned heterogeneity rather than exact duplication.

Case 18 — Predator reintroduction

One ecosystem shows a trophic cascade after predator return.

Replications in other ecosystems may show different strengths because:

• food webs differ;
• human activity differs;
• prey behaviour differs.

Replication can refine mechanism rather than simply confirm or reject.

Case 19 — Climate model fingerprint

A model predicts a spatial pattern associated with forcing.

Replication can involve:

• new observational period;
• different dataset;
• different model ensemble;
• independent measurement system.

This is conceptual replication across data sources.

Case 20 — Geological dating

One method dates a layer.

Replication can use:

• new samples;
• another laboratory;
• independent dating method.

Agreement across independent physical methods is especially strong evidence.

Case 21 — Fossil interpretation

A claimed species identification can be replicated through:

• independent morphological analysis;
• another specimen;
• new imaging;
• molecular evidence where possible.

The scientific question survives beyond one expert judgment.

Case 22 — Exoplanet detection

A transit signal suggests a planet.

Replication can involve:

• repeated transits;
• independent telescope;
• radial-velocity measurement;
• different wavelength observations.

Different methods rule out different false positives.

Case 23 — Gravitational-wave detection

One detector signal becomes far stronger when:

• multiple detectors observe compatible signals;
• future events show predicted waveforms;
• independent analyses agree.

Replication can occur as repeated natural events rather than repeated laboratory manipulation.

Case 24 — Particle discovery

High-energy physics often relies on:

• independent detectors;
• different decay channels;
• large datasets;
• replicated statistical signals.

One analysis channel alone is weaker than convergent channels.

Case 25 — Machine-learning benchmark

A model reports state-of-the-art performance.

Replication questions include:

• same dataset, same code;
• independent reimplementation;
• new test set;
• distribution shift;
• different random seeds.

This domain makes the replication/reproducibility distinction especially clear.

Case 26 — AI medical model

An AI predicts disease accurately at one hospital.

Replication across:

• new hospital;
• new time period;
• new population;
• prospective deployment;

tests whether the signal is clinically robust rather than site-specific leakage.

Case 27 — Software performance claim

A new algorithm is faster on one benchmark.

Replication should vary:

• hardware;
• compiler;
• dataset;
• implementation;
• workload.

Performance claims can be highly context-dependent.

Case 28 — Engineering fatigue curve

Material fatigue results should replicate across new specimens and machines.

Batch variation is part of the scientific phenomenon, not merely noise.

Case 29 — Sensor calibration

One instrument measures a standard accurately.

Replication across laboratories can reveal:

• calibration drift;
• operator effects;
• environmental sensitivity.

Case 30 — Public-health intervention

A campaign reduces disease in one region.

Replication should measure:

• intervention fidelity;
• baseline disease burden;
• health-system differences;
• population behaviour.

Effect variation can guide future targeting.

Case 31 — Nutrition study

Nutrition findings are vulnerable to:

• measurement error;
• confounding;
• diet substitution;
• long-term adherence.

Replication across cohorts, trials and biomarkers can strengthen causal interpretation.

Case 32 — Epidemiological risk factor

One cohort finds a risk association.

Replication in another cohort can fail because:

• exposure distribution differs;
• outcome definition differs;
• confounding differs.

Harmonisation matters.

Case 33 — Diagnostic test

One study reports high sensitivity and specificity.

Replication should test:

• new patients;
• new prevalence;
• new operators;
• real clinical workflow.

Performance often declines from development to external validation.

Case 34 — Meta-science result

A study claims a methodological practice reduces false positives.

The practice itself should be replicated across fields.

Meta-science also needs replication.

Case 35 — Survey instrument

A psychological scale shows reliability in one culture.

Replication elsewhere should test:

• factor structure;
• measurement invariance;
• predictive validity.

Same score does not guarantee same construct.

Case 36 — Qualitative research finding

Replication language differs in qualitative research.

Instead of expecting identical themes, researchers may seek:

• credibility;
• transferability;
• dependability;
• auditability.

The broader scientific principle remains transparent examination of whether conclusions survive new evidence and contexts.

Case 37 — Archaeological interpretation

A claim about ancient settlement can be tested with:

• new excavation;
• independent dating;
• different site;
• remote sensing.

Replication in historical science means new traces, not rerunning the past.

Case 38 — Ocean circulation finding

A measurement campaign detects a circulation pattern.

Replication can use:

• another season;
• new instrument network;
• different year;
• independent satellite observations.

Case 39 — Atmospheric chemistry result

One field campaign identifies a reaction pathway.

Replication should combine:

• laboratory chemistry;
• another field site;
• model prediction.

Cross-method convergence strengthens the pathway.

Case 40 — Student experiment

One group finds that temperature changes dissolving rate.

Another group repeats with new samples.

If results differ, students should compare:

• water volume;
• stirring;
• particle size;
• temperature measurement;
• timing.

Replication becomes a lesson in scientific diagnosis.

---

109. Replication and theory

A replication does more than check an effect.

It can test a theory’s scope.

If Theory T predicts Effect E broadly and E appears only in one narrow condition, the theory needs revision.

110. Replication can reveal boundary conditions

Original:

“X causes Y.”

After replication programme:

“X causes Y only when Moderator M exceeds threshold T.”

The revised claim is scientifically stronger because its domain is clearer.

111. Replication can reveal mechanism

Suppose Effect E replicates only when Mediator M is present.

This supports a mechanistic role for M.

Replication across planned mechanism variants can therefore deepen explanation.

112. Replication can falsify broad theories

If a theory predicts an effect across many contexts but repeated high-quality studies find no effect, support for that part of the theory should decline.

113. Replication can rescue a theory from one bad study

Conversely, one negative study should not outweigh a large body of replicated evidence automatically.

Total evidence matters.

---

114. Replication Failure Atlas

When a replication differs from the original, the scientific task is diagnosis. The following failure patterns help separate sampling noise, method mismatch, hidden moderators, bias and genuine contradiction.

Failure 1 — The replication asks a different scientific question

Pattern: same label, different construct or treatment.

Repair: compare the actual causal or theoretical target, not the title.

Failure 2 — The replication changes a theoretically essential feature

Pattern: dosage, timing, temperature or stimulus strength differs.

Repair: restore fidelity or explicitly test the changed feature as a moderator.

Failure 3 — The replication overmatches irrelevant details

Pattern: cosmetic features are copied while core mechanism differs.

Repair: identify the theory-relevant elements before standardising.

Failure 4 — Original sample was too small

Pattern: large original effect shrinks in a larger replication.

Repair: update toward the more precise estimate and synthesise.

Failure 5 — Replication sample is too small

Pattern: replication produces wide uncertainty and is labelled “failure”.

Repair: plan for precision, not merely significance.

Failure 6 — Original effect was a winner’s-curse estimate

Pattern: original effect is unusually large relative to later studies.

Repair: use meta-analytic estimate and publication-bias diagnostics.

Failure 7 — P-value mismatch

Pattern: original p=.04, replication p=.08, declared contradiction.

Repair: compare effect sizes and confidence intervals directly.

Failure 8 — Direction matches but magnitude differs

Pattern: both positive, replication smaller.

Repair: test compatibility and moderators.

Failure 9 — Direction reverses

Pattern: original positive, replication negative.

Repair: inspect sampling uncertainty, context, measurement and effect modification.

Failure 10 — Measurement instrument changed

Pattern: same named construct, different scale.

Repair: test measurement invariance and construct validity.

Failure 11 — Calibration changed

Pattern: physical measurement shifts systematically.

Repair: use standards and cross-calibration.

Failure 12 — Population changed

Pattern: original adults, replication adolescents.

Repair: treat age as potential moderator, not mere nuisance.

Failure 13 — Cultural context changed

Pattern: social effect fails across countries.

Repair: identify cultural mechanisms and test them prospectively.

Failure 14 — Time period changed

Pattern: old relationship disappears years later.

Repair: examine historical change, technology, policy and population drift.

Failure 15 — Implementation fidelity is low

Pattern: replication treatment delivered weakly.

Repair: measure dose and protocol adherence.

Failure 16 — Original procedure relied on tacit knowledge

Pattern: external lab cannot reproduce technical effect.

Repair: improve protocol documentation and training, then retest.

Failure 17 — Analysis differs

Pattern: same data pattern, different preprocessing or model.

Repair: preregister core analysis and run robustness checks.

Failure 18 — Exclusion rules differ

Pattern: effect appears only after selective exclusions.

Repair: disclose and test all defensible exclusion rules.

Failure 19 — Primary outcome differs

Pattern: replication measures a related but different endpoint.

Repair: align outcome or explicitly call it conceptual replication.

Failure 20 — Original had hidden multiple testing

Pattern: one significant outcome among many is replicated as primary.

Repair: account for discovery process and test prospectively.

Failure 21 — Publication bias

Pattern: visible literature contains positive studies only.

Repair: register studies and include unpublished results in synthesis.

Failure 22 — Replication publication bias

Pattern: failed replications are more likely to be public because they are controversial, or successful ones are ignored.

Repair: publish based on design quality rather than result.

Failure 23 — Researcher allegiance

Pattern: effects differ systematically by lab preference or theoretical camp.

Repair: adversarial collaboration and blinding where possible.

Failure 24 — Experimenter expectancy

Pattern: human scoring or participant interaction differs.

Repair: automate, blind or standardise procedures.

Failure 25 — Demand characteristics

Pattern: participants infer the hypothesis differently.

Repair: conceal purpose where ethical, use indirect measures, test demand sensitivity.

Failure 26 — Environmental condition differs

Pattern: temperature, humidity, time of day or noise changes.

Repair: model the environment as a moderator.

Failure 27 — Batch effect

Pattern: biological or material sample batch differs.

Repair: randomise across batches and replicate multiple lots.

Failure 28 — Reagent lot effect

Pattern: laboratory result depends on reagent batch.

Repair: record lot, cross-test and use reference standards.

Failure 29 — Software version effect

Pattern: numerical result changes after library update.

Repair: version computational environment and test sensitivity.

Failure 30 — Random seed effect

Pattern: machine-learning result varies widely across initialisations.

Repair: report seed distribution, not one best run.

Failure 31 — Data leakage in original

Pattern: replication on clean data performs much worse.

Repair: audit temporal and target leakage.

Failure 32 — Benchmark contamination

Pattern: model appears strong on a repeatedly used benchmark.

Repair: fresh holdout data and prospective evaluation.

Failure 33 — Hidden subgroup

Pattern: original sample accidentally enriched for responders.

Repair: identify effect modifier and test stratified prediction.

Failure 34 — Effect heterogeneity

Pattern: replications vary systematically by context.

Repair: use hierarchical models and planned moderator analysis.

Failure 35 — Ceiling or floor effect

Pattern: replication population begins near scale limit.

Repair: use sensitive measurement and suitable population.

Failure 36 — Learning or practice effects

Pattern: repeated testing changes participant performance.

Repair: counterbalance or use fresh participants.

Failure 37 — Carryover in crossover studies

Pattern: first condition affects second.

Repair: washout, counterbalancing or parallel design.

Failure 38 — Contamination between groups

Pattern: control participants receive treatment elements.

Repair: cluster design or stronger separation.

Failure 39 — Replication effect itself

Pattern: researchers or participants behave differently because the effect is now famous.

Repair: consider whether publicity altered the phenomenon.

Failure 40 — True false positive

Pattern: high-quality replications repeatedly show no meaningful effect.

Repair: reduce confidence, revise theory and stop protecting the original claim.

---

115. Replication Practice Bank: 30 reusable prompts

Practice 1 — One result or a finding?

Give one study. Ask what new evidence would be needed before treating it as a stable finding.

Practice 2 — Direct or conceptual?

Classify a replication that changes the measurement but tests the same theoretical relation.

Practice 3 — Reproducible or replicable?

Same code and data produce same result: reproducibility. New data produce consistent result: replication.

Practice 4 — Repeatability or replication?

Three measurements by the same student are repeatability; a new group running a new study is replication.

Practice 5 — Identify essential features

From a protocol, mark which details are theoretically necessary and which are cosmetic.

Practice 6 — P-value trap

Original p=.04, replication p=.06. Ask whether effect estimates actually differ.

Practice 7 — Effect-size compatibility

Compare two confidence intervals rather than significance labels.

Practice 8 — Plan replication size

Use a smallest effect of interest rather than the original point estimate.

Practice 9 — Hidden moderator

Original works, replication fails. Generate three context variables that could moderate the effect.

Practice 10 — Measurement invariance

Ask whether the same scale means the same thing across two populations.

Practice 11 — Replication ladder

Design: same lab → independent lab → new population → new method.

Practice 12 — Multi-lab design

Choose one core protocol and three planned lab-level variations.

Practice 13 — Adversarial collaboration

Two opposing researchers agree on method and interpretation criteria before data.

Practice 14 — Preregister the replication

Specify hypothesis, sample, primary outcome and analysis before collection.

Practice 15 — Publication bias thought experiment

Twenty studies, one significant. Ask what the literature looks like if only one is published.

Practice 16 — Winner’s curse

Explain why the first dramatic effect may be larger than later estimates.

Practice 17 — Meta-analysis

Combine original and replications rather than picking one as “winner”.

Practice 18 — Heterogeneity

Ask whether between-study differences are random noise or systematic moderators.

Practice 19 — Protocol drift

Compare original and replication methods line by line for theory-relevant changes.

Practice 20 — Tacit knowledge

Ask what skilled procedure is not captured in the written protocol.

Practice 21 — Open materials

List what should be shared so another lab can implement the study accurately.

Practice 22 — Null replication

Decide whether the replication can rule out the original claimed magnitude.

Practice 23 — Conceptual generalisation

Change surface context while preserving underlying mechanism.

Practice 24 — Cross-population replication

Identify which variables might modify the effect in the new population.

Practice 25 — Replication of mechanism

Test the proposed mediator in an independent experiment.

Practice 26 — Replication of prediction

Use the original model to predict new data prospectively.

Practice 27 — Replication audit

Check sample, measurement, procedure, analysis, context and theory.

Practice 28 — Failure diagnosis

Rank possible explanations for non-replication before declaring the original false.

Practice 29 — Boundary revision

Rewrite the claim to include the moderator discovered by replication.

Practice 30 — Full replication cycle

Original claim → preregistration → new data → effect comparison → synthesis → theory update.

---

116. Replication rubric

Dimension	Weak	Developing	Strong
Question match	different question	partly aligned	same scientific target clear
Protocol fidelity	essential features changed	some uncertainty	theory-relevant features preserved
Independence	same data	new data same team	new data independent team
Power/precision	too small	moderate	precision matched to scientific threshold
Transparency	analysis hidden	partial plan	preregistered and fully reported
Comparison	p-value labels	effect-size comparison	effect, uncertainty and synthesis
Heterogeneity	ignored	noted	planned moderator analysis
Update	binary success/fail	confidence updated	theory/domain revised prospectively

---

117. The Replication Lifecycle

Stage 1 — Original finding

A study reports an effect, relationship, mechanism or prediction.

Stage 2 — Claim extraction

Before replication begins, define exactly what is being claimed.

Is the claim:

• existence of an effect?
• effect magnitude?
• causal mechanism?
• predictive accuracy?
• generalisation across populations?

Stage 3 — Protocol reconstruction

Identify essential design features, materials, measures, timing, exclusion rules and analysis.

Stage 4 — Replication design

Choose whether the goal is:

• direct replication;
• conceptual replication;
• multi-lab replication;
• mechanism replication;
• transport replication.

Stage 5 — Prospective commitment

Preregister the key design and analysis so the replication cannot silently adapt itself after seeing the outcome.

Stage 6 — New data

Collect genuinely new observations.

This is the defining empirical step.

Stage 7 — Effect comparison

Compare:

• direction;
• magnitude;
• uncertainty;
• scientific threshold;
• predicted range.

Stage 8 — Failure diagnosis

If results differ, inspect sampling, measurement, protocol, population, analysis and context.

Stage 9 — Evidence synthesis

Combine original and replication evidence rather than declaring one study the winner automatically.

Stage 10 — Theory update

Update the scientific claim:

• stronger support;
• smaller effect;
• context-specific effect;
• no meaningful effect;
• new moderator;
• new mechanism.

Stage 11 — Further replication

If uncertainty remains, design the next test around the remaining disagreement.

Stage 12 — Mature finding

A finding becomes mature when its magnitude, variability, boundaries and failure modes are reasonably mapped across independent evidence.

---

118. Replication Design Manual

Decision 1 — What exactly are we replicating?

Write one sentence stating the scientific claim. Do not replicate a vague headline.

Decision 2 — Which parts of the original method are essential?

Use theory and original-author consultation where appropriate to separate core features from incidental detail.

Decision 3 — Do we want direct or conceptual replication?

Direct replication tests repeatability of the finding under similar conditions. Conceptual replication tests generality of the underlying idea.

Decision 4 — How independent should the replication be?

Same lab can test data independence. New lab adds procedural independence. Multi-lab adds distributional evidence.

Decision 5 — What effect size matters?

Do not power only for the original estimate. Define a scientifically meaningful threshold.

Decision 6 — How much precision do we need?

Choose sample size so the confidence interval can distinguish meaningful from negligible effects.

Decision 7 — What is the primary outcome?

Pre-specify the outcome most directly linked to the original claim.

Decision 8 — Which analysis is primary?

Match the original where scientifically justified, but correct known errors transparently.

Decision 9 — What changes are intentional?

Document planned differences as tests of generality or moderation.

Decision 10 — What counts as compatible?

Predefine the replication assessment: effect interval, equivalence margin, meta-analysis or prediction interval.

Decision 11 — Will we consult the original authors?

Consultation can improve fidelity. Independence can be protected through preregistration and transparent analysis.

Decision 12 — Will original materials be used?

Using original materials improves direct fidelity. New materials can support conceptual replication.

Decision 13 — How will protocol fidelity be measured?

Record treatment dose, timing, adherence and manipulation checks.

Decision 14 — How will deviations be handled?

Predefine which deviations trigger exclusion, sensitivity analysis or protocol note.

Decision 15 — How will results be published?

Commit to reporting regardless of whether the original result is supported.

---

119. Replication by field: physics

Replication in physics may involve:

• new apparatus;
• independent detector;
• new run;
• different laboratory;
• new natural event.

Measurement precision can be extremely high, so tiny discrepancies matter.

120. Replication by field: chemistry

Chemistry replication depends on:

• reagent purity;
• temperature;
• pressure;
• concentration;
• equipment;
• reaction time.

Lot and handling effects can be substantial.

121. Replication by field: biology

Biology contains natural variability.

Replication should account for:

• strain;
• sex;
• age;
• environment;
• batch;
• microbiome;
• assay variability.

Exact numerical duplication is rarely expected.

122. Replication by field: medicine

Medical replication can test:

• efficacy;
• effectiveness;
• adverse events;
• mechanism;
• subgroups.

Large multi-centre trials are a powerful form of replication across settings.

123. Replication by field: epidemiology

Different cohorts can replicate associations or causal estimates.

Key challenges include:

• exposure definitions;
• confounding structures;
• outcome coding;
• population differences.

124. Replication by field: psychology

Psychology replication often must manage:

• demand characteristics;
• culture;
• measurement scales;
• experimenter effects;
• participant pools.

Multi-lab replication has been especially important for estimating effect stability.

125. Replication by field: education

Education effects can depend on:

• teacher implementation;
• school culture;
• curriculum;
• student prior knowledge;
• dosage.

Replication across schools should measure implementation fidelity rather than treating school as irrelevant noise.

126. Replication by field: economics

Economic replications can involve:

• re-analysis of original data;
• new datasets;
• new jurisdictions;
• new policy episodes;
• new field experiments.

Re-analysis belongs to reproducibility; new observations belong to replication.

127. Replication by field: ecology

Exact environmental duplication is often impossible.

Replication can use:

• multiple sites;
• multiple years;
• multiple species;
• planned environmental gradients.

Heterogeneity is part of the science.

128. Replication by field: climate science

Researchers cannot replicate the Earth’s entire climate history.

Replication comes through:

• independent observational datasets;
• different model systems;
• different time periods;
• physical fingerprints;
• paleoclimate records.

129. Replication by field: geology

Geological claims can be replicated through:

• new sites;
• independent dating methods;
• new samples;
• new imaging;
• field mapping.

130. Replication by field: astronomy

Astronomical replication can mean:

• repeated events;
• independent telescopes;
• different wavelengths;
• another survey;
• new objects predicted to show the same relationship.

131. Replication by field: computer science

Separate:

• same code/data;
• independent reimplementation;
• new benchmark;
• new hardware;
• real-world deployment.

These test different layers.

132. Replication by field: machine learning

Replication should account for:

• random seeds;
• training data;
• hyperparameters;
• compute budget;
• benchmark contamination;
• distribution shift.

One best run is not a stable scientific result.

133. Replication by field: qualitative research

Qualitative work may not seek identical numerical effects.

Instead, transparent methods can support:

• credibility;
• dependability;
• transferability;
• auditability.

Replication language should respect the epistemic goals of the field.

---

134. Replication and ethics

Replication is not ethically neutral.

Repeating a study can expose participants to:

• risk;
• burden;
• privacy loss;
• withheld treatment.

Replication value must justify those costs.

135. Do not repeat harmful exposure unnecessarily

If a causal relationship is already extremely well established and the replication would deliberately expose participants to serious harm, repeating the experiment may be unethical.

Use observational or mechanistic evidence instead.

136. Replication can reduce harm

Replicating high-stakes biomedical findings can prevent:

• ineffective treatment;
• misleading policy;
• wasted resources.

Confirmation itself can have ethical value.

137. Animal research replication

Replication involving animals should consider:

• necessity;
• sample-size planning;
• replacement;
• reduction;
• refinement.

A poorly planned replication wastes animals and produces little scientific value.

138. Replication and participant consent

Replications should meet the same ethical standards as original studies.

A claim’s importance does not override informed consent or risk protections.

139. Data-sharing ethics

Reproducibility and replication benefit from shared data and materials.

But sharing must respect:

• privacy;
• consent;
• legal constraints;
• indigenous or community data governance where relevant.

Open science should not mean careless data release.

---

140. Replication incentives

Scientific careers often reward first discovery more than confirmation.

This can produce:

• too many novelty studies;
• too few replications;
• underpowered follow-up;
• publication imbalance.

141. Reward replication as infrastructure

Replication deserves:

• funding;
• publication;
• career credit;
• shared platforms;
• methods support.

Reliable science depends on maintenance, not only invention.

142. Replication databases

Fields can maintain registries of:

• original findings;
• replication attempts;
• protocols;
• outcomes;
• synthesis.

This prevents repeated rediscovery of the same uncertainty.

143. Replication priorities should be transparent

Fields can rank replication targets by:

• scientific importance;
• uncertainty;
• policy impact;
• sample size;
• method fragility;
• citation influence.

Replication strategy is a resource-allocation problem.

---

144. The Replication Notebook

A replication notebook keeps the scientific comparison visible from the beginning. Its purpose is to prevent a replication from becoming a post-hoc argument about whether two studies “look similar enough”.

Notebook field 1 — Original scientific claim

Write the exact claim being tested.

Examples:

• “Treatment A improves Outcome Y.”
• “Method R produces better delayed retention than rereading.”
• “Model M predicts new cases more accurately than Baseline B.”

Notebook field 2 — Original effect estimate

Record:

• effect size;
• uncertainty interval;
• sample size;
• primary outcome;
• analysis.

Notebook field 3 — Original context

Record:

• population;
• location;
• time period;
• equipment;
• treatment version;
• setting.

Notebook field 4 — Replication type

Choose:

• direct;
• conceptual;
• multi-lab;
• mechanism;
• transport;
• prediction.

Notebook field 5 — Essential features

List the method details believed to be theoretically necessary.

Notebook field 6 — Intentional changes

Document which features differ and why.

This prevents accidental conceptual replication from being called direct replication.

Notebook field 7 — Sample plan

Record:

• target sample;
• precision goal;
• smallest effect of interest;
• stopping rule.

Notebook field 8 — Primary outcome

Define the outcome that directly tests the original claim.

Notebook field 9 — Analysis plan

Pre-specify the main analysis and any planned robustness checks.

Notebook field 10 — Compatibility criterion

State how replication will be judged:

• effect interval;
• prediction interval;
• equivalence;
• meta-analysis;
• Bayesian evidence.

Notebook field 11 — Protocol fidelity

Record whether the intended intervention or procedure was delivered as planned.

Notebook field 12 — Deviations

Document deviations before looking at whether they help explain the result.

Notebook field 13 — Replication result

Record effect, uncertainty and raw scientific pattern.

Notebook field 14 — Original–replication comparison

Compare magnitude and compatibility directly.

Notebook field 15 — Heterogeneity hypotheses

If results differ, list possible moderators before testing them extensively.

Notebook field 16 — Synthesis

Combine the evidence where appropriate.

Notebook field 17 — Theory update

State how the scientific claim changed.

Notebook field 18 — Next replication

Design the next study to target the largest unresolved uncertainty.

---

145. Twenty replication audit questions

1. What exact scientific claim is being replicated?
2. Is this direct or conceptual replication?
3. What are the theory-relevant features?
4. Were new data collected?
5. Is the replication sufficiently independent?
6. Was sample size planned for useful precision?
7. Was the primary outcome defined beforehand?
8. Were analysis choices preregistered?
9. Were original materials available?
10. Were protocol deviations recorded?
11. Was implementation fidelity measured?
12. Are effect sizes compared directly?
13. Are uncertainty intervals compared?
14. Was replication judged only by p-values?
15. Could population differences moderate the effect?
16. Could measurement differ across studies?
17. Could publication or selection bias inflate the original?
18. Was evidence synthesized rather than treated as winner-take-all?
19. Did the replication change theory scope?
20. What new study would most reduce remaining uncertainty?

---

146. Twenty myths about replication

Myth 1 — A replication must produce the exact same number

False. New data contain new sampling variation.

Myth 2 — Same p-value means successful replication

False. Effect sizes can differ dramatically despite both being significant.

Myth 3 — Different p-value means failed replication

False. Similar effects can cross an arbitrary threshold differently.

Myth 4 — A failed replication proves the original study was fraudulent

False. Many scientific reasons can produce disagreement.

Myth 5 — A successful replication proves the original theory

False. It strengthens one part of the evidence.

Myth 6 — Direct replication is always better than conceptual replication

False. They answer different questions.

Myth 7 — Conceptual replication can replace direct replication

False. If the original effect itself is uncertain, a close repeat is important.

Myth 8 — Re-analysis of the same data is replication

Under the terminology used here, it is reproducibility or robustness analysis, not new-data replication.

Myth 9 — Repeating one measurement three times is a replication study

No. It is repeated measurement or repeatability within a study.

Myth 10 — A larger replication must be right

Larger samples reduce sampling error but do not fix design or measurement bias.

Myth 11 — Original authors should never be involved

False. They can clarify method; independence can be maintained through transparent prospective design.

Myth 12 — Original authors must be involved

False. A well-described scientific result should be independently testable.

Myth 13 — Replication should copy every detail

False. It should preserve theoretically important details.

Myth 14 — Context explanations are always excuses

False. Context dependence can be real, but it should generate specific testable moderator hypotheses.

Myth 15 — Non-replication means the effect is exactly zero

False. The replication may be imprecise or the true effect may be smaller.

Myth 16 — Replication is boring science

False. Replication can discover boundaries, moderators and mechanisms.

Myth 17 — Only controversial studies need replication

False. Foundational high-use findings deserve confirmation too.

Myth 18 — Replication wastes resources

Poorly targeted replication can; strategic replication protects far larger downstream investments.

Myth 19 — Open data guarantee replicability

Open data support reproducibility. Replication still requires new evidence.

Myth 20 — Science failed because some findings fail to replicate

Replication failures are evidence that science’s correction mechanisms are working, provided fields respond transparently.

---

147. Replication Glossary

Conceptual replication

A new study testing the same underlying scientific claim using a different method, measure, manipulation or context.

Direct replication

A new study attempting to repeat the original study’s essential design as closely as practical using new data.

Effect-size shrinkage

The tendency for later estimates to be smaller than an initially large reported estimate.

Generalisability

The extent to which a result applies to populations or contexts beyond those directly studied.

Heterogeneity

Real variation in underlying effect sizes across studies or contexts beyond sampling error.

Implementation fidelity

The degree to which an intervention or protocol was delivered as intended.

Independent replication

A replication performed by researchers or laboratories independent of the original team.

Meta-analysis

Statistical synthesis of results across studies to estimate combined effects and heterogeneity.

Multi-lab replication

A coordinated replication conducted across several laboratories or sites.

Non-replication

A replication result that is meaningfully inconsistent with the original claim under the chosen scientific criteria.

Open materials

Study materials, protocols, instruments and code shared so others can inspect or reuse them.

Preregistration

Time-stamped documentation of hypotheses, methods and analysis plans before outcomes are known.

Prediction interval

An interval describing where the effect estimate of a future similar study may plausibly fall under a meta-analytic model.

Registered report

A publication format in which study rationale and methods are peer-reviewed before results are known.

Replicability

Under the National Academies convention used here, obtaining consistent results across studies addressing the same scientific question with new data.

Replication

The act of conducting a new study with new data to test whether a prior scientific result is consistent.

Repeatability

A term often used for repeated measurement under the same or very similar conditions; terminology varies by field.

Reproducibility

Under the National Academies convention used here, obtaining consistent computational results using the same data, code, methods and conditions of analysis.

Robustness

The stability of a result across reasonable analytic choices, models or assumptions.

Small-study effect

A pattern in which smaller studies show systematically different, often larger, effect estimates.

Winner’s curse

Inflation of the first selected or most extreme effect estimate because selection favours unusually strong results.

---

148. Advanced FAQ: Scientific Replication

1. Why replicate a study if it was peer reviewed?

Peer review evaluates plausibility and method before or around publication. It does not provide independent new data.

2. Why replicate a statistically significant result?

Significance does not guarantee stability, magnitude or generalisability.

3. Why replicate a large study?

Large samples reduce sampling error but can still suffer measurement bias, design errors or context dependence.

4. Why replicate a randomized trial?

To test whether the causal effect appears again in another sample or setting and to map heterogeneity.

5. Why replicate an observational study?

To see whether the association or causal estimate persists under new data and possibly different confounding structures.

6. Is a meta-analysis a replication?

No. It synthesizes existing studies. It can include replications.

7. Is re-running the original code a replication?

Not under the terminology used here. That is computational reproducibility.

8. Is repeating measurements within one experiment replication?

Usually that is repeatability or technical replication, not independent study replication.

9. What is a technical replicate?

Repeated measurement of the same biological or physical sample to estimate measurement variability.

10. What is a biological replicate?

Independent biological samples representing separate experimental units.

11. Why does pseudoreplication matter?

Many measurements of one experimental unit do not create many independent units.

12. Can the same lab replicate itself?

Yes. New data can test whether the finding repeats, though independent labs add stronger protection against lab-specific effects.

13. How close must a direct replication be?

Close enough that differences do not change the scientific mechanism being tested. Exact cosmetic identity is unnecessary.

14. How different can a conceptual replication be?

Different enough to test generality, but similar enough that the same theoretical claim genuinely predicts the result.

15. What if the original effect is smaller in replication?

Update the estimated magnitude. The effect can replicate while shrinking.

16. What if the replication is significant but smaller?

Do not call it “successful” without discussing the magnitude difference and whether the original was inflated.

17. What if the replication is non-significant but same magnitude?

The replication may simply be imprecise. Compare intervals and power.

18. What if the replication is significant in the opposite direction?

Investigate heterogeneity, measurement, method and sampling; this is a serious contradiction.

19. What if two replications disagree with each other?

Use meta-analysis and moderator analysis. One effect may vary across settings.

20. How many replications are enough?

No universal number. It depends on effect size, heterogeneity, stakes, measurement quality and theory importance.

21. Should every paper be replicated?

Not necessarily. Replication resources should be strategically allocated.

22. Should surprising findings be replicated first?

Often yes, especially when they have large theoretical or practical implications.

23. Should boring foundational findings be replicated?

Yes when many later conclusions depend on them.

24. Can a replication be novel science?

Absolutely. It can reveal moderators, boundaries, bias and improved effect estimates.

25. Can replication change theory?

Yes. Replication is one of the main ways theory scope becomes clearer.

26. Is replication always independent?

No. Independence varies. The level should be reported.

27. Can industry replicate academic findings?

Yes. Conflicts of interest should be transparent, but evidence is evaluated by design and data.

28. Can academic teams replicate industry findings?

Yes. Independent external tests can be particularly valuable for high-stakes claims.

29. What does “failed to replicate” really mean?

It should mean the new data were meaningfully inconsistent with the original scientific claim under explicit criteria—not merely that one p-value crossed .05 and another did not.

30. What is the final replication question?

Does the finding survive when new data, new samples and ideally new researchers ask the same scientific question again?

---

149. Replication Interpretation Matrix

Replication becomes scientifically powerful when the comparison is structured. The matrix below separates several common patterns that are often collapsed into one word: “replicated”.

Original	Replication	Possible interpretation	Next step
large positive, precise	similar positive, precise	strong support	test generality and mechanism
large positive, imprecise	smaller positive, precise	effect likely real but original inflated	update magnitude
positive	near zero, precise	serious challenge	audit method/context; synthesize
positive	opposite, precise	strong contradiction or moderator	test heterogeneity directly
positive	near zero, imprecise	uninformative replication	collect more precise data
positive	positive only in subgroup	possible moderator	prospective subgroup replication
positive	positive with different measure	conceptual support	test more representations
positive	positive across many labs	robust cross-site support	map remaining boundaries

150. Compatible does not mean identical

Two studies can be scientifically compatible when their point estimates differ.

Compatibility depends on:

• uncertainty;
• expected heterogeneity;
• measurement precision;
• effect definition.

The goal is not numerical cloning.

151. Incompatibility can be scientifically productive

A large replicated disagreement can reveal:

• hidden moderator;
• population dependence;
• measurement non-equivalence;
• wrong theory;
• original bias.

Disagreement is not wasted research.

152. One replication should rarely erase the full evidence base

Scientific claims should update cumulatively.

One strong negative replication matters greatly.

But if twenty independent studies support the effect, the correct response is to ask why the new study differs rather than erase the entire literature instantly.

153. One original study should not outweigh many replications

The reverse also applies.

If one small original study is followed by many well-powered null replications, continued attachment to the first estimate becomes difficult to justify.

154. Replication confidence is cumulative

A practical confidence sequence is:

single finding → repeated same-lab finding → independent direct replication → multi-lab replication → conceptual replication → cross-population replication → mature evidence base

Each step adds a different form of independence.

---

155. Replication Design Workbook

The following transformations show how to turn an original finding into a replication programme rather than a single repeated experiment.

Workbook 1 — Plant-growth effect

Original: Fertiliser F increases plant dry mass.

Direct replication: new plants, same species, same dose, same environmental conditions.

Independent replication: second lab repeats.

Conceptual replication: another growth measure such as biomass or leaf area.

Boundary replication: different soil nutrient levels.

Theory update: effect exists only under nutrient limitation.

Workbook 2 — Retrieval-practice effect

Original: retrieval practice improves delayed recall relative to rereading.

Direct replication: same material, new student sample.

Conceptual replication: different subject matter.

Transfer replication: delayed novel questions rather than same-item recall.

Moderator test: prior knowledge.

Workbook 3 — Small-group tuition

Original: small-group programme improves assessment performance.

Replication programme:

• another tutor;
• another school level;
• same curriculum;
• same lesson duration;
• measure prompt count and feedback frequency.

Replication should test both outcome and proposed mechanism.

Workbook 4 — Drug efficacy

Original: randomized trial shows benefit.

Replication programme:

• another trial;
• another country;
• real-world effectiveness;
• mechanism biomarker;
• longer follow-up.

A mature claim includes efficacy, transport and durability.

Workbook 5 — Diagnostic test

Original: test has high accuracy in development dataset.

Replication: external cohort with frozen threshold.

Conceptual replication: different lab assay.

Operational replication: routine clinical workflow.

Development performance should not be treated as external replication.

Workbook 6 — Genetic association

Original: Variant V associated with Disease D.

Replication: independent cohort with pre-specified variant and model.

Cross-ancestry replication: test another population.

Mechanistic replication: functional study of gene pathway.

Workbook 7 — Psychology effect

Original: manipulation changes behaviour.

Replication: preregistered high-powered direct repeat.

Multi-lab replication: common protocol across sites.

Conceptual replication: alternate manipulation of same construct.

Measurement replication: behavioural and physiological outcomes.

Workbook 8 — Policy effect

Original: policy associated with outcome improvement.

Replication: another jurisdiction adopting similar policy.

Design replication: use another quasi-experimental strategy.

Mechanism replication: test the pathway through which policy should operate.

Workbook 9 — Climate relationship

Original: model predicts a spatial fingerprint.

Replication: new observational period.

Dataset replication: independent instrument.

Model replication: independent climate-model family.

Historical replication: paleoclimate evidence where relevant.

Workbook 10 — Engineering fatigue

Original: fatigue life follows a stress–cycle relationship.

Replication: new specimens, new batch, independent machine.

Boundary replication: higher temperature.

Material replication: related alloy.

Workbook 11 — Machine-learning benchmark

Original: Model M beats benchmark.

Reproducibility: same code/data result.

Replication: independent implementation or new data.

Generalisation: new domain.

Robustness: multiple seeds and preprocessing choices.

All four should be reported separately.

Workbook 12 — AI medical model

Original: retrospective hospital data show strong prediction.

Replication ladder:

1. temporal holdout;
2. external hospital;
3. prospective silent deployment;
4. live clinical deployment.

Each step removes another source of hidden dependence.

Workbook 13 — Social-media association

Original: usage correlates with wellbeing.

Replication: new cohort.

Measurement replication: device logs rather than self-report.

Causal replication: randomized reduction intervention.

Different replications can strengthen different scientific claims.

Workbook 14 — Ecological effect

Original: predator removal changes prey population.

Replication: another site.

Longitudinal replication: another season/year.

Mechanism replication: behavioural or food-web pathway.

Workbook 15 — Archaeological dating

Original: Method A dates layer to Period P.

Replication: new sample same method.

Conceptual replication: independent Method B.

Site replication: related layer elsewhere.

Workbook 16 — Exoplanet transit

Original: one periodic signal.

Replication: repeated transit.

Instrument replication: independent telescope.

Method replication: radial velocity.

Multiple independent signals reduce false-positive explanations.

Workbook 17 — New particle signal

Original: excess appears in one channel.

Replication: more data.

Detector replication: independent experiment.

Channel replication: another decay mode.

Discovery becomes stronger as independent channels converge.

Workbook 18 — Nutritional effect

Original: dietary component associated with health outcome.

Replication: another cohort.

Measurement replication: biomarkers.

Intervention replication: randomized feeding trial where feasible.

Workbook 19 — Public-health campaign

Original: one region improves.

Replication: another region using same campaign.

Mechanism: measure behaviour change.

Implementation: verify exposure dosage.

Workbook 20 — Sensor performance

Original: sensor meets precision claim.

Replication: another unit, another lab, environmental stress.

Generalisation: broader temperature/humidity range.

---

156. Replication and prediction

One of the strongest ways to replicate a scientific claim is to use the original model prospectively.

Do not refit it.

Do not change the threshold.

Ask:

Does the original claim predict genuinely new data?

157. Prediction replication is stricter than refitting

If a model is re-estimated on the replication data, it may recover good fit even if original parameters were unstable.

A frozen model creates a cleaner test of transport.

158. Calibration replication

A probabilistic model should preserve calibration.

If cases predicted at 70% occur only 30% of the time in the new sample, the model does not replicate operationally.

159. Discrimination replication

A model may preserve ranking but lose calibration.

Both dimensions should be checked.

160. Decision replication

Even if predictive metrics replicate, the decision benefit may not.

Costs, prevalence and workflow can differ.

Operational replication should measure real decision value.

---

161. Replication and causation

Causal claims deserve replication at several levels:

• total effect;
• effect size;
• mechanism;
• heterogeneity;
• transport.

162. A causal effect can replicate while mechanism fails

If Treatment X consistently improves Y but Mediator M does not reproduce, update the mechanism without discarding the effect.

163. Mechanism can replicate while total effect varies

A pathway can operate in multiple populations but contribute different amounts depending on background conditions.

164. Replication strengthens causal transport

If the same intervention works across populations with different confounder structures, causal confidence and generalisability increase.

---

165. Replication Synthesis Standard

Replication becomes mature science when individual studies are assembled into a coherent evidence base. A field should not keep asking “Did Study 2 match Study 1?” forever. It should eventually ask:

What is the best estimate of the effect, how much does it vary, which contexts modify it, and which explanations remain viable after all replications are considered?

166. Synthesis begins by aligning the scientific question

Before combining studies, check whether they truly estimate the same thing.

Align:

• population;
• intervention or exposure;
• comparator;
• outcome;
• time horizon;
• effect scale.

Combining unlike questions can produce a precise answer to no coherent scientific question.

167. Effect harmonisation

Studies may report:

• risk ratios;
• odds ratios;
• mean differences;
• standardised effects;
• correlations.

Meta-analysis may require transformation to a common scale.

That transformation should preserve meaning.

168. Study weighting

More precise studies usually receive greater weight in meta-analysis.

This reflects information, not status.

A famous study should not receive extra weight merely because it is famous.

169. Heterogeneity is part of the finding

A pooled average of 0.30 can hide:

• 0.80 in one context;
• 0.20 in another;
• 0.00 in another.

Report both:

• average effect;
• between-study variation.

170. Prediction intervals show future-study uncertainty

A prediction interval answers:

“If we ran another similar study, where might its effect estimate plausibly fall?”

This is often more useful for replication planning than the pooled mean alone.

171. Meta-analysis can be wrong if the inputs are biased

A meta-analysis of selectively published studies can produce a precise biased estimate.

Garbage in, precision out.

Synthesis should inspect:

• study registration;
• publication bias;
• risk of bias;
• selective outcome reporting;
• duplicated datasets.

172. Duplicate data can masquerade as independent replication

Two papers may use:

• the same cohort;
• overlapping participants;
• the same database.

They should not automatically be counted as independent studies.

173. Living meta-analysis

In fast-moving fields, synthesis can be updated as new replications arrive.

A living meta-analysis:

• adds new studies;
• updates pooled estimates;
• updates heterogeneity;
• updates bias assessment.

This turns replication into continuous evidence maintenance.

174. Bayesian evidence synthesis

Bayesian meta-analysis can:

• combine prior evidence;
• model heterogeneity;
• produce posterior distributions;
• compare models.

The conclusions depend on the model and priors, which should be transparent.

175. Cumulative meta-analysis

Cumulative meta-analysis asks how the evidence changed as studies accumulated through time.

It can reveal:

• when a finding became stable;
• whether early estimates were inflated;
• whether later replications changed direction.

176. Replication can reduce unnecessary research

Once a finding is extremely stable, repeating nearly identical studies may add little value.

Resources can shift toward:

• new populations;
• mechanisms;
• boundaries;
• long-term outcomes.

Replication strategy includes knowing when enough close repetition has been achieved.

---

177. The Replication Confidence Ladder

Level 1 — Single report

One study, one dataset, one team.

Level 2 — Internal repeat

Same team obtains similar new-data result.

Level 3 — Independent direct replication

Another team repeats the essential design.

Level 4 — Multi-lab replication

The effect survives multiple sites.

Level 5 — Conceptual replication

The effect survives alternate measurements or manipulations.

Level 6 — Cross-population replication

The effect appears across relevant populations.

Level 7 — Mechanistic replication

The proposed pathway survives independent tests.

Level 8 — Predictive replication

The original model predicts future unseen data.

Level 9 — Meta-analytic stability

Combined evidence is precise and heterogeneity is understood.

Level 10 — Mature scientific finding

Effect, mechanism, boundaries and variation are sufficiently mapped that the claim can support theory or application reliably.

This ladder is descriptive, not a formal universal ranking. Different sciences reach confidence through different combinations of evidence.

---

178. Replication Governance

High-impact fields benefit from explicit replication governance.

Questions include:

• which claims deserve priority?
• who funds the replication?
• who owns materials?
• how are disputes resolved?
• how are null replications published?
• how are original and replication teams credited?

179. Replication conflicts should be resolved prospectively where possible

When teams disagree, define:

• protocol;
• sample;
• analysis;
• interpretation criteria.

before the next data arrive.

This moves debate from rhetoric to prediction.

180. Replication committees

Large collaborations can use an independent methods committee to:

• review protocol fidelity;
• audit analysis;
• protect blinding;
• adjudicate deviations.

This is especially valuable when the scientific dispute is high profile.

181. Replication and conflicts of interest

Both original and replication teams may have:

• financial interests;
• theoretical commitments;
• professional incentives.

Conflict disclosure does not invalidate results.

It helps readers assess incentives and encourages independent checks.

182. Blinded replication analysis

Analysts can sometimes be blinded to:

• group labels;
• expected direction;
• site identity.

This reduces unconscious flexibility.

183. Replication data monitoring

For long or high-risk studies, monitoring can identify:

• protocol drift;
• unexpected harms;
• data-quality problems.

Monitoring rules should be pre-specified.

184. Replication stop rules

A replication programme may stop when:

• effect is estimated precisely enough;
• scientifically meaningful alternatives are excluded;
• remaining uncertainty is about generality rather than existence;
• further identical studies have low information value.

185. Replication reopen rules

A mature finding should be reopened when:

• new technology changes measurement;
• new population reveals different effect;
• new theory predicts a boundary;
• an influential replication contradicts the consensus;
• publication bias is discovered.

Scientific maintenance never ends completely.

---

186. Replication and generalisation

A replicated finding is not automatically universal.

Generalisability requires testing variation.

Ask:

• does effect travel across populations?
• does it travel across time?
• does it travel across implementation?
• does it travel across measurement?

187. Planned heterogeneity

Researchers can intentionally vary:

• sites;
• participants;
• stimuli;
• materials;
• conditions.

This creates a more informative estimate of generality.

188. Random-effects design begins at data collection

Heterogeneity should not be treated only as a statistical problem after the studies are done.

Design the replication programme to sample meaningful variation.

189. Multisite random-effects experiments

A treatment can be tested across many sites with site-specific effects estimated hierarchically.

This provides:

• average effect;
• site variation;
• moderator hypotheses.

190. Many-lab projects

Many-lab projects can:

• standardise core protocol;
• sample diverse populations;
• estimate lab-level variance;
• reduce one-lab dependence.

They are a powerful model for high-priority effects.

191. Generalisation requires theory

If a finding replicates in Population A but not B, theory should explain why.

A list of successful and failed sites is not enough.

Scientists should identify the moderator that makes the variation coherent.

192. Replication can transform a universal claim into a conditional one

Early:

“Method X improves outcome.”

Mature:

“Method X improves outcome for novices under high-feedback conditions but shows little benefit for experts.”

The mature statement is more useful precisely because replications revealed boundaries.

---

193. Replication and measurement science

Measurement quality can determine whether replication is even interpretable.

Before comparing studies, ask:

• same construct?
• same units?
• same calibration?
• same threshold?
• same measurement precision?

194. Measurement error attenuates replication

If one study uses noisy measurement, effect estimates may shrink toward zero or become unstable.

Differences in measurement reliability can create apparent non-replication.

195. Calibration standards improve replication

Physical sciences use reference standards so measurements across labs remain comparable.

Biological and social sciences also benefit from:

• reference samples;
• validated scales;
• standard protocols;
• shared coding rules.

196. Measurement invariance in social and behavioural research

A questionnaire can produce the same numeric score while representing slightly different constructs across cultures or ages.

Measurement-invariance testing helps determine whether replication comparisons are meaningful.

197. Assay drift

Laboratory assays can change with:

• reagent lots;
• instrument software;
• calibration;
• operator training.

Long replication programmes should monitor drift.

---

198. Replication and statistical modelling

A statistical model should not become the hidden definition of replication.

Replicability concerns the scientific result.

Different valid models can sometimes estimate the same effect.

199. Model dependence

If a finding appears under only one narrow specification, that fragility should be reported.

Use:

• robustness checks;
• alternative models;
• specification curves.

200. Replication of qualitative conclusions

Some results are qualitative:

• effect exists;
• direction reverses;
• mechanism present;
• threshold crossed.

Replication criteria should match the claim.

201. Replication of quantitative models

For quantitative models, test:

• parameter stability;
• prediction;
• calibration;
• residual patterns;
• boundary conditions.

202. Parameter replication

If a model estimates parameter θ, later studies can ask whether θ remains stable.

Variation can reveal context dependence.

203. Structural replication

Even if parameter values differ, the same mathematical relationship may persist.

This is a deeper form of replication.

204. Mechanistic replication

If the same causal mechanism is observed with different measurements and interventions, scientific confidence grows strongly.

205. Replication of absence

Replicating a null is hard.

A non-significant result does not prove absence.

Use:

• equivalence tests;
• Bayes factors;
• precision intervals;
• smallest effect thresholds.

206. Strong null evidence needs a capable test

A replication can support practical absence when it is precise enough to exclude effects large enough to matter.

---

207. Replication Outcome Categories

A mature field should avoid one-word labels when replication evidence is more nuanced. The following categories provide a more useful vocabulary.

Outcome A — Strongly replicated

Independent studies with adequate precision find compatible effects, and the combined estimate excludes scientifically negligible alternatives.

Outcome B — Replicated with smaller magnitude

The direction and phenomenon persist, but the original effect was inflated.

Scientific update:

retain the finding; revise expected magnitude.

Outcome C — Replicated conditionally

The effect appears only under a moderator or boundary condition.

Scientific update:

replace universal claim with conditional claim.

Outcome D — Replicated conceptually

The underlying relation appears under different measures or manipulations.

Scientific update:

confidence increases that the result is not a method artifact.

Outcome E — Inconclusive

The replication estimate is too imprecise to distinguish the original effect from negligible alternatives.

Scientific update:

collect better data; do not call success or failure.

Outcome F — Meaningful non-replication

A precise replication excludes the original claimed magnitude and may be compatible with no meaningful effect.

Scientific update:

substantially reduce confidence.

Outcome G — Reversal

The replication finds an effect in the opposite direction with useful precision.

Scientific update:

investigate effect modification, bias or theory failure urgently.

Outcome H — Measurement disagreement

Replication differs primarily because measurement systems do not align.

Scientific update:

resolve construct or calibration differences before theory interpretation.

Outcome I — Implementation disagreement

The treatment or procedure was not delivered in the same meaningful way.

Scientific update:

test treatment fidelity rather than declaring scientific contradiction.

Outcome J — Contextual heterogeneity

Different sites produce genuinely different effect sizes.

Scientific update:

model site or context as part of the phenomenon.

---

208. Claim Narrowing

Replication often improves science by narrowing the claim.

Original:

“X improves Y.”

After replications:

“X improves Y for novices, when feedback is immediate, under doses above D, with little evidence of benefit for experts.”

The later statement looks less dramatic.

It is more scientific because it is more accurate.

209. Claim widening

Replication can also broaden a claim.

Original:

“Effect exists in Population A.”

After cross-population replication:

“Effect appears across A, B and C with similar magnitude.”

Generalisation should be earned empirically.

210. Claim splitting

A single original claim may split into two phenomena.

Example:

• fast effect through Mechanism M1;
• slow effect through Mechanism M2.

Replication across time scales can reveal the hidden structure.

211. Claim merging

Two apparently separate effects can turn out to be manifestations of one mechanism.

Conceptual replications can reveal this unification.

212. Claim retirement

A claim should be retired when:

• multiple strong direct replications fail;
• original analysis is shown to be biased;
• effects shrink below meaningful thresholds;
• no coherent moderator explains disagreement;
• a better theory explains the full evidence.

Retirement is not scientific defeat.

It is evidence doing its job.

213. Claim resurrection

A retired claim can return if new evidence shows:

• hidden moderator;
• measurement failure;
• wrong replication implementation;
• new mechanism.

But resurrection requires new prospective tests.

---

214. Replication as Theory Stress Test

A theory can look strong if every supporting study comes from one laboratory, one population and one measurement.

Replication deliberately removes those supports.

Ask:

• does the effect survive another lab?
• another measure?
• another population?
• another intervention?
• another time?

The more dimensions survive, the more theory-general the finding becomes.

215. The theory should predict replication boundaries

A mature theory should not merely explain why replication failed after the fact.

It should predict:

• where effect weakens;
• which population responds;
• which dosage matters;
• which measurement is sensitive.

Boundary prediction converts context explanations from excuses into science.

216. Failed replication creates a new hypothesis

Suppose original lab observes Effect E and replication does not.

Possible hypotheses:

• Moderator M differs;
• original effect inflated;
• replication treatment weak;
• measurement non-invariant.

The next study should discriminate these hypotheses.

217. Replication is recursive

The cycle is:

claim → replication → discrepancy → moderator hypothesis → new replication → revised claim

This is why replication is a research programme, not a stamp.

---

218. Replication and Research Integrity

Replication is strongest when both original and replication teams adopt transparent integrity standards.

Archive the original protocol

Do not modify the historical record silently.

Archive the replication protocol

Timestamp before data collection.

Report all primary outcomes

Do not hide an inconvenient replication endpoint.

Report deviations

Protocol deviations can explain results and should be visible.

Report exclusions

Readers should know which observations were removed and why.

Report all prespecified analyses

Selective reporting recreates the original problem.

Label exploratory analyses

Unexpected patterns can generate valuable new hypotheses.

Publish regardless of direction

Replication science fails if only dramatic disagreements appear publicly.

219. Replication disputes should preserve data

When teams disagree, share:

• protocol;
• analysis code;
• summary data or de-identified data where possible;
• deviation logs.

Disagreement becomes scientifically productive when others can inspect it.

220. Corrections and retractions

If replication reveals a genuine error in the original paper, responses can include:

• correction;
• expression of concern;
• retraction.

The appropriate action depends on the nature and severity of the problem.

Non-replication alone does not automatically require retraction.

221. Retraction is not the same as scientific falsification

A paper may be retracted because:

• data are unreliable;
• ethical violations;
• major error;
• misconduct.

A well-conducted study can also produce a result later shown to be false without needing retraction.

Science distinguishes honest error from invalid research conduct.

222. Replication and trust

Public trust is not strengthened by pretending science never fails.

It is strengthened when failures are:

• detected;
• published;
• explained;
• corrected.

Replication is visible evidence that scientific knowledge can self-correct.

---

223. Replication Planning Matrix

Original claim	Best first replication	Why
surprising small-study effect	large preregistered direct replication	tests existence and magnitude
high-impact intervention	independent randomized replication	confirms causal effect
predictive model	frozen external test	tests transport
new mechanism	independent pathway experiment	tests explanation
effect in one culture	cross-cultural conceptual replication	tests generality
lab-only biological effect	multi-lab standardised repeat	tests portability
benchmark algorithm	independent reimplementation + new data	separates reproducibility and replication
field observation	new site/time period	tests natural variability

224. Replication resource allocation

When resources are scarce, prioritize claims with high expected information value.

Ask:

• How uncertain is the claim?
• How influential is it?
• What decisions depend on it?
• How expensive is replication?
• How much would the result change theory or policy?

Replication strategy can itself be evidence-driven.

225. Expected value of replication

A replication has high expected value when:

• claim is uncertain;
• consequences are large;
• replication is feasible;
• either outcome would change decisions.

This helps fields choose what to repeat first.

---

226. Replication maturity model

Stage 0 — No replication culture

Novel findings published; failed repeats rarely visible.

Stage 1 — Informal repeats

Labs sometimes repeat findings but protocols are not registered.

Stage 2 — Transparent direct replication

High-priority effects receive preregistered repeats.

Stage 3 — Multi-lab replication

Field estimates lab and context variability.

Stage 4 — Integrated synthesis

Replications feed meta-analysis and theory revision.

Stage 5 — Strategic maintenance

Field maintains registries, living synthesis and replication priorities.

Mature science treats replication as infrastructure.

---

227. Replication in scientific education

Students should learn that science is not a collection of one-time demonstrations.

A classroom can teach:

• repeat measurements;
• independent group tests;
• comparison of variation;
• transparent disagreement;
• revision of claims.

228. Classroom replication question

After each experiment, ask:

If another group performed this tomorrow with new samples, what result should they expect, and how much variation would still be consistent with our conclusion?

This one question turns a classroom demonstration into scientific reasoning.

229. Teach disagreement as data

If two groups differ:

• do not average immediately;
• do not assume one cheated;
• compare method;
• compare measurement;
• look for moderator.

Disagreement is an invitation to investigate.

230. Teach uncertainty with replication

Repeated independent results reveal:

• natural variability;
• measurement error;
• effect stability.

Students learn why one number is rarely the whole answer.

231. Teach claim revision

Original:

“Warm water dissolves faster.”

After class replications:

“Within our tested range and with the same stirring and particle size, warmer water reduced dissolving time.”

The revised claim is more precise.

---

232. Replication versus Reproducibility: the boundary in practice

These two ideas are often blended in ordinary conversation, but separating them creates much clearer scientific diagnosis.

Scenario A — Same code, same data, same result

This demonstrates computational reproducibility.

It does not test whether the scientific finding appears in new data.

Scenario B — Same code, same data, different result

This is a reproducibility problem.

Possible causes include:

• software version;
• random seed;
• undocumented preprocessing;
• missing dependencies.

Scenario C — New data, same design, similar result

This is direct replication.

Scenario D — New data, different design, same theoretical relationship

This is conceptual replication.

Scenario E — Same data, different valid analysis, different conclusion

This is an analytic robustness problem.

It may reveal model dependence.

Scenario F — New population, same effect

This provides replication plus evidence of generalisability.

Scenario G — New population, different effect

This suggests heterogeneity or a boundary condition.

233. Reproducibility supports replication

A computationally reproducible original study makes replication easier because others can inspect:

• analysis;
• data transformations;
• exclusions;
• reported statistics.

But reproducibility cannot replace new-data testing.

234. Replication supports reproducibility culture

Replication teams often discover:

• missing code;
• ambiguous protocols;
• undocumented variables.

The pressure to replicate can therefore improve documentation.

235. A field needs both

Reliable science needs:

same-data auditability + new-data confirmation.

These are two separate dimensions of trust.

---

236. Advanced FAQ: 30 more replication questions

31. Can a replication improve on the original method?

Yes, but then it may become a partially conceptual replication. Improvements should be documented and, when possible, the original method should also be tested so disagreement can be interpreted.

32. Should a replication copy a known original error?

No. If the original method contains a demonstrated error, reproducing the error is not scientifically useful. A replication can report the corrected method and explain how it differs.

33. What if the original paper is unclear?

Contact authors, inspect supplements, preregister reasonable interpretations and document uncertainty. Ambiguity itself is a reproducibility weakness.

34. What if original authors refuse materials?

Use available documentation and state what could not be reconstructed. Independent replication can still proceed if the scientific claim is clear enough.

35. Can proprietary methods be replicated?

Sometimes only partially. If crucial details are inaccessible, scientific confidence is limited because independent verification is constrained.

36. Can a commercial product claim be scientifically replicated?

Yes, if the outcome and intervention can be independently tested. Funding source and conflicts should be transparent.

37. Does pre-registration guarantee a good replication?

No. It limits some forms of analytic flexibility but does not fix poor design, weak measurement or low power.

38. Does open data guarantee good science?

No. Transparent bad data are still bad data. Openness supports audit, not validity by itself.

39. Does multi-lab replication eliminate bias?

No. Shared protocol flaws can affect every lab. Method diversity still matters.

40. What if every lab gets a different effect?

Estimate heterogeneity and look for moderators. The phenomenon may be context-dependent.

41. What if every lab gets the same null?

If studies are precise and valid, confidence in a meaningful effect should decrease strongly.

42. What if the original lab replicates but independent labs do not?

Investigate tacit knowledge, experimenter effects, population differences and hidden procedural features.

43. What if independent labs replicate but original lab cannot?

Investigate drift in materials, population or procedure. The broader evidence can still support the effect.

44. Can replication expose fraud?

Repeated non-replication can motivate audit, but non-replication alone does not establish misconduct.

45. Can replication vindicate controversial work?

Yes. Independent successful replication can strengthen a disputed claim substantially.

46. How should replication authors cite the original?

Precisely and fairly. The goal is to test a scientific claim, not to personalise the dispute.

47. Who gets credit for a replicated finding?

The original team deserves credit for the discovery; replication teams deserve credit for establishing reliability, scope and boundaries.

48. Is replication less creative?

No. Designing a rigorous replication can require deep theory, measurement design and statistical innovation.

49. Can replication be exploratory?

The primary replication test should be confirmatory where possible, but unexpected differences can generate exploratory moderator analyses.

50. Should exploratory moderators be trusted immediately?

No. They should be prospectively tested in another replication.

51. Can a replication be too close?

Yes. If every detail is copied, the replication may confirm only one narrow procedure and say little about generality.

52. Can a replication be too different?

Yes. If core theory-relevant features change, disagreement cannot clearly test the original claim.

53. How should replications handle updated technology?

Preserve the scientific measurement target. If new technology is more accurate, document the change and consider parallel use of old and new methods.

54. Can historical studies be replicated?

The exact event cannot recur, but independent samples, sites, dating methods and traces can test the same historical inference.

55. Can observational causal effects be replicated?

Yes, through new populations, new policy episodes, different causal designs and mechanism evidence.

56. Can theory itself be replicated?

Theory is not replicated like one experiment. Its predictions, mechanisms and constituent findings are replicated across evidence streams.

57. How should a textbook describe a once-famous non-replicated effect?

Historically and proportionally: report the original claim, later evidence and current scientific status.

58. When should a replication result change clinical or policy practice?

When the cumulative evidence is strong enough that the expected benefit of updating decisions exceeds the cost of waiting, with appropriate attention to safety and generalisability.

59. Is a replication more credible if it is adversarial?

Potentially. Adversarial collaboration can reduce interpretation disputes if both sides agree prospectively on design and criteria.

60. What is the final purpose of replication?

To discover which scientific findings survive beyond the particular data, people, methods and circumstances that first produced them.

---

237. Replication Reporting Standard

A high-quality replication report should state:

1. original scientific claim;
2. replication type;
3. degree of independence;
4. essential protocol features;
5. planned deviations;
6. sample-size rationale;
7. primary outcome;
8. pre-registered analysis;
9. effect estimate and uncertainty;
10. direct comparison with original;
11. protocol fidelity;
12. unplanned deviations;
13. heterogeneity hypotheses;
14. evidence synthesis;
15. scientific update.

This structure makes the replication useful even when the result is ambiguous.

238. Report magnitude, not only direction

“We replicated the effect” can hide a drop from 0.80 to 0.10.

Report effect sizes.

239. Report uncertainty

A replication estimate without an interval hides whether the study distinguishes meaningful alternatives.

240. Report deviations before explaining results with them

If a protocol deviation is noticed only after a surprising result, it may still matter.

But prospective deviation logs carry more credibility.

241. Report exact independence

State whether:

• same lab;
• same personnel;
• same materials;
• same equipment;
• same analysis code.

Replication independence is multidimensional.

242. Report null results completely

Do not compress a null replication into “no effect”.

Report:

• estimate;
• interval;
• minimum meaningful effect;
• power/precision;
• protocol fidelity.

243. Report successful replications completely

Success also needs scrutiny.

A replication can match direction but differ in magnitude or context.

244. Report synthesis

Whenever possible, show how the replication changes the full evidence base rather than presenting it as isolated competition with the original.

---

245. Replication and Scientific Consensus

Scientific consensus is partly built from replication.

A claim becomes consensus-level when:

• independent groups repeatedly observe it;
• methods converge;
• boundaries are understood;
• alternatives perform worse.

246. Consensus should be more stable than one paper

If one new study overturns “consensus” instantly, the prior consensus may have been shallow.

Mature consensus rests on a network of replicated findings.

247. Replication can change consensus gradually

If repeated high-quality replications shrink or eliminate an effect, consensus should move.

Scientific communities can lag, but evidence should eventually dominate.

248. Failed replication and textbooks

Educational materials should be updated when:

• classic effects no longer replicate;
• theory boundaries change;
• new synthesis revises magnitude.

Students deserve current scientific status, not historical inertia.

---

249. Replication, Robustness and Generalisability: three axes

A strong finding can be evaluated on three separate axes:

Axis	Question
Replicability	Does it appear again in new data?
Robustness	Does it survive reasonable analysis choices?
Generalisability	Does it apply in new populations or contexts?

A result can be:

• replicable but not generalisable;
• robust but not replicable;
• generalisable in direction but heterogeneous in magnitude.

Do not collapse these dimensions.

250. Replicability without robustness

Two studies can replicate using the same narrow analysis while alternative reasonable analyses change the conclusion.

This suggests analytic fragility.

251. Robustness without replication

One dataset can show the same effect across every analysis yet fail in new samples.

This suggests sample-specific structure.

252. Replication without generalisability

The effect can repeat reliably in one population while disappearing elsewhere.

This is a stable local effect, not a universal one.

253. Mature findings score on all three axes

The strongest scientific findings:

• repeat in new data;
• survive reasonable analysis choices;
• transport across appropriate contexts.

---

254. Final Replication Audit

Before calling a finding replicated, answer:

• Was new data collected?
• Was the same scientific question tested?
• Were essential features preserved?
• Was the replication sufficiently precise?
• Was analysis prospective?
• Was the effect compared directly?
• Were differences in population and measurement assessed?
• Was protocol fidelity measured?
• Was the result synthesized with prior evidence?
• Did theory scope change?

255. Replication scope and boundaries

This guide focuses on new-data confirmation of scientific findings.

Related scientific-method topics are treated separately:

• reproducibility — same data/code/workflow producing consistent computational results;
• evidence — how observations change confidence;
• experiments — how tests are designed;
• hypotheses — candidate claims;
• theories — broad explanatory frameworks;
• causation — counterfactual intervention effects.

Replication connects those topics by asking whether the scientific claim survives another independent empirical encounter.

256. Where this page sits in the eduKateYishun scientific-method system

• How Scientific Hypotheses Work — claims that replication may retest.
• How Scientific Theories Work — frameworks refined by replicated evidence.
• How Scientific Causation Works — causal effects that should replicate across settings.
• How Scientific Evidence Works — confidence updating from the accumulated evidence.
• How Scientific Experiments Work — experimental architecture replicated here.
• How Scientific Prediction Works — prospective predictions tested on new data.
• When Science Results Disagree — how conflicting replication results guide the next test.

257. Authoritative further reading

• National Academies — Reproducibility and Replicability in Science: Definitions
• National Academies — Replicability
• NIH — Strengthening Replication and Reproducibility of NIH-funded Research
• NIH — Enhancing Reproducibility through Rigor and Transparency

258. The Replication Contract

A serious replication makes a contract with the scientific record:

I will identify the claim precisely, preserve the theory-relevant features, collect genuinely new evidence, commit to the key analysis before the outcome, compare effect sizes rather than headlines, report deviations and nulls, and allow the new data to change the scientific claim.

That contract prevents replication from becoming either ritual confirmation or adversarial theatre.

259. How Scientific Replication Works: the larger lesson

A scientific result becomes more valuable when it survives people who did not produce it, samples that did not generate it, and contexts that were not chosen to make it look good.

Replication is how science asks whether knowledge belongs to the world or only to one study.

The complete loop is:

original claim → transparent protocol → new data → independent comparison → effect-size synthesis → boundary diagnosis → theory update → next replication

Sometimes the result grows stronger.

Sometimes the effect shrinks.

Sometimes a moderator appears.

Sometimes the claim disappears.

Every one of those outcomes can be scientific progress if the new evidence is honest and well designed.

The purpose of replication is not to make science look certain.

It is to discover which parts of science deserve certainty.

---

260. Replication Verification Standard

A replication should be auditable from question to conclusion. Another researcher should be able to reconstruct why the study counts as a replication, which parts of the original claim it tested, and how the new result changed the evidence base.

Verification 1 — Claim identity

State the original claim in one sentence. If the replication cannot identify the exact claim, comparison is already unstable.

Verification 2 — New-data independence

Confirm that the key empirical result comes from genuinely new observations rather than reanalysis of the original dataset.

Verification 3 — Scientific-question identity

Check that the replication addresses the same underlying scientific question even if surface materials differ.

Verification 4 — Method fidelity

List which theory-relevant features were preserved and which were changed intentionally.

Verification 5 — Population identity

State whether the population matches the original or deliberately tests generalisation.

Verification 6 — Measurement identity

Check whether the same construct is being measured with comparable validity.

Verification 7 — Intervention identity

If the original study manipulated a treatment, record dose, duration, implementation and adherence.

Verification 8 — Outcome identity

Primary outcome definitions should align unless the replication is intentionally conceptual.

Verification 9 — Analysis identity

Use the original primary analysis when valid, but do not reproduce a known error. Any corrected analysis should be labelled.

Verification 10 — Prospective criteria

Decide before seeing data how replication compatibility will be evaluated.

Verification 11 — Precision

Ensure the replication can distinguish effects large enough to matter from negligible effects.

Verification 12 — Deviation logging

Record protocol deviations when they occur, not only after a surprising result.

Verification 13 — Effect comparison

Compare point estimates and uncertainty intervals, not only binary significance.

Verification 14 — Synthesis

Combine original and replication evidence where appropriate.

Verification 15 — Scientific update

State how the claim changed: stronger, weaker, smaller, conditional, uncertain or retired.

---

261. Lab Handover Standard

A finding that cannot be handed to another laboratory easily may depend on tacit knowledge, local equipment or undocumented assumptions. A high-quality handover packet should include:

• scientific question;
• theory-relevant protocol features;
• materials list;
• equipment settings;
• calibration procedure;
• training requirements;
• sample inclusion criteria;
• randomisation or allocation rules;
• primary outcome;
• analysis code;
• known failure modes;
• contact point for clarification.

The goal is not to force another lab to imitate every cosmetic detail. The goal is to transfer enough of the scientific method that a new team can test the same claim independently.

262. Replication protocol as executable knowledge

A protocol should behave like an executable description of the experiment.

Another group should know:

• what to do;
• in what order;
• how precisely;
• what can vary;
• what cannot vary without changing the scientific question.

This is one reason high-quality protocols are themselves scientific outputs.

263. Replication stop rule

A research programme should know when close replication has produced enough information.

Consider reducing further identical replication when:

• effect estimate is precise;
• multiple independent teams agree;
• major alternative explanations are weak;
• remaining uncertainty concerns generality rather than existence.

At that point, resources can move to conceptual replication, mechanism and transport.

264. Replication reopen rule

A finding should be reopened when:

• a new high-quality study contradicts it;
• measurement technology changes;
• the population shifts;
• the intervention changes;
• publication bias is discovered;
• a new theory predicts a boundary.

Mature findings remain revisable.

265. Replication retirement rule

A claimed effect can be retired when:

• multiple well-powered direct replications fail;
• combined evidence excludes scientifically meaningful magnitude;
• no credible moderator explains the discrepancy;
• original analysis is shown to depend on bias or error.

Retirement should be documented, not erased.

266. Replication revival rule

A retired effect can be reconsidered if new evidence identifies:

• a hidden moderator;
• a measurement failure;
• a wrong treatment implementation;
• a new mechanistic reason the original population differed.

The revived claim must face a new prospective replication.

---

267. Result Classification Standard

Class 1 — Confirmatory replication

New data support a similar effect with useful precision.

Class 2 — Magnitude revision

Effect persists but is smaller or larger than originally estimated.

Class 3 — Boundary discovery

Effect appears only under specified conditions.

Class 4 — Measurement-specific replication

Effect appears with one instrument but not another.

Class 5 — Population-specific replication

Effect replicates in one population and not another.

Class 6 — Mechanism-specific replication

Total effect replicates but proposed mechanism does not.

Class 7 — Inconclusive replication

New data are too imprecise to distinguish meaningful alternatives.

Class 8 — Contradictory replication

New precise evidence is incompatible with the original claim.

Class 9 — Protocol-invalid replication

Implementation differed enough that the study does not test the original claim clearly.

Class 10 — Generalisation replication

Finding survives a deliberately changed population, method or context.

These categories make discussion more informative than “replicated / not replicated”.

---

268. Replication Risk Matrix

Original-study feature	Replication risk	Recommended response
small sample	effect inflation	larger precision-focused replication
many outcomes	selective reporting	pre-specify original focal outcome
complex protocol	implementation drift	detailed protocol + training
subjective scoring	observer bias	blind raters
one population	limited generality	cross-population replication
one lab	lab-specific effect	independent multi-lab replication
one instrument	measurement artifact	independent measurement method
flexible analysis	analytic overfitting	preregistered fixed analysis
famous benchmark	benchmark overfitting	fresh holdout data
dramatic effect	winner’s curse	estimate shrinkage prospectively

269. Replication Information Value

Not every replication adds equal information.

High-information replications often:

• use independent data;
• reduce uncertainty sharply;
• test a serious boundary;
• compare rival theories;
• change a policy decision;
• clarify a mechanism.

Low-information replications may repeat an already secure effect under nearly identical conditions without improving precision or scope.

270. Replication as Bayesian updating

Conceptually, replication can be understood as repeated belief updating.

Start with a prior belief about effect size.

Original data update that belief.

Replication data update it again.

After many studies, the posterior distribution can become narrower—unless heterogeneity reveals the effect itself varies.

271. Replication and prior plausibility

Extraordinary claims often begin with lower prior plausibility.

A single surprising result may therefore need especially strong independent replication before confidence becomes high.

This is not prejudice against novelty.

It is proportional evidence updating.

272. Replication and theory priors

A finding strongly predicted by an established theory starts in a different evidential position from a finding that conflicts with many well-supported constraints.

Both can be tested.

The amount of replication needed for broad acceptance can differ.

---

273. Replication Communication Standard

Public communication should avoid:

• “Study proven wrong” after one replication;
• “Science confirms” after one successful repeat;
• “Effect does not exist” from an imprecise null;
• “Replication succeeded” based only on p < .05.

Better language includes:

• “The replication estimated a similar effect.”
• “The replication was incompatible with the originally reported magnitude.”
• “The replication was too imprecise to distinguish meaningful effects.”
• “The effect appeared only under Condition C.”

274. Report the whole evidence path

A public summary should identify:

• original study;
• replication design;
• new result;
• combined evidence;
• scientific update.

This prevents one-study storytelling.

275. Replication and media incentives

Media can overemphasise:

• dramatic original finding;
• dramatic failed replication.

The middle result—smaller, conditional, more precise—is often scientifically most valuable but less headline-friendly.

276. Science communication should reward revision

A shrinking effect estimate should not be described as embarrassment.

It is the normal movement from noisy discovery toward stable knowledge.

---

277. Replication Integrity Standard

Replication deserves the same ethical and methodological care as discovery.

Integrity means:

• test the real claim;
• avoid caricaturing the original method;
• preregister important decisions;
• report all results;
• preserve deviations;
• credit original and replication teams fairly;
• update proportionally.

278. No replication theatre

A replication should not be designed to produce a predetermined rhetorical outcome.

Both “we will prove them wrong” and “we will validate our discovery” are poor scientific starting points.

The replication question should be open to either result.

279. No hidden rescue

Original authors should not invent new boundary conditions only after every failed replication unless those conditions generate new prospective predictions.

280. No hidden downgrade

Replication authors should not quietly change the scientific claim to an easier target and then declare the original disproven.

Claim identity matters.

281. Replication stewardship

A field should maintain the history of:

• original claim;
• major replications;
• updated magnitude;
• known moderators;
• current consensus.

This is the lifecycle of reliable scientific knowledge.

---

282. Final Replication Checklist

• The original claim is stated precisely.
• New data are genuinely new.
• Replication type is labelled.
• Theory-relevant protocol features are preserved.
• Intentional changes are documented.
• Sample size targets useful precision.
• Primary outcome is pre-specified.
• Analysis is pre-specified where possible.
• Implementation fidelity is measured.
• Effect size and uncertainty are reported.
• Replication is not judged solely by p-values.
• Measurement comparability is checked.
• Population differences are considered.
• Heterogeneity is analysed.
• Publication bias is considered.
• Evidence is synthesised cumulatively.
• Claim boundaries are updated.
• The next replication targets remaining uncertainty.

283. Final replication standard

This guide remains focused on one durable scientific question:

Does a finding survive another study with new data?

Future expansion should deepen:

• replication design;
• independence;
• effect comparison;
• heterogeneity;
• multi-lab science;
• evidence synthesis;
• claim revision.

Reproducibility is a neighbouring but distinct problem: same-data/code/workflow reconstruction rather than new-data confirmation.

284. Final close

The first study discovers a possibility.

Replication asks whether that possibility is stable enough to become knowledge.

The transition is not:

significant → true

It is:

observed once → tested again → compared quantitatively → tested independently → mapped across contexts → synthesised → revised into a stable scientific claim.

That is how science moves from one result to something the rest of the world can rely on.

---

285. Replication Decision Thresholds

Replication does not only answer whether a claim is stable. It helps decide what science should do next. The next action depends on the combination of effect magnitude, uncertainty, heterogeneity, theory importance and practical stakes.

Decision A — Replicate again closely

Choose another direct replication when:

• the first replication was imprecise;
• the original claim is important;
• protocol fidelity was uncertain;
• the result sits near a meaningful decision threshold.

The goal is to reduce uncertainty about whether the effect exists at all under the original conditions.

Decision B — Move to conceptual replication

Choose conceptual replication when:

• direct replication is strong;
• the underlying mechanism is the next question;
• the field needs to know whether the effect survives another measure or manipulation.

This tests generality rather than simple repeatability.

Decision C — Test a moderator

Choose a moderator study when:

• replication effects differ systematically;
• one population shows the effect and another does not;
• dose, age, culture or implementation seems responsible.

The moderator should be specified prospectively.

Decision D — Improve measurement

Choose measurement development when:

• replication disagreement tracks instruments;
• scale reliability is poor;
• construct meaning differs across populations.

Do not keep repeating a study with a weak measurement system.

Decision E — Synthesize

Choose meta-analysis when:

• several comparable studies exist;
• effect estimates vary;
• the field needs a combined estimate and heterogeneity map.

Synthesis turns replication from a sequence of papers into cumulative evidence.

Decision F — Narrow the claim

Narrow the claim when:

• effect appears only under specific conditions;
• original wording was too universal;
• moderator evidence is strong.

Decision G — Retire the claim

Retire the claim when:

• multiple precise replications exclude meaningful effects;
• original estimate appears to be a false positive;
• no coherent boundary condition rescues the theory;
• a stronger rival explanation fits the total evidence.

Decision H — Change application but retain science

A small replicated effect can remain scientifically real while becoming practically unimportant.

Separate:

• existence;
• magnitude;
• decision relevance.

286. Replication and decision relevance

A replicated effect does not automatically justify action.

Decision-makers also need:

• absolute magnitude;
• cost;
• benefit;
• harm;
• feasibility;
• population impact.

Replication creates trust in the scientific estimate. Decision analysis determines what to do with it.

---

287. Independent Confirmation Standard

Independent confirmation becomes especially valuable when:

• the original finding has major policy consequences;
• the original team has a commercial interest;
• the result is highly surprising;
• the method is technically complex;
• many later studies depend on the claim.

Independence should be described precisely.

Data independence

New samples or observations are used.

Researcher independence

A different team collects or analyses the data.

Institutional independence

A different laboratory, hospital, school, company or university conducts the work.

Method independence

A different instrument or operationalisation tests the same theoretical relationship.

Funding independence

A different funder supports the replication.

No single form of independence is mandatory in every case. The strongest evidence often combines several.

288. Why independence matters

Shared systems can carry shared errors.

If original and replication use:

• same dataset;
• same equipment;
• same analysts;
• same code;
• same assumptions;

then a hidden flaw can reproduce perfectly.

Independence creates new failure modes and therefore stronger tests.

289. Independence should not destroy fidelity

Too much independence can accidentally change the scientific question.

A useful replication balances:

enough fidelity to test the same claim + enough independence to challenge shared error.

---

290. Replication Handover Standard

Once a result has replicated, the field should hand over more than the final effect size.

The handover should include:

• original claim;
• replication history;
• pooled estimate;
• known heterogeneity;
• moderators;
• measurement standards;
• known failure modes;
• remaining open questions.

This prevents future researchers from treating a mature effect as if it had only one study behind it.

291. Replication summary card

A mature finding can be summarised as:

Field	Current status
Original claim	precise scientific statement
Direct replications	number and compatibility
Conceptual replications	methods and populations
Combined effect	meta-analytic estimate
Heterogeneity	known variation
Moderators	supported boundaries
Mechanism	current evidence
Current consensus	strong / conditional / uncertain / unsupported

This creates a durable evidence object.

292. Replication version history

Scientific claims should be versioned.

Version 1:

“Effect X is large.”

Version 2:

“Effect X exists but is smaller.”

Version 3:

“Effect X exists mainly under Condition C.”

The version history shows how replication improved the claim.

293. Replication can strengthen credibility without increasing effect size

A small effect replicated across many settings can be more scientifically reliable than one spectacular unreplicated effect.

Reliability and magnitude are different.

294. Large effect does not compensate for no replication

The larger and more surprising the effect, the more important independent confirmation becomes.

295. Small effect can matter at scale

A small replicated effect applied to millions of people can have substantial population impact.

Decision relevance depends on scale.

---

296. Replication Stop-and-Reopen Protocol

Stop close replication when:

• existence is highly secure;
• effect magnitude is precise enough;
• multiple independent teams agree;
• further identical studies add little information.

Move to boundary work when:

• generalisation is uncertain;
• heterogeneity appears;
• mechanism is unresolved.

Reopen the finding when:

• new contradictory high-quality evidence appears;
• measurement standards change;
• important publication bias is uncovered;
• a new population behaves differently;
• a new theory predicts an exception.

297. Replication is never “finished” in the absolute sense

Science cannot test every future context.

A mature finding is one whose current domain is mapped well enough for reliable use, not one protected from all future revision.

---

298. Final Source and Standards Note

The National Academies definition used in this owner deliberately separates:

• reproducibility: obtaining consistent computational results with the same data and computational procedures;
• replicability: obtaining consistent results across studies aimed at the same scientific question using new data.

The National Academies also emphasises that replication is nuanced, that results need not be numerically identical, and that scientific validity should be interpreted across the full body of evidence rather than a single original or replication study.

NIH’s current replication and reproducibility initiative similarly treats independent confirmation, rigorous methods, transparency, data sharing and replication infrastructure as central to reliable biomedical science.

299. Final Replication Integrity Close

The scientific value of replication does not come from repeating work mechanically.

It comes from creating a second chance for reality to disagree.

A genuine replication:

• uses new evidence;
• protects the identity of the scientific claim;
• reduces hindsight flexibility;
• measures effect and uncertainty;
• treats disagreement as information;
• updates the full evidence base.

The finding that survives this process deserves more confidence than the finding that has only been seen once.

Replication is science refusing to confuse first discovery with final reliability.

---

300. Replication Uncertainty and Claim-Retirement Appendix

Replication becomes most useful when uncertainty is carried forward rather than replaced with a new binary label. The scientific question after a replication is not simply “Did it work?” It is “How did the distribution of plausible scientific claims change?”

Uncertainty after agreement

When original and replication estimates agree, uncertainty usually narrows. But the scientific claim should still retain:

• sampling uncertainty;
• measurement uncertainty;
• between-study heterogeneity;
• domain uncertainty.

Agreement strengthens confidence without making every future context identical.

Uncertainty after disagreement

When studies disagree, uncertainty can initially increase.

This is not failure of science.

The disagreement may reveal that the original claim was too simple.

The next study should aim to distinguish among:

• sampling variation;
• hidden moderator;
• measurement difference;
• protocol difference;
• original false positive.

Do not average before understanding

Pooling studies immediately can hide scientifically meaningful differences.

Before averaging, ask whether the studies truly estimate the same underlying effect.

If one effect occurs only under Condition C, a single pooled average can blur the mechanism.

Replicated absence

Repeated precise studies can support the conclusion that any true effect is smaller than a scientifically meaningful threshold.

This is stronger than saying:

“Several studies were non-significant.”

Replicated absence requires:

• adequate precision;
• valid measurement;
• faithful implementation;
• predefined meaningful effect threshold.

Replicated heterogeneity

Sometimes the most replicable result is that the effect varies.

For example:

• large in one population;
• small in another;
• absent under another protocol.

If that pattern itself repeats, heterogeneity becomes part of the scientific finding.

Claim retirement should be explicit

When a claim no longer deserves support, the scientific record should say so clearly.

A retirement note should include:

• original claim;
• key replications;
• updated combined effect;
• reason the claim is no longer supported;
• whether any narrower version remains viable.

Retirement protects future researchers from repeatedly rediscovering the same failed result.

Retire the claim, not the question

A failed effect can still leave an important question.

Example:

Old claim:

“Intervention X improves outcome universally.”

After replications:

“Universal effect unsupported.”

New question:

“Does X help under Condition C or through Mechanism M?”

Science can abandon one answer while keeping the question alive.

Replication handoff to theory

Once replication evidence stabilises, the result should be handed back to theory.

Theory should update:

• expected magnitude;
• known boundary conditions;
• supported mechanism;
• uncertainty;
• future predictions.

Replication is therefore not a side activity detached from theory. It is one of the main routes by which theories become more accurate.

Replication handoff to application

Applications should use the replicated estimate, not the most dramatic original result.

Medicine, education, engineering and policy should prefer:

• combined effect;
• known heterogeneity;
• replicated harms;
• validated population boundaries.

This reduces the risk of building systems on unstable evidence.

301. The Replication Handover Contract

A mature replicated finding should be handed to future researchers with enough information that they do not have to reconstruct the evidential history from scratch.

The handover should state:

1. what the original study claimed;
2. how many meaningful replications exist;
3. how independent they were;
4. what effect magnitude now appears most plausible;
5. how much heterogeneity exists;
6. which moderators are supported;
7. which methods have reproduced the effect;
8. which populations remain untested;
9. what mechanism evidence exists;
10. what would reopen the claim.

This turns a literature into usable scientific memory.

302. The final replication standard

A finding should earn increasing confidence as it survives increasingly independent encounters with reality.

The durable sequence is:

new data → independent test → quantitative comparison → diagnosis of differences → cumulative synthesis → revised claim → new boundary test.

The purpose is not to force identical outcomes.

It is to discover which parts of the original finding are stable, which are context-dependent, and which should disappear from the scientific record.

Replication therefore performs one of science’s most important acts of quality control: it converts a result that happened once into a claim that has demonstrated it can happen again.

---

303. Replication Integrity Close

A replication programme should remain useful even when the result is inconvenient. The original team should not be allowed to redefine the claim after failure, and the replication team should not be allowed to redefine the method until the original becomes impossible to reproduce. Both sides owe the scientific record a stable target.

The final integrity test is simple. Before data are collected, both teams should be able to answer:

• What exact claim are we testing?
• Which parts of the original procedure are essential?
• Which changes are intentional?
• What result would count as compatible?
• What result would count as a serious contradiction?
• How will either outcome change the scientific claim?

If those questions are answered prospectively, replication becomes a genuine test rather than a rhetorical contest.

The scientific standard is therefore:

Preserve the claim, preserve the protocol logic, collect new evidence, compare quantitatively, publish either outcome, and let the replication change the status of the finding.

A result seen once can be interesting. A result that survives independent repetition becomes dependable. A result that survives multiple methods, populations and laboratories becomes part of the durable architecture of science.

---

304. Final Replication Owner Note

This owner should remain useful as methods, statistics and research infrastructure evolve. New meta-analytic tools, automated laboratories, AI-assisted experiment design and larger collaborative networks may change how replication is organised, but the durable scientific obligations do not change.

A replication must still:

• test a clearly identified scientific claim;
• collect genuinely new evidence;
• preserve the theory-relevant logic of the original study;
• state intentional differences openly;
• compare effect magnitude and uncertainty rather than rely on significance labels;
• distinguish sampling variation from meaningful heterogeneity;
• publish agreement and disagreement with equal integrity;
• feed the result back into theory, application and the next experiment.

The point is not repetition for its own sake. The point is independence. Science becomes more reliable when a claim can leave the laboratory, dataset or research team that first produced it and still survive a new test.

That is the final replication standard: new data, honest comparison, independent scrutiny, cumulative update.

Integrity: A replicated finding deserves confidence only while new independent evidence is still allowed to refine its magnitude, boundaries, mechanism, generalisability and practical meaning without being filtered through loyalty to the original result.