What actually makes a research finding reliable?

The eight checks we run on every project

1. 1 Ensure high-quality and accurate data The dataset is cleaned before analysis: data-entry errors, duplicates, impossible values, internal inconsistencies and missing data are identified and resolved. No analysis is more trustworthy than the data underneath it, and a model will run perfectly happily on a corrupted dataset.
2. 2 Use an appropriate study design and adequate sample size The design has to match the research question, and a sample-size or power calculation should establish that the study can detect an effect worth detecting. A well-analysed underpowered study is still an underpowered study.
3. 3 Use the appropriate statistical analysis Tests and models are selected according to the type of variables, the study design, the number of groups and the question. ANOVA may be right for comparing means across several groups; regression is needed when associations must be estimated with adjustment for confounders.
4. 4 Test the assumptions of the statistical methods Before conclusions are drawn, the relevant assumptions are evaluated: normality, homogeneity of variance, independence of observations, linearity, multicollinearity, and proportional hazards or proportional odds where applicable. Where an assumption fails, a robust or alternative approach is used and reported.
5. 5 Perform sensitivity and robustness analyses Important findings are re-tested under different reasonable assumptions: excluding influential observations, alternative models, alternative missing-data assumptions, leave-one-out analyses in meta-analysis. A finding that stays similar across these is far more convincing than one that does not.
6. 6 Address missing data appropriately The amount and pattern of missingness are evaluated, and an appropriate strategy — often multiple imputation — is chosen rather than deleting every participant with an incomplete record. Complete-case analysis is a decision about your results, not a neutral default.
7. 7 Have the analysis independently checked Data cleaning, statistical code, results, tables and figures are reviewed by a second independent researcher or statistician. Independent verification is how coding errors, inappropriate analyses and internal inconsistencies are caught before a reviewer catches them.
8. 8 Ensure reproducibility of the analysis Methods are documented and analysis scripts retained, so another researcher can reproduce the reported results from the same dataset. If a number cannot be traced back to a line of code and a row of data, it cannot really be defended.

Seven further standards, where the design calls for them

Not every study needs every one of these. Where a study does, leaving it out is not a shortcut — it is the thing a reviewer will find.

• Control for potential confounding factors. Multivariable adjustment, stratification, matching or propensity-score methods in observational research.
• Report effect sizes and confidence intervals, not only p-values. Magnitude and precision, not just statistical significance.
• Correct for multiple testing when necessary. Holm, Bonferroni or false-discovery-rate control across multiple outcomes, comparisons or subgroups.
• Use meta-analysis where the evidence genuinely supports pooling. And decline to pool where the studies are clinically or methodologically incompatible.
• Assess the risk of bias and methodological quality. Selection, measurement, attrition and reporting bias, using the instrument appropriate to the design.
• Compare the findings with previous evidence. Consistency with independent research increases confidence; discrepancies are investigated rather than ignored.
• Avoid selective reporting and data-driven conclusions. Pre-specify primary outcomes, report negative findings, and distinguish prespecified from post-hoc analyses.

Five questions to ask before you bring a biostatistician onto your project

1. Do they ask about your research question before your data?
   The question determines the analysis. A statistician who asks for the spreadsheet first will fit a method to whatever arrives.
2. Can they explain the method in plain language, not jargon?
   If they cannot explain it to you, you cannot defend it to a reviewer.
3. Do they review your study design, or only analyse what is already collected?
   Design problems cannot be repaired by analysis, however sophisticated.
4. Do they document every step so someone else could reproduce it?
   Reproducibility is the difference between a defensible result and a hopeful one.
5. What does their own publication record look like?
   Ours: every team member has 50+ publications and an h-index of at least 5.