A Handbook of Advanced Meta-Analysis Techniques: 53 Concepts, One Honest Page Each

Most meta-analysis methods are taught in one of two unhelpful ways. Either you get the textbook derivation — dense, correct, and disconnected from anything you'll actually decide when you're staring at your own forest plot — or you get the software default, silently applied, with no sense of what it assumes or when it breaks. Neither teaches you to use a method, only to run it.

This post introduces a small reference we built to close that gap: A Handbook of Advanced Meta-Analysis Techniques, a free PDF covering the pooling, heterogeneity, robustness, publication-bias, and certainty methods a clinical or medical-device meta-analysis needs, one concept per page. It is generic, not tied to any one project or dataset, and it is built for a specific learning method: read one concept, close the book, rewrite it in your own words, then try to explain it to someone with no background at all.

Download the handbook (free PDF) → Leave your work email on the handbook page and the PDF downloads immediately — no sales call, no account.

In short

• The handbook covers 53 concepts, one per A4 page, grouped into 12 parts — pooling models, heterogeneity (how much and where from), robustness and sensitivity, covariates, publication bias, special data structures (time-to-event, diagnostic accuracy, dependent effect sizes), claim-making, certainty of evidence, and descriptive synthesis.
• Every page has the same five-part shape: the nutshell, the working idea plus the maths (only where the formula is the idea), what it cannot do, how it's used in practice, and a kitchen-table analogy.
• It is deliberately honest about failure modes — Trial Sequential Analysis, for instance, gets its own page on when to drop it, not just how to run it.
• It's free. Leave your email on the handbook page and the PDF downloads immediately.

Why one concept per page

Anyone who has sat through a statistics course knows the failure mode: you can follow a derivation on a whiteboard and still have no idea what to do with the result, or whether it's trustworthy. The handbook is built against that failure mode specifically. Every concept — Cochran's Q, the prediction interval, Egger's test, GRADE — gets exactly one page, and every page answers the same five questions in the same order:

1. The nutshell — the one- or two-sentence version. If you remember nothing else about a method, this is what you should remember.
2. The maths — the formula, but only when the formula is the idea (not every page has one), with every symbol spelled out underneath in plain words.
3. What it cannot do — the honest limits. This is the section that separates someone who has read about a method from someone who understands it, and it is the section most references skip.
4. In practice — how the method is actually used, and what to report alongside it.
5. At the kitchen table — an everyday analogy, because if you can't explain a method to someone with no background at all, you don't fully understand it yet either.

Here is what that looks like on the very first page of the book — the concept everything else is built on:

Walking through one page: what a meta-analysis actually is

This first concept is the simplest in the book, and it's worth walking through in full, because almost every mistake later in the handbook is a violation of this one idea.

The nutshell: a meta-analysis is a weighted average of the results of several studies, plus an honest statement of how much those studies disagree. That's it. It's arithmetic on summaries — the effect size and its uncertainty from each study — not a re-run of the original patients. Nobody's raw data re-enters the calculation; each study has already been boiled down to one number and one measure of how sure that number is.

Why it's not a new experiment: a meta-analysis is a synthesis method. It cannot create information the underlying studies didn't collect, and it inherits every bias baked into them. The handbook states the governing law bluntly: garbage in, garbage out — pooling ten biased studies gives you a very precise, very wrong number. A narrow confidence interval around a bad pool is false comfort, not proof, and no amount of statistical sophistication downstream (heterogeneity tests, sensitivity analyses, GRADE ratings) fixes a pool built on the wrong studies.

The one real decision: pool, or don't. Before any of the other concepts apply, a systematic review has to earn the right to pool at all — the studies need to be similar enough in population, intervention, and outcome. When they're too few or too different, the honest move is to describe them instead of averaging them (Part X of the handbook is entirely about this "when not to pool" judgment call).

At the kitchen table: five friends each guess the weight of a suitcase. You don't just shout the loudest guess — you take an average, but you trust the friend who actually lifted the case more than the one who only glanced at it. And if the guesses range from 5 kg to 40 kg, that spread is itself worth reporting, not hiding. That's a meta-analysis: a weighted average that also tells you how much the inputs disagreed.

Every one of the other 52 concepts is a refinement of some part of this page — how exactly you weight the guesses (inverse-variance weighting, Concept 6), how you quantify "how much they disagreed" (Cochran's Q, $I^2$, $\tau$, the prediction interval — Concepts 14 through 17), what to do when the outcome isn't a simple yes/no or mean difference (hazard ratios, diagnostic accuracy, dependent effect sizes — Concepts 49 through 51), and what to do when the disagreement is suspiciously large or one guess looks wrong (leave-one-out, the Baujat plot, influence diagnostics — Concepts 21 and 25 through 26).

The whole book, compressed to one page

If concept 1 is where the book starts, its final page is where it ends — and it's the one page we'd tell you to read even if you read nothing else. It distills the handbook into twelve named mistakes, each of which has sunk a real, published meta-analysis: choosing a fixed- or random-effects model from the heterogeneity test's p-value, reporting $I^2$ without $\tau$ or a prediction interval, treating trim-and-fill's imputed studies as real, mistaking an odds ratio for a risk ratio when events are common, letting one influential study wear the costume of a consensus.

This page alone is a fast, honest pre-submission check for any pooled analysis heading into a manuscript, a Clinical Evaluation Report, or a systematic-review write-up: run down the twelve items, and see how many you can honestly tick off.

Who this is for

If you've read our four-part series on meta-analysis — covering the standard toolkit, multilevel and multivariate models, forest plots in R, and why the prediction interval matters more than the confidence interval — this handbook is the reference version of that series and a good deal more: the same rigor, compressed to one page per idea, with nothing that requires you to remember which post covered which method. It's built for anyone doing evidence synthesis for a Clinical Evaluation Report, a systematic review, or a State-of-the-Art analysis under the EU MDR, and equally for anyone who just wants a straight answer to "what does $\tau^2$ actually mean" without wading through a package vignette.

It is not a substitute for running the analysis carefully, and it is not a substitute for a statistician when the stakes are regulatory. It is the thing you read before that conversation, so the conversation starts from the right questions.

Get it

The handbook is free. Leave your work email on the handbook page and the PDF — 53 concepts, 12 parts, 58 pages, no code required — downloads immediately. If you'd like help applying any of it to your own evidence synthesis, get in touch.

Dr. Marc Harms MH-Analytics