Need help now? Support now: crisis lines by country.
Questions about pain · Mathematical Institute for Research of Pain
A treatment did well in a study. Does that mean it will work for me?
Not necessarily. A pooled result is an average over the people studied, and it can even point the opposite way from the result inside every group.
Every one knows best where the shoe pinches him.
The short answer
A person choosing care needs population evidence without being told the average is their destiny. A study reports what happened across the people in it. When a pooled trend is read as the trend inside every group, without checking who was in each group and in what numbers, the conclusion can go further than the record. The Institute’s Pain Bias Codex names this step “The pooled trend speaks for every group”.
What is known
It is a mathematical fact that pooling groups can weaken, erase or even reverse a comparison that holds inside each group, when each option’s people are spread across the groups in different proportions. The effect is called Simpson’s paradox, after E. H. Simpson, whose 1951 paper on pooled tables described it. The arithmetic is short, and the example below is all of it.
A worked example
An illustration with invented numbers. Two options are offered in two groups of people.
| Group | Option 1 | Option 2 |
|---|---|---|
| Group A | 9 of 10 (90%) | 80 of 100 (80%) |
| Group B | 20 of 100 (20%) | 1 of 10 (10%) |
| Pooled | 29 of 110 (26%) | 81 of 110 (74%) |
Option 1 does better in group A and better in group B, yet the pooled figures favour option 2 by a wide margin. The reason is in the weights. A pooled rate is a weighted average of the group rates, each weighted by the share of that option’s people who were in that group:
Here and are the rates in groups A and B, and and are the shares, which add up to 1. For option 1, and , so most of its weight sits in group B, where both options do poorly. For option 2 the weights are the other way round. The pooled comparison mostly compares the two groups, not the two options.
What is not yet known
A reversal is a mathematical possibility, not a claim that every pooled trend reverses. In a well-randomised trial each option’s people are spread across the groups in about the same proportions, so a full reversal is unlikely there; it is a bigger risk when people chose their option, or when raw counts from different studies are simply added together. Splitting by groups also needs a justified question: the arithmetic does not decide which groups to split by, and the split that answers a question about cause has to be argued for separately.
What would tell us
Showing the pooled comparison and the relevant group comparisons side by side, each with its denominators, and saying which comparison answers the question being asked. Where a study reports a pre-planned result, with enough people, for a group that matches a person’s situation, that result can say more about the person than the pooled figure does; it is still an average over that group.
These pages explain reasoning and evidence. Nothing here is medical advice or a forecast about any person.
Sources
- Simpson EH. The interpretation of interaction in contingency tables. J R Stat Soc Series B. 1951;13(2):238–241. doi:10.1111/j.2517-6161.1951.tb00088.x
Related questions
Pain words: N-of-1 trial · Minimal clinically important difference · Regression to the mean · All words
Related, in Nociflux, the Institute’s ecosystem for living with pain: the N-of-1 Lab.
Inside the Institute
The Pain Bias Codex gathers checks on conclusions about pain, in four families. With an invite code you can read this check there beside the others, with its full evidence notes. Pain math explains what the Institute does.
No code yet? Ask for an invite. A person reads every request and writes back by hand with an invite code, so it may take a few days.
Need help now? Support now: crisis lines by country.
Except where a page says otherwise, the Institute’s own text and figures are shared under CC BY-NC 4.0. Pictures and quotations from others keep their own credits and licences.