Need help now? Support now: crisis lines by country.

Questions about pain · Mathematical Institute for Research of Pain

A treatment did well in a study. Does that mean it will work for me?

Not necessarily. A pooled result is an average over the people studied, and it can even point the opposite way from the result inside every group.

Every one knows best where the shoe pinches him.

French proverb, translated by Henry G. Bohn, A Polyglot of Foreign Proverbs, 1857 (London: Bell & Daldy reprint, 1867), French Proverbs, p. 12 · Source

The short answer

A person choosing care needs population evidence without being told the average is their destiny. A study reports what happened across the people in it. When a pooled trend is read as the trend inside every group, without checking who was in each group and in what numbers, the conclusion can go further than the record. The Institute’s Pain Bias Codex names this step “The pooled trend speaks for every group”.

What is known

It is a mathematical fact that pooling groups can weaken, erase or even reverse a comparison that holds inside each group, when each option’s people are spread across the groups in different proportions. The effect is called Simpson’s paradox, after E. H. Simpson, whose 1951 paper on pooled tables described it. The arithmetic is short, and the example below is all of it.

A worked example

An illustration with invented numbers. Two options are offered in two groups of people.

People helped, out of those who had each option (invented numbers)
GroupOption 1Option 2
Group A9 of 10 (90%)80 of 100 (80%)
Group B20 of 100 (20%)1 of 10 (10%)
Pooled29 of 110 (26%)81 of 110 (74%)

Option 1 does better in group A and better in group B, yet the pooled figures favour option 2 by a wide margin. The reason is in the weights. A pooled rate is a weighted average of the group rates, each weighted by the share of that option’s people who were in that group:

ppooled=wA pA+wB pBp_{\text{pooled}} = w_A\,p_A + w_B\,p_B

Here pAp_A and pBp_B are the rates in groups A and B, and wAw_A and wBw_B are the shares, which add up to 1. For option 1, wA=10/110w_A = 10/110 and wB=100/110w_B = 100/110, so most of its weight sits in group B, where both options do poorly. For option 2 the weights are the other way round. The pooled comparison mostly compares the two groups, not the two options.

What is not yet known

A reversal is a mathematical possibility, not a claim that every pooled trend reverses. In a well-randomised trial each option’s people are spread across the groups in about the same proportions, so a full reversal is unlikely there; it is a bigger risk when people chose their option, or when raw counts from different studies are simply added together. Splitting by groups also needs a justified question: the arithmetic does not decide which groups to split by, and the split that answers a question about cause has to be argued for separately.

What would tell us

Showing the pooled comparison and the relevant group comparisons side by side, each with its denominators, and saying which comparison answers the question being asked. Where a study reports a pre-planned result, with enough people, for a group that matches a person’s situation, that result can say more about the person than the pooled figure does; it is still an average over that group.

These pages explain reasoning and evidence. Nothing here is medical advice or a forecast about any person.

Sources

  1. Simpson EH. The interpretation of interaction in contingency tables. J R Stat Soc Series B. 1951;13(2):238–241. doi:10.1111/j.2517-6161.1951.tb00088.x

Pain words: N-of-1 trial · Minimal clinically important difference · Regression to the mean · All words

Related, in Nociflux, the Institute’s ecosystem for living with pain: the N-of-1 Lab.

Inside the Institute

The Pain Bias Codex gathers checks on conclusions about pain, in four families. With an invite code you can read this check there beside the others, with its full evidence notes. Pain math explains what the Institute does.

No code yet? Ask for an invite. A person reads every request and writes back by hand with an invite code, so it may take a few days.

Need help now? Support now: crisis lines by country.

Except where a page says otherwise, the Institute’s own text and figures are shared under CC BY-NC 4.0. Pictures and quotations from others keep their own credits and licences.

Back to the invite page · Sign in

Paper