### Number Watch Web Forum

This forum is about wrong numbers in science, politics and the media. It respects good science and good English.

Number Watch Web Forum
Author
Comment
Re: confidence levels

"A metastudy takes a lot of little insignificant trials and pretends they are one big significant one."

Would it be true to say that a metastudy does little more than to increase the probabilty that a conclusion is incorrect?

It would seem that the amount of said increase would be up to the maths of 'conditional probability".

Going back to coin flipping.

If you take two coins in your hand and then throw them under a cloth:

Each coin has a 50% chance of being a heads; but, given the condition that your first coin comes out from under the cloth as a heads, there is only a 25% probability the second coin will also do so.

Perhaps, part of the problem with metastudies, is that there is no consideration of 'conditional probability'?

Re: confidence levels

Now you have lost me.

Re: confidence levels

That is certainly not your fault.

I was trying to explain a concept and it was, I'm afraid, poorly done.

Re: confidence levels

This free to view paper taken from the Lancet should give an idea how the confidence interval for the "meta analysis" relates to the confidence intervals for the individual contributing studies in the meta analysis. The meta analysis still has 95% confidence levels.

http://tobaccodocuments.org/pm/2047231315-1318.pdf

The paper seems to be using a technique for combining results from the contributing studies called the Mantel-Haenszel method (it quotes some other methods as well), for which details are given on this webpage:

https://www.ctspedia.org/do/view/CTSpedia/StudyAnalysisMH

Re: confidence levels

Meta-analysis can work fine if you are combining several similarly-executed trials which then gives you the statistical power to see effects that wouldn't be powered for in the smaller trials.

The problem with most of the epidemiological stuff is not the use of meta-analysis per se, it is that the statistical testing is applied to large numbers of post-hoc hypotheses and the tests are really designed to look for effects of interventions. Since you can't ethically perform an interventional study with tobacco smoke (we know it's bad for you and the study has no prospect of benefitting the participants) you have to assign "treatment groups" on the basis of asking people about past, incidental exposure. This is not only notoriously inaccurate it introduces a range of biases you can't control for. One odd result off the top of my head: Dutch tea drinkers are more likely to smoke than Dutch non-tea drinkers. If the correlation is strong enough (or the study large enough) you could demonstrate that drinking tea causes heart disease in the Netherlands. Bigger studies also take disproportionate effort to do important things like age and sex matching of controls - importantly failing to do this is likely to dilute real effects but potentiate stochastic effects.

In the clinical world, statistically significant results (uncorrected for multiplicity) on things other than your powered, primary efficacy variable, are considered interesting things that might or might not be worthy of further investigation. At best in efficacy they are supportive of a claim. You couldn't usually base a strong enough claim to get a marketing license on them, for example. In the public health world, one P<0.05 among twenty risks (never benefits) tested for is considered adequate justification for draconian legislation.