NDT Advance Access originally published online on April 24, 2008
Nephrology Dialysis Transplantation 2008 23(9):2729-2732; doi:10.1093/ndt/gfn196
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
An infallible recipe? A story of cinnamon, soufflé and meta-analysis
Renal Division, Department of Internal Medicine, University Hospital Ghent, Belgium
Correspondence and offprint requests to: Wim Van Biesen, Renal division, Department of internal Medicine, University Hospital Ghent, Belgium. E-mail: wim.vanbiesen{at}ugent.be
Keywords: guidelines; meta-analysis; methodology
 |
A recipe by Escoffier... |
|---|
 Top A recipe by Escoffier... Selection of the ingredients...Finding the right ingredients...Spicing up ingredients...Blue cheese and cheddar...A dot of harisha...References |
|---|
In the slipstream of evidence-based medicine, the technique of meta-analysis risks to become ‘the philosophers stone’ to turn dodgy facts into sound and trustworthy evidence. The ritual of painstakingly performing some methodological and mathematical acts renders the results to indubitable truth in most readers eyes. This magic has resulted in an explosion of meta-analyses on all imaginable topics in medicine. A Pubmed search demonstrates an exponential increase of papers reporting or discussing results of meta-analyses (Figure 1).
|
Meta-analyses certainly do have their place in scientific research. Like herbs, if used in the correct dish, and not too much or too often, they can give that extra bit of flavour that turns ‘food’ into a ‘delicious dish’. Well-performed meta-analyses can be of great value to guide general recommendations (e.g. aspirin reduces mortality in the general population) or to answer well-defined questions. A meta-analysis may also offer a comprehensive visual overview of a specific research field. A meta-analysis can also be very valuable to underscore that two treatments have, in general, an equal outcome. In a randomized controlled trial (RCT), the number of events is often lower than expected [1], due to the strict monitoring of patients in such trials and the exclusion of the most comorbid patients, which decreases the power of the study. As a result, it may become quite difficult to ascertain whether a negative result really is attributable to the absence of the difference between the treatments, or to the lack of power. A meta-analysis can in these situations increase power: even if the compilation of different larger trials, including a sufficient number of patients, does not show a difference in the outcome between two treatments, it implies that, if any difference does exist in reality, it is probably clinically negligible. Another advantage of meta-analyses is that they test the ‘generalizibility’ of effects in different situations, rather than in one unit or country. This by itself does not, however, exclude that one treatment cannot be superior or more effective in certain specific conditions.
However, meta-analyses are like cinnamon: very tasteful in small quantities and in the right dish, but if you use them too much or in the wrong dish, it ruins all other flavours and you get nausea. Just as for the cinnamon, it requires skills and insight to know when and how to use a meta-analysis. To start, one should appreciate the difference between a ‘systematic review’ and a ‘meta-analysis’. In a systematic review, a systematic overview of the available evidence is given, with a well-depicted and well-described strategy on how and on which criteria the information was searched and evaluated. The endpoint of a systematic review is still a somewhat subjective interpretation by the authors, but with the great advantage that the reader exactly knows which criteria and data were (or were not) included, so that he/she can gauge the quality and validity of the conclusions. The lack of a ‘digital’ outcome (i.e. a number, or an exact graphic representation) obliges the reader to make more efforts to squeeze out the final conclusion, making it more likely that he/she will have a balanced view. A meta-analysis on the other hand is a formal mathematical procedure with stringent presumptions, which can be used to present the data of a systematic review more objectively. The final result is an exact number, or a graphical representation, tempting the reader to skip the underlying reasoning and accept the result as is.
Just as for cinnamon, it is not the methodology of meta-analysis itself that is wrong, but the way it is used. The advent of new statistical software has made the meta-analysis accessible to everyone. However, as everybody who ever tried preparing a soufflé will know, even following literally the recipe of Escoffier does not always lead to a palatable dish. There are, just as for the soufflé, many reasons why a meta-analysis can go wrong.
|
Selection of the ingredients... |
|---|
|
TopA recipe by Escoffier...Selection of the ingredients...Finding the right ingredients...Spicing up ingredients...Blue cheese and cheddar...A dot of harisha...References |
|---|
A worthy meta-analysis can only be based on studies of sufficient quality, preferentially RCTs. As a consequence, a meta-analysis can only be of value if it can compile RCTs of sufficient quality but which by themselves were not conclusive because they lacked power. This understanding has important consequences.
Using a meta-analysis to correct for the lack of well-performed studies results in a catch 22 situation: you want to perform a meta-analysis because you have not enough well-performed RCTs, which you need to make a qualitative meta-analysis.
Like most dishes, a meta-analysis is only as palatable as the quality of the ingredients it contains. This strength is determined by several different factors: the completeness of the studies included, the quality of the studies included and the presence or absence of publication bias in the field of research.
|
Finding the right ingredients... |
|---|
|
TopA recipe by Escoffier...Selection of the ingredients...Finding the right ingredients...Spicing up ingredients...Blue cheese and cheddar...A dot of harisha...References |
|---|
In some fields, it may be difficult to find enough studies of sufficient quality, which results in ‘small’ meta-analyses, where the true effect is likely to be overestimated (as often only positive studies get published). An indication for the use of magnesium in myocardial infarction has been shown in several smaller studies, and in the meta-analyses based on them, a conclusion completely overruled by properly designed larger RCTs. When the conclusion of a meta-analysis is modified with the addition or deletion of a single trial, this should also raise suspicion to the true meaning of the results. This often happens when a large trial is added to a pleiad of smaller trials. There is an indisputable (but regrettable) negative publication bias for smaller, inconclusive studies [2]. In cancer research, for example, nearly all studies on markers are positive, which raises the suspicion that negative studies simply do not get published [3]. This bias can partially be detected by the use of funnel plots. However, there is another catch here: how can one be sure about the effect and the number of unpublished studies? A ‘positive’ funnel plot raises the suspicion of publication bias, but a negative one can never give certainty that there is none. Another method is to calculate the number of ‘unpublished negative’ trials that would be needed to abolish the positive effect of the current meta-analysis. This method is useful, but presumes that unpublished studies should demonstrate no effect. Most ethical committees and journals now request that research protocols are registered before the study starts. This should decrease the likelihood that ‘negative’ trials, or trials where the outcome was not the desired one, are classified without a trace [4]. Again, however, a systematic review seems to be preferable here, as it can verbally report on the presence of unpublished studies, whereas the mathematical rigour of meta-analysis can however not handle this type of information.
|
Spicing up ingredients... |
|---|
|
TopA recipe by Escoffier...Selection of the ingredients...Finding the right ingredients...Spicing up ingredients...Blue cheese and cheddar...A dot of harisha...References |
|---|
One way of compensating for the lack of well-performed studies is to include all studies, but ‘weigh’ their impact on the final result. Whereas this gives an impression of scientific objectivity, it is hard to conceive how including badly performed studies into a meta-analysis can result in quality results, just as adding rotten eggs to a soufflé will not improve the flavour. Preferentially, bad studies should simply be ruled out of the analysis, and at least, the weighing should be performed using objective criteria like sample size, or inverse variability, rather than a subjective evaluation of ‘quality’.
Most authors of meta-analyses proudly report the (often impressive) number of papers they have screened. It is often worrying how few publications are withheld as being ‘relevant and well performed’, as was apparent in the meta-analysis of Continuous Renal Replacement Therapy vs. Intermittent HemoDialysis by Tonelli et al. [5]. Of the 116 papers retrieved, only 6 were RCTs, and only 3 had been published as a full paper. Whereas most advocate to include all available evidence, for most of these papers not reaching the level of quality to warrant publication in an A journal, there is a reason why they are not published, and it can be questioned whether including this ambiguous information will increase the reliability of the obtained answers. Again, it appears that the abracadabra of a meta-analysis is used to camouflage the lack of real evidence or to ‘massage’ an ‘inconvenient truth’ into a ‘palatable misconception’. It is sometimes deplorable how confident authors are about the results of their meta-analysis, based on studies they themselves declared of ‘low quality’: a compilation of low-quality studies cannot result in one good meta-analysis, just as mixing low quality ingredients with spicy herbs does not result in a tasteful dish. In such conditions, it would be preferable to perform a systematic review, and just drop the meta-analysis.
Some authors use the meta-analysis to perform subgroup analyses of questions that were not the primary endpoint of the RCTs included. Again, this might be a dangerous practice, whereby the results should be seen as ‘hypothesis generating’ rather than ‘hypothesis proving’ [6]. When a subgroup analysis was performed on the data of the ISIS trial [7], zodiac sign was an important covariate to predict the impact of aspirin on the outcome of myocardial infarction. In this educational statistical experiment, it was obvious that the ‘outcome’ did not really make sense, but it becomes more problematic if the observation has a potential patho-physiological background, or is the desired outcome.
|
Blue cheese and cheddar are both cheeses but... |
|---|
|
TopA recipe by Escoffier...Selection of the ingredients...Finding the right ingredients...Spicing up ingredients...Blue cheese and cheddar...A dot of harisha...References |
|---|
A meta-analysis should only compile RCTs studying the same, well defined and circumscript question, as homogeneity is a necessary prerequisite [8,9]. Whereas this seems evident, too often meta-analyses are performed using studies where treatment A is compared to treatment B, but in different combinations with other treatments X, Y or Z. In a small study where A + X was compared to placebo, more subjects got drunk in the test group, but not a statistically significant amount. In another small study comparing A + Y with placebo, again more subjects got drunk in the test group. In a large trial, A + Z was compared to placebo. Again the study lacked power to statistically prove that subjects were more drunk in the test group. A meta-analysis concluded that the test solution A did indeed increase the risk of becoming inebriated. A further chemical analysis of the solutions showed A to be water, X whiskey, Y cognac and Z beer. Besides differences in applied treatment, inhomogeneity can be induced by differences in diagnostic criteria to accept the indication for treatment, in investigated end-points or in patient mix. Inhomogeneity of studies is also, at least partially, a consequence of the ‘publication terror’ whereby a repetitive study has a low probability to be published in a high-ranking journal. Therefore, most researchers prefer to slightly alter the protocol compared to what has already been published. Compiling all these different protocols in one meta-analysis can however lead to erroneous interpretations. Although the homogeneity can be tested, most of the methods used are neither sensitive nor reliable [10]. In a systematic review, these inhomogeneities can be explored and reported in a more explicit way than in the stringent mathematical strait jacket of a meta-analysis.
|
A dot of harisha... |
|---|
|
TopA recipe by Escoffier...Selection of the ingredients...Finding the right ingredients...Spicing up ingredients...Blue cheese and cheddar...A dot of harisha...References |
|---|
If a large number of patients are needed to prove a given point, it suggests that the overall impact is weak: it will only take a few experiments to prove that jumping from a plane without a parachute is quite unhealthy. As a consequence, the results of a meta-analysis should include the number needed to treat or the size of treatment effect as additional information. This number gives a more representative idea of the real impact of the analysed effect than the P-value, and is less dependent on the total number of included cases. A P-value can yield an idea on the statistical, but not on the clinical, significance of a finding. Statistical significance can easily be ‘improved’ by enhancing the patient number included in the analysis, without changing the clinical impact. A relevant example is the benefit of dietary protein restriction in chronic kidney disease (CKD): in the Modification of Diet in Renal Disease (MDRD) trial [11], no benefit was observed, whereas a later meta-analysis did show a protective effect [12], albeit a limited one. The meta-analysis demonstrated that dietary protein restriction slowed down the decrease of glomerular filtration by 0.53 ml/min/ 1.73 m2/year compared to normal protein intake. The importance of this effect can best be judged by comparing it to the impact of blood pressure control: a reduction of systolic blood pressure from 150 to 130 mmHg results in a slower decline of the glomerular filtration rate of 8 ml/min/ 1.73 m2/year [13]. It should also not be forgotten that most RCTs test ‘efficacy’ (does the treatment work when used as specified in the protocol), not ‘effectiveness’ (does the treatment work in real life conditions). In the MDRD e.g., there was a very tight follow-up of the patients and their nutritional intake, which is hardly feasible in real life.
In conclusion, a meta-analysis can have great value in well-circumscribed conditions, but one should be aware of the potential pitfalls. The research question should be well defined and relevant. The included studies should be of good quality (preferentially RCT), and homogeneous. The results should be interpreted in terms of the number needed to treat or the effect size observed, rather than based on the P-value alone, and accordingly, authors should report and discuss the clinical rather than the statistical impact of their findings. The readers should construct their advise to individual patients by mixing these results with their own experience and their knowledge on the preferences of the patient. At the end, the difference in skills explains why not everybody who can read a cooking recipe will turn out to be a chef de cuisine, which applies not only to those performing but also to those reading meta-analyses.
Conflict of interest statement. None declared.
|
References |
|---|
|
TopA recipe by Escoffier...Selection of the ingredients...Finding the right ingredients...Spicing up ingredients...Blue cheese and cheddar...A dot of harisha...References |
|---|
- Wanner C, Krane V, Marz W, et al. Atorvastatin in patients with type 2 diabetes mellitus undergoing hemodialysis. N Engl J Med (2005) 353:238–248.
[Abstract/Free Full Text] - Littner Y, Mimouni FB, Dollberg S, et al. Negative results and impact factor: a lesson from neonatology. Arch Pediatr Adolesc Med (2005) 159:1036–1037.
[Abstract/Free Full Text] - Browman GP. Evidence-based paradigms and opinions in clinical management and cancer research. Semin Oncol (1999) 26:9–13.[ISI][Medline]
- Browman GP. Evidence-based paradigms and opinions in clinical management and cancer research. Semin Oncol (1999) 26:9–13.[ISI][Medline]
- Tonelli M, Manns B, Feller-Kopman D. Acute renal failure in the intensive care unit: a systematic review of the impact of dialytic modality on mortality and renal recovery. Am J Kidney Dis (2002) 40:875–885.[CrossRef][ISI][Medline]
- Kasiske BL. Meta-analysis as a clinical tool in nephrology. Kidney Int (1998) 53:819–825.[CrossRef][ISI][Medline]
- ISIS-2 (Second International Study of Infarct Survival) Collaborative Group. Randomised trial of intravenous streptokinase, oral aspirin, both, or neither among 17,187 cases of suspected acute myocardial infarction: ISIS-2. Lancet (1988) 2:349–360.[Medline]
- Kasiske BL. Meta-analysis as a clinical tool in nephrology. Kidney Int (1998) 53:819–825.[CrossRef][ISI][Medline]
- Schork MA. Publication bias and meta-analysis. J Hypertens (2003) 21:243–245.[CrossRef][ISI][Medline]
- Kulinskaya E, Dollinger MB, Knight E, et al. A Welch-type test for homogeneity of contrasts under heteroscedasticity with application to meta-analysis. Stat Med (2004) 23:3655–3670.[CrossRef][ISI][Medline]
- Levey AS, Adler S, Caggiula AW, et al. Effects of dietary protein restriction on the progression of advanced renal disease in the Modification of Diet in Renal Disease Study. Am J Kidney Dis (1996) 27:652–663.[ISI][Medline]
- Kasiske BL, Lakatua JD, Ma JZ, et al. A meta-analysis of the effects of dietary protein restriction on the rate of decline in renal function. Am J Kidney Dis (1998) 31:954–961.[ISI][Medline]
- Bakris GL, Williams M, Dworkin L, et alNational Kidney Foundation Hypertension and Diabetes Executive Committees Working Group. Preserving renal function in adults with hypertension and diabetes: a consensus approach. Am J Kidney Dis (2000) 36:646–661.[ISI][Medline]
Accepted in revised form: 13. 3.08
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||