SQLite format 3@ o @o.G
6L
utabletemplatestemplatesCREATE TABLE "templates" (
"seqnr" INTEGER PRIMARY KEY AUTOINCREMENT UNIQUE,
"phenomenon" TEXT,
"template" TEXT
)StablecategscategsCREATE TABLE "categs" (
"seqnr" INTEGER PRIMARY KEY AUTOINCREMENT,
"categ" TEXT,
"key" INTEGER
)j7tablerefsrefsCREATE TABLE "refs" (
"seqnr" INTEGER PRIMARY KEY AUTOINCREMENT,
"ref" TEXT(1024)
)P++Ytablesqlite_sequencesqlite_sequenceCREATE TABLE sqlite_sequence(name,seq);1Eindexsqlite_autoindex_templates_1templates
w+o\I5$sdUD2
~
l
Z
J
:

}n_P@0 u`K7#
{
i
W
E
3
!
~ l Z E 2
taN='H'metaanalysisG'metaanalysisE'metaanalysisDbaselineC!adjustmentB!adjustmentAgeneralG@journalG >test ?testy5#terminology# < test;#terminology):#terminology*9#terminology(8#terminology&7#terminology>6#terminology$4'randomisation_3!psychologyJ2!psychologyL1!psychologyM0!psychologyN/!propensity:.!propensity.+!prediction!prediction,!prediction=traumaB*!predictionz)!prediction(!parameters='!parametersu&!parameters4%!parameters"$!parametersh#!parameters "!parameters%!!parameters! !parameters!parameters!orthopedicI'observationalm'observationall%multiplicity%multiplicityj'metaanalysis'metaanalysis'metaanalysislabDlabjournalOjournalSjournalXjournalijournalQjournalsjournalU
journalrjournalPjournalR
journal@ journalTgeneralogeneralvgeneralwgeneralqgeneralWgeneral~endpointseendpointsZendpointsYdesigna~designb}design'design2{designqzdesign_ydentistryCxdentistryEw!adjustment<v!adjustmentpu!adjustmentt!adjustments!adjustment+r!adjustmentq!adjustmentp!adjustmento!adjustment3n!adjustment,m!adjustmentl!adjustmentk!adjustmentj!adjustment
ibaseline
hbaseline]
gbaseline[
fbaseline?
ebaselineg'observationalkd!adjustmentranking!psychologyK!propensity!predictionx!parameters!orthopedicH%neuroscienceF
%multiplicity'metaanalysisn
mediaV
lab8
journalinterimdgeneralfendpointsdesign{dentistryA
baseline\
#
templatescategs refs
mT<
wP^
W8
[DG4RPP=Steyerberg EW, Harrell Jr. FE. Prediction models need appropriate internal, internalexternal, and external validation. J Clin Epidemiol 2015, April 18.?Cawley GC, Talbot NLC. On Overfitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. J Machine Learn Res 2010;11:20792107.@Tu YK, Gunnell D, Gilthorpe MS. Simpson’s Paradox, Lord’s Paradox, and Suppression Effects are the same phenomenon – the reversal paradox. Emerging Themes in Epidemiology 2008, 5:2./Westreich D, Greenland S. The table 2 fallacy: presenting and interpreting confounder and modifier coefficients. Am J Epidemiol 2013;177:292298.
Sjölander A, Greenland S. Ignoring the matching variables in cohort studies – when is it valid and why? Stat Med 2013;32:46964708.UfQShrier I, Platt RW. Reducing bias through directed acyclic graphs. BMC Med Res Methodol 2008;8:70.Schisterman EF, Cole SR, Platt RW. Overadjustment bias and unnecessary adjustment in epidemiological studies. Epidemiol 2009;20:488495.Joseph KS, Mehrabadi A, Lisonkova S. Confounding by Indication and Related Concepts. Current Epidemiology Reports 2014;1:18.
Greenland S. Quantifying biases in causal models: classical confounding vs colliderstratification bias. Epidemiology 2003; 14:300–306.k[Hanley JA, Foster BJ. Avoiding blunders involving ‘immortal time’. Int J Epidemiol 2014;43:949961.*YCole SR, Platt RW, Schisterman EF, Haitao C, Westreich D, Richardson D, Poole C. Illustrating bias due to conditioning on a collider. Int J Epidemiol 2010;39:417420.hUBender R, Lange S. Adjusting for multiple testing: when and how? J Clin Epidemiol 2001; 54: 343–9.%ORanstam J, Cook JA. Considerations for the design, analysis and presentation of in vivo studies. Osteoarthritis Cartilage 2016 Jul 30. pii: S10634584(16)301911
EMA. Guideline on the choice of noninferiority margin. European Medicines Agency. London, 27 July 2005. EMEA/CPMP/EWP/2158/99.+EMA. Points to consider on switching between superiority and noninferiority. European Medicines Agency, London, 27 July 2000, CPMP/EWP/482/99.
EMA. Points to consider on adjustment for baseline covariates. European Medicines Agency, London, 22 May 2003, CPMP/EWP/2863/99.s
kEMA. Points to consider in missing data. European Medicines Agency, London, November 15 2001, CPMP/EWP/1776/99. EMA. ICH E9 Statistical principles for clinical trials. European Medicines Agency, London, September 1998, CPMP/ICH/363/96.Evans S. When and How Can Endpoints Be Changed after Initiation of a Randomized Clinical Trial? PLoS Clin Trials. 2007;2:e18.
Riley RD, Higgins JPT, Deeks JJ. Research Methods & Reporting: Interpretation of random effects metaanalyses. Br Med J 2011;342:d549.Higgins JPT, Thompson SG, Spiegelhalter DJ. A reevaluation of randomeffects metaanalysis. JRSS Ser A Stat Soc 2009;172:137–159.'Higgings JPT. Commentary: Heterogeneity in metaanalysis should be expected and appropriately quantified. Int J Epidemiol 2008;37:1158–1160Abraha I, Montedori A. Modified intention to treat reporting in randomised controlled trials: systematic review. BMJ. 2010;340:c269.xuAsh AS, Fienberg SE, Louis TA, Normand SLT, Stukel TA, Utts J. Statistical issues in assessing hospital peformance. Commissioned by the Committee of Presidents of Statistical Societies. Downloaded from https://www.cms.gov/Medicare/QualityInitiativesPatientAssessmentInstruments/HospitalQualityInits/ Downloads/StatisticalIssuesinAssessingHospitalPerformance.pdf. Wasserstein RL, Lazar NA. The ASA's Statement on pValues: Context, Process, and Purpose, The American Statistician, 2016;70:129133.CRuxton GD. The unequal variance ttest is an underused alternative to Student's ttest and the Mann–Whitney U test. Behavioral Ecology 2006;17:688–690.
r~xrl
t
M
} 5 tY'SummaryThis manuscript presents an evaluation of X. Data from Y subjects are analysed statistically, and the authors find that Z. The manuscript is generally wellwritten, but I have some methodological comments.=uPrediction or estimationThe authors describe their aim as to assess predictors. However, the presented conclusions from the assessment are presented in terms of risk. It is thus unclear if it is the authors' ambition to develop a statistical model for individual prediction or a model for evaluating average risk factors. The former approach should have been based on an evaluation of sensitivity and specificity and include validation to avoid overfitting, see Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. European Heart Journal 2014;35:1925–1931. The latter approach would need to be based on parameter estimation and include adjustment for potential confounding factors, see e.g. Shrier I, Platt RW. Reducing bias through directed acyclic graphs. BMC Med Res Methodol 2008;8:70 and Westreich D, Greenland S. The table 2 fallacy: presenting and interpreting confounder and modifier coefficients. Am J Epidemiol 2013;177:292298. A rationale for the adjustment, in terms of causeeffect relationships, would be expected. In both cases, I recommend complying with developed checklists, the TRIPOD Statement for prediction and the STROBE Statement for risk factor estimatimation (https://equatornetwork.org/).\Validity testingConfounding bias is a validity problem and cannot be solved by hypothesis testing as pvalues are precision measures. Adjustment for confounding factors needs to be based on assumptions regarding causeeffect relationships. For example, while including a confounder in the statistical model will reduce confounding bias, the inclusion of a mediator or collider will induce adjustment bias, see e.g. Shrier I, Platt RW. Reducing bias through directed acyclic graphs. BMC Med Res Methodol 2008;8:70. Please provide a rationale for the adjustment variables in terms of cause and effect.fATable 2Table 2 seems to represent a case of the socalled Table 2 fallacy, see Westreich D, Greenland S. The table 2 fallacy: presenting and interpreting confounder and modifier coefficients. Am J Epidemiol 2013;177:292298.b9Table 1Table 1 presents pvalues from tests of baseline imbalance after randomisation. Such pvalues are generally considered misleading and the CONSORT Statement guidelines recommends not presenting them, see also Roberts C, Torgerson DJ. Understanding controlled trials: baseline imbalance in randomised controlled trials. Br Med J 1999;319:185.. hETable 1Table 1 provides a description of the background data but includes pvalues. These do, however, measure the inferential uncertainty visàvis specific hypothesis. They cannot be interpreted as indicators of practical importance or scientific relevance and are not useful for identifying confounders. Please explain the purpose of the presentation. 7Clinical significancePvalues indicate inferential uncertainty visàvis specific hypotheses. They do not indicate whether or not a finding is clinically relevant. To show that a specific estimated effect is clinically relevant, first define a minimal clinically significant difference (MCSD), then show that only clinically significant effects are included in the confidence interval of the estimated effect.
'No differenceStatistical nonsignificance is not evidence of equivalence. It just indicates uncertainty, and this cannot be used as an argument for "no difference". An equivalence trial or a noninferiority trial is necessary to show equivalence or noninferiority.
[ S!ParametersThe results are presented in terms of odds ratios. Can these be interpreted in terms of relative risk? Or would such an interpretation be misleading (see Davies HTO. When can odds ratios mislead? BMJ 1998;316:989). In the latter case, I recommend converting the odds ratio to the corresponding relative risk (see Zhang J, Yu KF. What’s the Relative Risk? A Method of Correcting the Odds Ratio in Cohort Studies of Common Outcomes. JAMA 1998;280:16901691) or using a statistical method that provides direct estimates of the relative risk (see e.g. McNutt LA, Wu C, Xue X, Hafner JP. Estimating the Relative Risk in Cohort Studies and Clinical Trials of Common Outcomes. Am J Epidemiol 2003;157:940–943).
[w
'WMetaanalysisObservational studies differ from randomised trials in the respect that validity problems cannot be prevented in the study design, e.g. by randomisation, concealed allocation, and blinding. Instead, the statistical analysis needs to include considerations regarding validity oriented adjustments. Please describe in more detail how other sorts of bias than publication bias were evaluated in the review. See also Faber T, Ravaud P, Riveros C, Perrodeau C, Dechartres A. Metaanalyses including nonrandomized studies of therapeutic interventions: a methodological review. BMC Medical Research Methodology 2016:35.Y/Confounding testsMultivariable modeling is performed using factors with significant associations in univariable analysis. Developing a statistical model for effect estimation can, however, not be performed on the basis of statistical significance because pvalues are measures of statistical precison and the model development should be made with respect to validity. The inclusion of variables needs instead to be based on assumptions regarding cause and effect, see e.g. Shrier I, Platt RW. Reducing bias through directed acyclic graphs. BMC Med Res Methodol 2008;8:70. Please provide a rationale for the included covariates in terms of causeeffect relationships. For the presentation of results, see Westreich D, Greenland S. The table 2 fallacy: presenting and interpreting confounder and modifier coefficients. Am J Epidemiol 2013;177:292298.
M/ =1Prediction or estimationI recommend avoiding the term "predictor" as this refers to individual prediction and not to the average effects that are estimated by the autors."
#1TerminologyThe term "multivariate" is used incorrectly, see Hidalgo B, Goodman M. Multivariate or multivariable regression? Am J Public Health 2013;103:13.
Ni]
'n=
} 0Y&"a}NT5Felson DT, Cupples LA, Meenan RF. Misuse of statistical methods in Arthritis and Rheumatism. 1982 versus 196768. Arthritis Rheum 1984;27:10181022.S'Welch GE, Gabbe SG. Review of statistics usage in the American Journal of Obstetrics and Gynecology. Am J Obstet Gynecol 1996;175:11381141.
R#Hanif A, Ajmal T. Statistical Errors in Medical Journals (A Critical Appraisal). Annals of King Edward Medical University, 2011;17:178182.QPrescott RJ, Civil I. Lies, damn lies and statistics: Errors and omission in papers submitted to Injury 20102012. Injury 2013;44:611.APHassan S, Yellur R, Subramani P, Adiga P, Gokhale M, Iyer MS, Mayya SS. Research Design and Statistical Methods in Indian Medical Journals: A Retrospective Survey. PLoS One 2015;10:e0121268>OWu S, Wei X, Gao Q, Lu J Ma X, Wu C, He Q, Wu M, Wang R, Xu J, He j. Misuse of statistical methods in 10 leading Chinese medical journals in 1998 and 2008. Sci World J 2011;11:21062114.}NCaperos JM, Pardo A. Consistency errors in pvalues reported in Spanish psychology journals. Psicothema. 2013;25:408414.M Nuijten MB, Hartgerink CHJ, van Assen MALM, Epskamp S, Wicherts JM. The prevalence of statistical reporting errors in psychology (1985–2013). Behav Res 2015 DOI: 10.3758/s1342801506642. Epskamp, S. & Nuijten, M. B. (2016). statcheck: Extract statistics from articles and recompute p values. Retrieved from http://CRAN.Rproject.org/package=statcheck. (R package version 1.2.2)./LcPetrocelli JV, Clarkson JJ, Whitmire MB, Moon PE. When ab ≠ c – c’: published errors in the reports of singlemediator models. Behav Res Methods 2013;45:595601.KBakker M, Wicherts JM. The (mis)reporting of statistical results in psychology journals. Behav res methods 2011;43:666678.J)Schatz P, Jay KA, McComb J, McLaughlin JR. Misuse of statistical tests in Archives of Clinical Neurospsychology publications.2005;8:10531059.I7Park MS, Kim SJ, Chung CY, Choi IH, Lee SH, Lee KM. Statistical Consideration for Bilateral Cases in Orthopaedic Research. JBJS Am 2010;92:17321737.#HKBryant D, Havey TC, Roberts R, Guyatt G. How Many Patients? How Many Limbs? Analysis of Patients or Limbs in the Orthopaedic Literature. JBJS Am 2006;88:4145.$GMIalongo C, Bernardini S. Preanalytical investigations of phlebotomy: methodological aspects, pitfalls and recommendations. Biochem Med (Zagreb) 2017;27:177191.#FKNieuwenhuis S, Forstmann BU, and Wagenmakers EJ. Erroneous analyses of interactions in neuroscience: a problem of significance. Nat Neurosci 2011;14:11051107.=ELucena C, Lopez JM, Abalos C, Robles V, Pulgar R. Statistical errors in microleakage studies in operative dentistry. A survey of the literature 20012009. Eur J Oral Sci 2011;6:504510.D
Simundic AM, Nikolac N. Statistical errors in manuscripts submitted to Biochemia Medica journal. Biochemia Med 2009;19:294300.'CSLucena C, Lopez JM, Pulgar R, Abalos C, Valderrama MJ. Potential errors and misuse of statistics in studies on leakage in endodontics. Int Endod J 2013;46:323331.6BqBurke DA, Whittermore SR, Magnuson DS. Consequences of common data analysis inaccuracies in CNS trauma injury basic research. J Neurotrauma 2012, November 27 [Epu ahead of print]wAsKim JS, Kim DK, Hong SJ. Assessment of errors and misused statistics in dental research. Int Dent J 2011;3:163167.:@yFernandesTaylor S, Hyun JK, Reeder RN, Harris AH. Common statistical and research design problems in manuscripts submitted to highimpact medical journals. BMC Res Notes 2011;4:304.?Harvey LA. Statistical testing for baseline differences between randomised groups is not meaningful. Spinal Cord 2018;56:919.>Haynes B. Can it work? Does it work? Is it worth it? The testing of healthcare interventions is evolving. BMJ. 1999;319:652–653=Zhang J, Yu KF. What’s the Relative Risk? A Method of Correcting the Odds Ratio in Cohort Studies of Common Outcomes. JAMA 1998;280:16901691.
Z
bv
2 H+HbX}mWalter S, Tiemeier H. Variable selection: current practice in epidemiological studies. Eur J Epidemiol 2009;24:733–736.%lONiven DJ, Berthiaume LR, Fick GH, Laupland KB. Matched casecontrol studies: a review of reported statistical methodology. Clinical Epidemiology 2012:4;99–110.&kQLiu C, Cripe TP, Kim MO. Statistical Issues in Longitudinal Data Analysis for Treatment Efficacy Studies in the Biomedical Sciences. Mol Ther 2010;18:1724–1730.j
Lazic SE. The problem of pseudoreplication in neuroscientific studies: is it affecting you analysis? BMC Neuroscience 2010;11.5.iVail A, Gardener E. Commons statistical errors in the design and analysis of subfertility trials. Hum Rep 2003;18:10001004.chKKnol MJ, Duijnhoven RG, Grobbee DE, Moons KGM, Groenwold RHH. Potential Misinterpretation of Treatment Effects Due to Use of Odds Ratios and Logistic Regression in Randomized Controlled Trials. PLoS One. 2011; 6(6): e21248..Schuster T, Lowea WK, Platt RW. Propensity score model overfitting led to inflated variance of estimated odds ratios. J Clin Epidemiol 2016 September, Doi:10.1016/j.jclinepi.2016.05.017.King G, Nielsen R. Why Propensity Scores Should Not Be Used for Matching. Political Analysis 2019;27:120. DOI: 10.1017/pan.2019.11 ,EKahan BC, Morris TP. Reporting and analysis of trials using stratified randomisation in leading medical journals: review and reanalysis. BMJ 2012;345:e5840.+}Tangri N, Kitsios GD, Su SH, Kent DM. Accounting for center effects in multicenter trials. Epidemiology 2010;21:912913.N*!Koletsi D, Pandis N, Polychronopoulou A, Eliades T. What’s in a title? An assessment of whether randomized controlled trial in the title means that it is one. Am J Dentofacial Orthop 2012:141:679685.)The International Statistical Institute. The Oxford Dictionary of Statistical Terms. Oxford University Press, New York 2003.
(Peters T. Multifarious terminology: multivariable or multivariate? univariable or univariate? Paediatric Perinatal Epidemiol 2008;22:506.t'mLewallen S, Courtright P. Epidemiology in Practice: CaseControl Studies. Community Eye Health. 1998;11:57–58.d&MHidalgo B, Goodman M. Multivariate or multivariable regression? Am J Public Health 2013;103:13.}%Graubard BI, Korn EL. Inference for Superpopulation Parameters Using Sample Surveys. Statistical Science 2002;17:73–96.Q$'Ernst E, Pittler MH. Efficacy or effectiveness? J Int Med 2006;260:488–490._#CAltman DG, Bland JM. Quartiles, quintiles, centiles, and other quantiles. BMJ 1994;309:996.!"GMcNutt LA, Wu C, Xue X, Hafner JP. Estimating the Relative Risk in Cohort Studies and Clinical Trials of Common Outcomes. Am J Epidemiol 2003;157:940–943.!Dignam JJ, Zhang Q, Kocherginsky MN. The Use and Interpretion of Competing Risks Regression Models. Clin Cancer Res 2012;18:2301–2308.7 sGreenland S, Schlesselman JJ, Criqui, MH. The fallacy of employing standardized regression coefficients and correlations as measures of effect. Am J Epidemiol. 1986;123:203–208.?Davies HTO. When can odds ratios mislead? BMJ 1998;316:989.#Coutinho LMS, Scazufca M, Menezes PR. Methods for estimating prevalence ratios in crosssectional studies. Rev Saúde Pública 2008;42:16.JBarros AJD, Hirakata VN. Alternatives for logistic regression in crosssectional studies: an empirical comparison of models that directly estimate the prevalence ratio. BMC Med Res Method 2003, 3:21_Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. European Heart Journal 2014;35:1925–1931.
}e}d
MSalanti G. Indirect and mixedtreatment comparison, network, or multipletreatments metaanalysis: many names, many benefits, many concerns for the next generation evidence synthesis tool. Res Synth Methods. 2012;3:80–97. 3Baethge, C., GoldbeckWood, S. & Mertens, S. SANRA—a scale for the quality assessment of narrative review articles. Res Integr Peer Rev 2019;4,5.
/SMethodsThe ICMJE recommendation is to "Describe statistical methods with enough detail to enable a knowledgeable reader with access to the original data to judge its appropriateness for the study and to verify the reported results". The current manuscript does not comply with the recommendation.
h#=TerminologyThe ICMJE recommends avoiding 'nontechnical uses of technical terms in statistics, such as “random” (which implies a randomizing device), “normal,” “significant,” “correlations,” and “sample.”'.
K#TerminologyThe manuscript presents an observational study, but it seems to be based on trialrelated terminology including terms such as "efficacy", "primary outcome" and "serious adverse events". These terms have clear definitions in randomised trials but not in observational studies. For example, an adverse event is generally known as any untoward medical occurrence that has a temporal but not necessarily causal relation to the studied treatment. The subgroup of adverse events that are causally related to the treatment are usually described as treatmentrelated adverse events, and if they cause death, are lifethreatening, or leads to hospital treatment, they are usually described as serious treatmentrelated adverse events. While treatmentrelated adverse events may be registered in an observational database, I doubt that temporally related adverse events are can be identified or even defined, in a retrospective study. Primary and secondary outcomes usually play important roles in strategies for addressing multiplicity issues in confirmatory trials, but multiplicity issues are hardly relevant in observational studies, see e.g. Bender R, Lange S. Adjusting for multiple testing: when and how? J Clin Epidemiol 2001; 54: 343–349. As for efficacy, see Ernst E, Pittler MH. Efficacy or effectiveness? J Int Med 2006;260:488–490. L!ConclusionPlease describe in more detail the empirical support for the authors' conclusion. Include information about the estimation uncertainty (confidence intervals) of effect and safety estimates.'oMetaanalysisNetwork metaanalyses are based on underlying assumptions such as of transitivity, i.e. that there are no systematic differences between the comparisons other than the treatments being compared (see Salanti G. Indirect and mixedtreatment comparison, network, or multipletreatments metaanalysis: many names, many benefits, many concerns for the next generation evidence synthesis tool. Res Synth Methods 2012;3:80–97). Are these assumptions fulfilled and the calculated effect estimates valid?4#UConfoundingThe authors analysed the influence of potential confounders and did not find any "significant difference". Please specify if the word "significant" here refers to practical importance (clinical significance) or to inferential uncertainty (statistical significance). In the former case, what is the minimal clincally significant difference? and was this included in or excluded from the parameter estimate's confidence interval? In the latter case, why would this be relevant? How is the tested null hypothesis related to the estimated effect size?