(Last Updated On: March 18, 2021)

Rowena Natividad S.Flores-Genuino, Maria Christina Filomena R. Batac, Anne Julienne M. Genuino, Ian Theodore G. Cabaluna

ABSTRACT

BACKGROUND: AMSTAR 2 enables a more detailed assessment of systematic reviews and includes non-randomised studies of healthcare interventions, compared to its earlier version, AMSTAR. We validated AMSTAR 2 in a group of systematic reviews in dermatology in the Philippines.

METHODS: We used a cohort of systematic reviews (SRs) in dermatology from the Philippine that were previously described in Part 1 of this 2-part series. The SRs included clinical trials on any intervention for the treatment or prevention of a dermatologic disease or for maintenance of healthy skin, hair or nails. Two reviewers independently extracted data and used AMSTAR 2 to appraise the methodological quality of each included SR. We determined construct validity by comparing the number of critical flaws between a set of non-Cochrane and matched Cochrane reviews, using Wilcoxon rank sum test. We
tested for interrater reliability of the AMSTAR 2 tool using Gwet’s AC1 statistic.

RESULTS: We included 20 non-Cochrane systematic reviews in dermatology by Philippine-based authors, and a set of 20 reviews from the Cochrane skin group, matched by year and randomly chosen. Construct validity testing showed a significantly greater number of AMSTAR 2 critical flaws (median 4.5 vs 0.0; z=3.64; P=0.000) and non-critical weaknesses (5 vs 2.0; z-score=3.10; P-value=0.001) by non-Cochrane reviews compared to a matched set of Cochrane skin group reviews. There was good interrater reliability (average Gwet’s AC1 statistic = 0.87) with the lowest agreement (0.62) for discussion of heterogeneity (item 14), and the highest agreement (0.97) for study selection criteria (item 3). CONCLUSION: The AMSTAR 2 was a valid and reliable tool for assessing systematic reviews using a cohort of reviews by dermatology reviews, both non-Cochrane and Cochrane. Further validation of the AMSTAR 2 is needed to determine if it can be applied to a wide variety of systematic reviews.

Keywords: AMSTAR, AMSTAR 2, dermatology, validity, reliability, systematic reviews, meta-analysis

Citation:
Flores-Genuino, RNS, Batac, MCFR, Genuino, AJM, Cabaluna, ITG. Assessment of quality of systematic reviews in dermatology using AMSTAR 2 Part 2 of 2. Validity and reliability testing of AMSTAR 2. J Phil Dermatol Soc 2019, 29(2), 6-19

1. Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. Br Med J. 2017;358:1-9. doi:10.1136/bmj.j4008
2. Gates M, Gates A, Duarte G, et al. The reliability, usability, and applicability of tools to appraise quality and risk of bias in systematic reviews: A prospective evaluation of AMSTAR, AMSTAR 2 and ROBIS. In: 26th Cochrane Colloqium. 23 October 2019.
3. Lorenz RC, Matthias K, Pieper D, et al. A psychometric study found AMSTAR 2 to be a valid and moderately reliable appraisal tool. J Clin Epidemiol. 2019;114:133-140. doi:10.1016/j.jclinepi.2019.05.028
4. Gates A, Gates M, Duarte G, et al. Evaluation of the reliability, usability, and applicability of AMSTAR, AMSTAR 2, and ROBIS: Protocol for a descriptive analytic study. 2018:1-7.
5. Hunt H, Pollock A, Campbell P, Estcourt L, Brunton G. An introduction to overviews of reviews: Planning a relevant research question and objective for an overview. Syst Rev. 2018;7(1):1-9. doi:10.1186/s13643- 018-0695-8
6. MW C. Quantitative Survey Methods in Health Research. In: Saks M, Allsop J, eds. Researching Health: Qualitative, Quantitative and Mixed Methods. Third Edition. 3rd ed. London: SAGE Publications Ltd,; 2019:253.
7. Lange R. Inter-rater Reliability. In: Kreutzer J, DeLuca J, Caplan B, eds. Encyclopedia of Clinical Neuropsychology. New York: Springer; 2011:98. doi:10.1007/978-0-387-79948-3_1203
8. Flores-Genuino R, Batac M, Genuino A, Cabaluna I. Assessing quality of systematic reviews in dermatology from the Philippines using AMSTAR 2 Part 1: Methodologic quality of dermatological systematic reviews from the Philippines. J Phil Dermatol Soc. 2020.
9.Wongpakaran N, Wongpakaran T, Wedding D, Gwet KL. A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: A study conducted with personality disorder samples. BMC Med Res Methodol. 2013;13(1):1-7. doi:10.1186/1471-2288-13-61
10. Ganeshkumar P, Murhekar M V, Poornima V, et al. Dengue infection in India: A systematic review and meta-analysis. PLoS Negl Trop Dis. 2018:2-3. doi:10.1371/journal.pntd.0006618
11. Canlas KM, Macalintal-Canlas RA, Sakai F. Efficacy of calcitonin gene- related peptide antagonists in the treatment of acute migraine: A systematic review and meta-analysis. Acta Med Philipp. 2019;53(1):44-51.
12. Lazzerini M, Wanzira H. Oral zinc for treating diarrhoea in children (Review). Cochrane Database Syst Rev. 2016;(12). doi:10.1002/14651858. CD005436.pub5
13. Lundh A, Lexchin J, Mintzes B, Jb S, Bero L. Industry sponsorship and research outcome (Review). Cochrane Database Syst Rev. 2017;Issue 2. doi:10.1002/14651858.MR000033.pub3.www.cochranelibrary.com
14. Gagnier JJ, Moher D, Boon H, Beyene J, Bombardier C. Investigating clinical heterogeneity in systematic reviews: A methodologic review of guidance in the literature. BMC Med Res Methodol 2012,. 2012;12:111. practice in Nigeria. Int J Dermatol 2012;51(9):1086-9.

More Articles

A randomized, double-blind, comparative study on the safety and efficacy of virgin coconut (Cocos nucifera l.) oil against 1% hydrocortisone lotion as an anti-inflammatory and antipruritic preparation for mosquito reactions

  Authors: Uy, Veronica S, MD; Gracia B. Teodosio, MD, FPDS; Ma. Teresita G. Gabriel, MD, FPDS; Mary Catherine T. Galang, MD; Mohammad Yoga A. Waskito, MD; Johannes F. Dayrit, MD, FPDS     Abstract Background: Virgin coconut oil (VCO) has been reported...

read more