A Proposed Categorization of Meta-Analysis, Their Respective Example Conceptual Frameworks, and Applicable R Packages for Education Research: A Review

Categorization


Introduction
Meta-analysis methods have evolved, leading to the emergence of different categories of meta-analysis.However, classifications of meta-analysis are rarely documented in existing literature.This can be viewed as a significant gap.If a classification is not clear, it makes it difficult for research novices to come to terms with the different categories of metaanalyses.Thus, a classification of meta-analysis is needed and must be adequately explained.Another omission is that metaanalysis studies do not typically make use of conceptual frameworks to explain the foundation of their respective studies.However, it may be will very helpful to include a conceptual framework in a meta-analysis study.This would make it easier for a wider audience to comprehend a meta-analysis study.Therefore, this review article aims to propose a classification of common categories of meta-analyses with explanations for each category.In addition, conceptual frameworks will also be included to accompany each category of meta-analysis in our proposed classifications.Finally, R packages suitable for these categories of meta-analyses will be presented.After reading this article, it is anticipated that our audience will gain an understanding of the categories of meta-analyses from basic to advanced, know how to develop a conceptual framework for their meta-analysis studies, and know which R packages are suited to their meta-analysis studies.Hopefully, our presentation in this paper will serve as a good starting point for novice researchers to learn about meta-analysis and be able to use it in their meta-analysis research.

Definition of Meta-analysis
A meta-analysis is a quantitative technique that combines the results of multiple primary studies and ends with a summative figure/estimate (i.e.pooled effect size).A meta-analysis essentially is a summary, in effect, of numerous studies to form one finding that covers the area of study (Pigott & Polanin, 2019).The pooled effect can be computed in two ways: fixed-and random-effects models.The fixed-effects model assumes that the true effect size is fixed across studies.The random-effects model assumes that the true effect sizes are varied across studies.The latter is preferred when the studies of a meta-analysis study are heterogeneous.

Conceptual Framework and Hypothesis
It is not common to include a conceptual framework in a meta-analysis study.However, we think, for learning purposes, an understanding of conceptual frameworks would help learners who are less familiar with meta-analysis to understand the concept of analyzing meta-analysis data much more easily.When prompted with "define, a conceptual framework for research", the Bard-generated text indicated that a conceptual framework is a "visual representation of the key variables and [their] relationships involved in your [meta-analysis] study" (Google, 2023).In addition, based on Google, conceptual frameworks can take different forms including diagrams, flowcharts, tables, or even written descriptions.Finally, a conceptual framework can serve as a base for further hypothesis development for a meta-analysis study.

R Packages for Meta-Analysis
Several R packages can conduct meta-analyses, such as meta package, dmetar package, metafor package, NetMeta package, rmeta package, MAd package, metaSEM package, and esc package.These packages are the common ones that are used to conduct meta-analysis.However, they have different strengths.It is recommended that users explore their strengths to make certain that the packages chosen will fit their meta-analysis needs.The R packages suitable for which meta-analysis categories will be recommended after the introduction of each category of meta-analysis presented ahead.

A Proposed Categories of Meta-Analyses
This paper reviewed common categories of meta-analyses.Based on our literature review, different categories of metaanalyses exist.Thus, this paper proposes five common categories of meta-analyses, namely, univariate meta-analysis, multivariate meta-analysis, network meta-analysis, multilevel meta-analysis, and structural equation meta-analysis.These five common categories of meta-analyses are captured in Figure 1.The five categories of these meta-analyses are described following Figure 1.The categorization captured in Figure 1 is developed by the authors of this paper and does not exist in existing literature.

Univariate Meta-Analysis
The idea of univariate meta-analysis rests on pooling a single effect size of treatment/intervention of several primary studies.When prompted with "define, a univariate meta-analysis", the Bard-generated text stated that a univariate meta-analysis focuses on "synthesizing the evidence on a single outcome variable across multiple studies.It combines the results of these studies to provide a more comprehensive and statistically robust estimate of the effect size for that specific outcome" (Google, 2023).An example of a univariate meta-analysis conceptual framework is captured in Figure 2 below to help the audience of this paper see a univariate meta-analysis in a visual form.Figure 2 draws upon the work of Aloe et al. (2013).Their paper, however, does not provide an example of a univariate meta-analysis conceptual framework.Based on Figure 2, there is only one intervention called classroom management self-efficacy (CMSE) and one outcome (pooled effect size), emotional exhaustion (EE).In short, univariate meta-analysis focuses on one intervention and one outcome (pooled effect size) of multiple studies.The R package recommended for univariate meta-analysis is the meta package (Balduzzi et al., 2019).Based on Figure 2, a hypothesis is that the effect of CMSE on EE is significant.

Multivariate Meta-Analysis
The idea of multivariate meta-analysis focuses on measuring multiple outcomes (effect sizes) of one treatment/intervention.This differs from univariate meta-analysis which only focuses on measuring one outcome (effect size).When prompted with "multivariate meta-analysis", the Bard-generated text stated that "while univariate meta-analysis focuses on analyzing a single outcome across multiple studies, multivariate meta-analysis offers a more holistic approach.It delves into the complex interplay between two or more outcome variables measured in the same studies.This allows for a richer understanding of the overall effects and their interdependencies" (Google, 2023).An example of a multivariate meta-analysis conceptual framework is captured in Figure 3 below to help the audience of this paper see a multivariable meta-analysis in a visual form.Figure 3 draws upon the work of Aloe et al. (2013).Their paper, however, does not provide an example of a multivariate meta-analysis conceptual framework.Based on Figure 3, there is only one intervention called classroom management self-efficacy (CMSE) and three outcomes (pooled effect sizes), namely, emotional exhaustion (EE), depersonalization (DP), and personal accomplishment (PA).In short, multivariate meta-analysis focuses on one intervention but multiple outcomes (pooled effect sizes).The R packages recommended for multivariate meta-analysis are the metaSEM package (Cheung, 2015) and metafor package (Viechtbauer, 2010).The recommended packages need to be used in combination.Based on Figure 3, a hypothesis is that CMSE has a significant effect on EE, DP, and PA.

Multilevel Meta-Analysis
When prompted with "multilevel meta-analysis", the Bard-generated text stated that "Multilevel meta-analysis, also known as hierarchical meta-analysis, takes the analysis of combined research findings a step further.It delves into situations where studies themselves are nested within broader contexts or groups" (Google, 2023).In short, the idea of multilevel meta-analysis focuses on the cluster effects on a single outcome (effect size).Here, users need to know the concepts of meta-analysis, multilevel modeling, and R to learn multilevel meta-analysis.Thus, it makes it quite challenging to learn multilevel metaanalysis.An example of a multilevel meta-analysis is captured in Figure 4 below to help the audience of this paper see a multilevel meta-analysis in a visual form.Figure 4 draws upon the work of Bornmann et al. (2007).Their paper, however, does not provide an example of a multilevel meta-analysis conceptual framework.Based on Figure 4, the levels of IVs are classified into three levels: one micro and two macro levels.The two macro levels are viewed as cluster (contextual) effects.The three IV levels affect only one outcome (pooled effect size).In short, multilevel meta-analysis focuses on the effects of cluster levels on a single outcome (pooled effect size).The R package recommended is the metaSEM package (Cheung, 2015).Based on Figure 4, we can propose a hypothesis that all IVs (from different levels) have significant effects on DV (pooled effect size).

Figure 4. Multilevel meta-analysis conceptual framework 2.2.4. Network Meta-analysis
When prompted with "Network meta-analysis (NMA)", the ChatGPT-generated text stated that "Network meta-analysis (NMA) is a powerful statistical technique that goes beyond traditional pairwise comparisons in research.It takes you on a journey through a web of interconnected studies, allowing you to analyze the effectiveness of multiple interventions indirectly compared to each other, even if they haven't been directly compared in single studies" (OpenAI, 2023).When prompted with "Network meta-analysis", the Bard-generated text stated that "Network meta-analysis (NMA), also known as multiple treatment comparison meta-analysis or mixed treatment comparison, is a statistical technique used to compare and integrate evidence from multiple studies that directly or indirectly compare different interventions.This method is particularly useful when there are multiple treatments for a particular condition and head-to-head comparisons are limited or absent" (Google, 2023).An example of a network meta-analysis is captured in Figure 5 below to help the audience of this paper see a network meta-analysis in a visual form.Figure 5 draws upon the work of Chayaban et al. (2021).Their paper, however, does not provide an example of a network meta-analysis conceptual framework.Based on Figure 5, there are three different models: traditional teaching model (A), interaction teaching model (B), and coaching teaching model (C).Thus, there are three outcomes (pooled effect sizes).These outcomes can be compared directly (A and B, and A and C) and indirectly (B and C).In short, the idea of network meta-analysis focuses on measuring multiple outcomes of multiple treatments/interventions.Unlike multivariate meta-analysis JPSE 5 (3): 1-7 Savatsomboon, G., Yurayat, P., Chanprasitchai, O., Narkbunnum, W., Sharma, J. K., & Svetsomboon, S. (which only shows one treatment/intervention), network meta-analysis shows multiple treatments/interventions and their respective outcomes (effect sizes).For network meta-analysis, the netmeta package for network meta-analysis (Balduzzi et al., 2023) is recommended.Based on Figure 5, a hypothesis is that the pooled effect size between A and B is significant.In addition, a hypothesis is that the pooled effect size between A and C is significant.Furthermore, the pooled effect size between B and C can be indirectly compared by comparing the weights of the effect sizes between A and B, and A and C. Based on Figure 5, a hypothesis is that the pooled effect size (between A and B) is significant.In addition, the pooled effect size (between A and C) is significant.

Structural Equation Meta-analysis
When prompted with "structural equation meta-analysis (MASEM)", the ChatGPT-generated text stated that "Structural Equation Modeling Meta-Analysis (MASEM) is an advanced statistical approach that combines structural equation modeling (SEM) with meta-analysis techniques.MASEM allows researchers to examine relationships among variables within each study using SEM while also synthesizing these relationships across multiple studies through meta-analysis.This approach is particularly useful when dealing with complex models and investigating the generalizability of relationships [in the model]" (OpenAI, 2023).This category of meta-analysis is also called meta-analysis structural equation modeling (MASEM).This is probably the most complex category of meta-analysis presented in this paper in terms of complexities and analysis using R.The idea of MASEM is to conduct a structural equation model using the meta-analysis data.Figure 6 is an example of a conceptual framework for MASEM.The R software package that can conduct MASEM is called metaSEM.We hope that there will be more R packages available to run MASEM soon.To conduct MASEM, users need to understand R and SEM.The learning curves for both R and SEM are quite steep.Now, let's try to develop a MASEM conceptual framework.MASEM shares the conceptual framework with a traditional SEM but requires a different data input.In addition, the calculations between MASEM and SEM are not quite the same.One essential capability is that the metaSEM package can compute heterogeneity for MASEM.This is a necessary statistic for MASEM.Based on the metaSEM package, two types of MASEM can be analyzed by the package.The first one is the one-stage SEM (OSSEM), which does not compute the heterogeneity.Whereas the two-stage SEM (TSSEM) does compute heterogeneity.Some authors, for example, Jaroentaku & Kao-iean (2020), reported the heterogeneity value in their work.On the other hand, many authors do not report the heterogeneity values in their papers, for example, the work of Than et al. (2021), Thanapad et al. (2023), andWangsin (2017).These authors may have to mention which approach they are using, OSSEM or TSSEM.An example of a MASEM conceptual framework is captured in Figure 6 below to help the audience of this paper see a meta-analysis structural equation modeling in a visual form.Figure 6 draws upon the work of Becker & Schram (1994).Their paper, however, does not provide an example of a MASEM conceptual framework.Based on Figure 6, the effects of IVs (Spatial and Verbal) on Math are computed using the meta-analysis structural equation modeling approach.For MASEM, we recommend the metaSEM package (Cheung, 2015).Based on Figure 6, a hypothesis is that the spatial and verbal exam scores significantly affect the math score.Savatsomboon, G., Yurayat, P., Chanprasitchai, O., Narkbunnum, W., Sharma, J. K., & Svetsomboon, S.

Conclusions
As mentioned in the introduction, the categorization of meta-analyses is rarely seen, if ever.In addition, explanations of the differences between different categories of meta-analyses are also rarely seen.The application of a conceptual framework is also not typically used in a meta-analysis study.By now, the categories of meta-analyses and their respective conceptual frameworks should become clear to researchers.The R packages suitable for the different categories of meta-analyses are recommended in the paper.Thus, it is recommended that the audience of this paper use the categories of meta-analyses presented in this paper as their road map to learn the concepts of different categories of meta-analyses, develop conceptual frameworks for their metaanalysis studies, and select the right R packages for conducting the meta-analyses of their choices.What needs to be done beyond this paper?It would be very helpful if more learning resources were available for each category of meta-analysis presented in this paper.Therefore, it is recommended that more learning resources (e.g.tutorial papers) be made available.Finally, the categorization of meta-analyses, their respective conceptual framework development, and applicable R packages presented in this paper can be applied to other social science research contexts (e.g.economics, marketing, and business management).

Figure 6 :
Figure 6: Structural equation modeling meta-analysis conceptual framework