Enhancing Educational Technology Research Through Mixed Methods and Visual Joint Displays

Introduction

Educational Technology (EdTech) is a vast and evolving field, drawing increasing attention for its role in formal education and lifelong learning. As EdTech's influence expands, so does the scrutiny of the research methods used to evaluate its effectiveness. Mixed methods research (MMR), which integrates qualitative and quantitative approaches, offers a promising avenue for addressing complex educational problems. However, the quality and rigor of MMR in EdTech are often criticized. Explicit integration of qualitative and quantitative data remains scarce, with few studies implementing recommended integration strategies like visual joint displays. This article explores the procedures, opportunities, and practical challenges of MMR integration, focusing on visual joint displays as an analytical tool for data interpretation and reporting.

The Rise of Mixed Methods Research in EdTech

Motivated by the potential added value of combining qualitative and quantitative methods, educational researchers are increasingly adopting MMR designs. In the EdTech field, researchers like Poth and Ngulube and Ukwoma have highlighted MMR's usefulness in addressing the multidimensional challenges in teaching and learning environments in higher education. MMR allows access to multiple perspectives, improves the validity of evidence, and generates mixed insights unattainable through a single method.

Systematic mapping of research patterns has become a lively and well-established research area. Common concerns in the literature include ambiguity and confusion toward how best to approach methodological design, a lack of theoretical development, and a tendency for basic forms of descriptive research where researchers seek out evidence of ‘what works’ when technology is applied in the classroom.

Integration: The Defining Element of MMR

Integration is the hallmark of MMR, distinguishing it from monomethod and multimethod research. It involves deliberately mixing quantitative and qualitative components throughout the research process to generate a more holistic understanding. Researchers combine distinct research elements to achieve a common research purpose. However, integration poses challenges due to the inherent differences between quantitative and qualitative approaches.

Several authors have developed practical strategies to help researchers integrate effectively, focusing on the purpose and practical aspects of the integration process. These strategies include building, connecting, exploring, comparing, and expanding:

Building occurs when data from one component informs the data collection and analysis of the subsequent component.
Connecting is similar to building, but data from one component determines the sampling strategy for the other.
Exploring involves collecting and analyzing qualitative data to examine a concept before a follow-up quantitative study.
Comparing entails examining the relationship between quantitative and qualitative data on a phenomenon of interest.
Expanding implies using both qualitative and quantitative data collection methods to develop an expanded but overlapping perspective on a phenomenon.

Despite the recognized value of integration, MMR studies that explicitly integrate quantitative and qualitative components are infrequent in EdTech, and even fewer utilize recommended integration strategies and joint displays. Failure to integrate limits the potential for a deeper understanding of the phenomena under investigation.

Visual Joint Displays: Enhancing Integration in MMR

Joint displays offer a structured approach to integrative thinking by facilitating the systematic synthesis of quantitative and qualitative data and the production of integrated interpretations of MMR findings. Joint display analysis is an iterative process where researchers use the joint display structure to identify connections between qualitative and quantitative constructs and generate meta-inferences based on the 'fit' of the two data types. This process allows researchers to gain new insights beyond those provided by the individual components.

Joint displays can facilitate the "cognitive process of merging, comparing, relating, and linking qualitative and quantitative data or results", allowing for "integration and inferential transparency".

Types of Joint Displays

McCrudden et al. (2021) identified three main types of joint displays applicable to various research use cases in MMR studies:

Quantitative Results Matrix: Juxtaposes quantitative findings to identify individuals who vary systematically from most of the sample.
Integrated Results Matrix: Merges quantitative and qualitative findings to enable comparison and the generation of meta-inference displays.
Integrated Visual Display: Represents the integration of quantitative and qualitative findings through visuals like graphs, images, or figures.

While early joint displays primarily used integrated results matrices, recent innovations have led to integrated visual displays incorporating diagrams, figures, maps, and illustrations. Visual joint displays can reduce the cognitive burden of integration analysis and reporting, making integration more explicit and facilitating the interpretation of complex findings.

Read also: Blue Sea Consulting Services

Examples of Visual Joint Displays in EdTech

Bustamante (2019) presented an outstanding example of a visual joint display in EdTech, evaluating a professional development program on Web 2.0 technologies for teachers of Spanish using a convergent design. Based on the Technological Pedagogical Content Knowledge (TPACK) model, the author developed a circular joint display with concentric rings representing quantitative findings, qualitative findings, and integrated findings for each dimension of the TPACK model. This model facilitated comprehensive integration of theory across all study components.

Another example is Poth (2018), who conducted an embedded MMR case study design of two technology-enhanced formative assessment classroom strategies. Their visual joint display presented qualitative data sources and thematic categories on the left, quantitative data sources and items on the right, and mixed insights emerging from integrating the quantitative findings with the qualitative thematic categories in the center.

Steps involved in analytical MMR strategy

Haynes-Brown and Fetters (2021) described the iterations involved in creating a joint display with histograms in an explanatory sequential MMR study examining classroom teachers' use of technology. These authors described how additional rounds of analysis enabled them to reveal insights into the integrated findings by evaluating the organizational intent, analytical intent, and effectiveness of the visuals created in each iteration.

The Benefits of Visual Joint Displays

Visual joint displays offer several benefits for MMR in EdTech:

Enhanced Integration: Facilitate the systematic synthesis of quantitative and qualitative data, fostering deeper understanding.
Improved Transparency: Provide a clear and structured approach to integrative thinking, making the integration process more explicit.
Reduced Cognitive Burden: Simplify the analysis and reporting processes compared to tabular joint displays.
Effective Communication: Aid in the interpretation of complex integration findings, making them more accessible to a wider audience.
Theoretical Integration: Help incorporate robust theoretical frameworks into the research design, strengthening the study's foundation.

Practical Considerations and Challenges

Despite their potential, visual joint displays also present practical considerations and challenges:

Read also: Shaping the Future of Translation

Complexity: Constructing and interpreting joint displays can be complex, requiring expertise in both quantitative and qualitative methods.
Time-Consuming: The iterative nature of joint display analysis can be time-consuming.
Subjectivity: The interpretation of integrated findings may be influenced by researcher bias.
Limited Examples: Few examples of visual joint display analysis exist in EdTech, limiting guidance for researchers.

Addressing Criticisms of EdTech Research

The COVID-19 pandemic has brought greater public attention and academic scrutiny to the influence of EdTech at all stages of formal education and lifelong learning, including criticism of the quality and rigor of EdTech research. Critical commentary on the methods used to solve complex educational problems remains an important and consistent debate. Addressing these criticisms requires a commitment to methodological rigor, theoretical development, and transparent reporting. MMR, when implemented effectively with strategies like visual joint displays, can contribute to more robust and insightful EdTech research.

The Role of Theory in Mixed Methods Research

A research methodology needs a philosophical and epistemological underpinning. Cresswell provides a useful introduction to the different worldviews that accompany different research methodologies. Most researchers would see the research question(s) and/or hypotheses as fundamental to selecting the type of research methodology. Questions of frequency may best be explored by quantitative methods, and perception and opinion by qualitative. If the questions deal with both of these, then mixed methods are likely to be preferred.

tags: #mixed #methods #research #educational