FST Graduate Research Topic

Research Topic: Learning Analytics: Observational Study
Suggested By: Vive Kumar
Last Update: September 3rd, 2019

Observational study refers to research that explores cause and effect whereby researchers limit their control of independent variables for ethical and logistical reasons. Nevertheless, many researchers believe that observational studies overestimate treatment effects, which can reduce their validity. In addition, the results of observational studies may contain undetected confounding bias, thus leaving them also subject to debate. The literature has shown that observational studies – using larger and more diverse population samples over longer follow-up periods – can supplement the findings of randomized experiments. This research proposes an observational study built on the matching techniques prescribed by Prof King (Why Propensity Scores Should Not Be Used for Matching), in which increasingly available new sensors can better observe and/or record teaching and learning experiences in real time. It will demonstrate how learning analytics processes can incorporate observational sensors and advance blocked randomized experiments that measure the impact of analytics. Basically, such techniques strive to enable teachers to step into the roles of analytics researchers using interactive analyses. In addition, this research sets forward matching techniques such as Propensity Score Matching, Coarsened Exact Matching, and Mahalanobis Distance Matching along with their corresponding imbalance metrics (i.e., L1 vector norm, Average Mahalanobis Imbalance, and Difference in Means) to measure and control for sources of bias. Propensity Score Matching is the most popular matching technique in observational studies. This research demonstrates its suboptimality by presenting an observational study with both randomized and non-randomized data using R libraries (MatchingFrontier, CEM, and MatchIt) and the web application framework for R called Shiny. We measure the data imbalance accuracy of the proposed observational study design using the three aforementioned matching techniques and compare their measured levels of imbalance against the level of data imbalance found an equivalent randomized experiment.