SMERC Seminar: Challenge of Estimating Graduation Rates for Small Cohorts

SMERC Seminar:
The Challenge of Estimating Graduation Rates for Small Cohorts

Dr. Adan Vela
Assistant Professor
University of Central Florida

Register Here for Zoom Link

A look at using regularly-updated multi-level absorbing Markov chains (RUML-AMCs) to eliminate estimation bias and minimize variance in estimates of university graduate rates of small cohorts.

Abstract:
American universities use a procedure based on a six-year graduation rate to calculate statistics regarding their students' final educational outcomes (graduate or not graduate). As an alternative to the six-year graduation rate method, many studies have suggested the use of Absorbing Markov chains to estimate graduation rates. In both cases, a frequentist approach is used. For the standard six-year graduation rate method, the frequentist approach corresponds to counting the number of students who finished their program within six years and dividing by the number of students who entered that year. In the case of Absorbing Markov chains, the frequentist approach is used to compute the underlying transition matrix, which is then used to estimate the graduation rate. In our research presented here we apply a sensitivity analysis to compare the performance of the standard six-year graduation rate method with Absorbing Markov chains.  Through the analysis we highlight significant limitations with regards to the estimation accuracy of both approaches when applied to small sample sizes or cohorts at a university. Additionally, we note that the Absorbing Markov chain method introduces a significant bias, which can lead to under-reporting of the true graduation rate.  To overcome both these challenges, we propose and evaluate the use of a regularly updating multi-level absorbing Markov chain (RUML-AMC) in which the underlying transition matrix is updated year-to-year as new student data become available.  We empirically demonstrate that the proposed RUML-AMC approach nearly eliminates estimation bias while reducing the variance of estimates by more than 40%, especially for cohorts with small sample sizes.

Speaker Bio:
Dr. Adan Vela serves as an Assistant Professor Dept. of Industrial Engineering and Management Systems. Dr. Vela received his B.S. from UC Berkeley, M.S. from Stanford University, and Ph.D. from the Georgia Institute of Technology, all in Mechanical Engineering. Prior to joining UCF Dr. Vela served as Technical Staff at MIT Lincoln Laboratory. Dr. Vela’s primary research domain is in air transportation systems, however his work more broadly focuses on the application of machine learning, modeling, simulation, and optimization to understand and improve decision-making in human-in-the-loop systems. Fun fact, Dr. Vela comes from a rich family history of working in STEM education, his father, Charles Vela, coined and helped popularize the acronym STEM in the early 90's through his non-profit organization CAHSEE (which had a summer program called the STEM Institute) and his work on Congressional and NSF panels.