Conference Presentation Abstracts
American Educational Research Association
Division D (Measurement and Research Methodologies): 2021 (Pending review)
Paper Title: An Examination of Intragroup Variation Using the Academic Support Index
Abstract: The disaggregation of data has been critical to the identification and monitoring of progress on achievement gaps. This study evaluated the capacity of the Academic Support Index (ASI) to differentiate performance within groups identified by these gaps. In all seven models the ASI demarcated student scores along the index (p < .001) and reliably identified sub-populations of high-performing students. By using the ASI as a second dimension to within-group analysis, achievement gaps are reframed to include context regarding the influence of multiple demographic contributors to student performance. The capacity to better understand and predict performance both across and within groups has the potential to reframe how researchers identify, describe, and address the various gaps.
Division D (Measurement and Research Methodologies): 2020
Paper Title: Evidence for the Validity and Reliability of Performance Clusters within the Academic Support Index
Abstract: Reliable predictors of student performance are critical to efficiently directing educational resources. This study provides evidence for the validity and reliability of two specific performance clusters within the Academic Support Index (ASI): one cluster where students are highly likely to meet or exceed standards and one cluster where students tend to fail to meet standards. Students identified within the latter group are those most likely to benefit from early support and intervention. In the analysis of four years of Smarter Balanced Assessments for two school districts across seven different grade levels I found a large average effect size between the clusters (Mean d=1.22). The ASI cluster of lower performing students identified up to 89% of those students who failed to meet standards. Because students’ ASI can be calculated as early as their first day of school, using ASI clusters to identify students for higher levels of academic monitoring and/or additional supports can be an effective way to interrupt the predictability of student outcomes and help close achievement gaps.
Division H (Research, Evaluation, and Assessment in Schools): 2019
Paper Title: Maximizing Assessment Performance of At-Risk Students Using the Academic Support Index to Engineer a Low Stress Testing Environment
Abstract: The chronic underperformance on standardized assessments of students identified as at-risk is foundational to racial and socioeconomic achievement gaps (Reardon, 2011). Testing students in academically heterogenous groups has the potential to raise testing anxiety for mid to low- performing students and negatively impact student performance (Cassady, 2002). Our study attempted to mitigate the impact of negative stereotypes students may have about themselves based on their academic status relative to their higher-achieving peers. We used the Academic Support Index (Stevens, 2015) to create academically homogeneous groups to engineer testing environments where concerns about comparisons should be lessened. We used a randomized controlled design to assign students to either the treatment or control groups. We confirmed homogeneity across groups for both historical academic performance (prior Smarter Balanced Assessment English Language Arts scores, 10th grade local assessment writing scores) and two psychosocial constructs (Academic Self-Perception and Motivation). The rate of students performing at grade level was higher for students randomly assigned to the treatment group (64%, n = 28) vs. the control group (28%, n = 32). Results were statistically significant (p = 0.004) and the effect size was substantial (Cohen’s d = 0.74). Post-assessment surveys provided further insight into how students experienced the testing environments. This study validated results from two prior experiments conducted in 2014 and 2015 (Stevens, 2015).
Division H (Research, Evaluation, and Assessment in Schools): 2018
Demonstration Session: Building Your Own Academic Support Index for Research, Evaluation, and Intervention Design
Session Description: Disaggregating data by demographic categories such as gender, race, and class ignores the fact that students exist in multiple categories simultaneously and that these categories are inherently interactive. The Academic Support Index (ASI) addresses this by accounting for the additive impact of students’ characteristics. The ASI is a tool based on the statistical relationship between demographic fields and student academic performance. The ASI has strong correlation to outcomes including Smarter Balanced Assessments, grade point averages, and post-secondary degree attainment. This session will include an introduction to the background, development, and effective applications of the ASI as well as a practicum for researchers and educators to calculate the ASI of their students.
Division H (Research, Evaluation, and Assessment in Schools): 2018
Paper Title: Revisiting the Academic Support Index: A Validation Study Using Data from Rural, Semi-Urban, and Urban School Districts
Abstract: Previous studies have shown that the Academic Support Index (ASI) has strong correlations to academic outcomes and can be a valuable tool in educational research and practice. In this validation study, the ASI was evaluated against standardized test performance and grade point average in three school districts: rural, semi-urban (original district of study), and urban. The results validated the earlier findings that the ASI is a strong predictor of academic performance. The study also replicated the original ASI point assignment protocol creating local versions of the ASI and evaluated these against the same outcomes. Correlations for the locally developed ASI were not as strong as with the original ASI.
Division H (Measurement and Research Methodologies): 2017
Paper Title: Using the Screening Tool for At-Risk Students Protocol for Identifying Students at Risk During the Transition to High School
Abstract: There is a need in educational practice to reliably identify students who will struggle during the transition to high school. Students who do not transition smoothly experience long-lasting impacts on graduation progress and post-secondary options. Identifying students who will require additional support, both academic as well as socioemotional, is key for early intervention. The goal of this study was to develop a statistically valid tool that would identify these students while still in their eighth grade. The Screening Tool for At-Risk Students (STARS) protocol reliably identified and differentiated at-risk students by grade point average, credits earned, attendance rates, and discipline. The protocol also facilitated the transmission of specific actionable information to the receiving school.
State and Regional Educational Research Associations: Distinguished Paper: 2015
Paper Title: Building and Utilizing an Academic Support Index to Identify and Support Students At-Risk for Academic Underachievement
Paper Abstract: With greater access to student data, there is an opportunity for educators to develop more effective practices for identifying and supporting students at-risk for academic under- performance. When attempting to address and discuss gaps in student performance, traditional disaggregation categories such as race, ethnicity, and gender contribute unintentionally to stereotype threat and support a narrative that negatively impacts students. Additionally, waiting for summative student performance results in the secondary school setting can delay intervention to the point where students’ post- secondary options can be severely impacted. There is a significant need to be able to identify in advance students who may need academic and other available supports to maximize student potential. Through an Academic Support Index (ASI) using a variety of widely available demographic and other data points, Berkeley High School has been able to score each student and reliably identify students at-risk for academic underperformance, particularly those transitioning from middle to high school, and prioritize them for appropriate interventions. Additionally, the ASI provides context for classroom, program, and intervention evaluation, assessment data, and promotes more precise data disaggregation allowing for apples to apples comparisons across programs. The ASI has shown strong statistical correlations when compared against a variety of metrics including California High School Exit Exam passing rates (math R2 =0.90 and ELA R2 =0.92), student grade point averages (semester R2 =0.83) and University of California A-G eligibility rates (R2 = 0.97).
California Educational Research Association