Copyright © Academic Support Index. All rights reserved.

The development of the Academic Support Index (ASI)

began with questions we needed to answer:


   •How can we tell if our programs or interventions are actually making a difference for students?

   •How can we figure out which students might struggle academically in advance so we can target them for

     support?

   •How can we target our limited resources to the students most in need?

   •How can we talk about the achievement gap without contributing to stereotype threat?

   •Fundamentally, how can we become more effective with our outcomes and more efficient with our

     resources?

Below is a review of the Academic Support Index presented to the Berkeley Unified School District School Board in 2017


BERKELEY UNIFIED SCHOOL DISTRICT
TO:            Donald Evans, Ed.D, Superintendent
FROM:        Pasquale Scuderi, Associate Superintendent
                     David Stevens, Evaluation and Assessment        
DATE:        September 13, 2017
SUBJECT:      Update on the progress and implementation of the Academic Support Index

BACKGROUND INFORMATION:
All of our students do not enter school on an equal footing. They enter with a combination of "headwinds" and "tailwinds". Tailwinds are the things that make school easier for students: Parents with high education levels, cultural capital, stable homes, good attendance, and past academic success are examples of tailwinds. Headwinds are things that might make success in school more difficult: Having a learning difference, being an English Learner, a history of academic struggles, or being socioeconomically disadvantaged. We have students with a lot of headwinds and others with a lot of tailwinds, and every combination in between.

The Academic Support Index (ASI) is a method for providing each student with a quantitative measure of the headwinds that may impact their academic success. It can also be thought of as a measure of the amount of support that we owe each student.

There is a strong correlation between the ASI and student academic outcomes ranging from Kindergarten screeners, Smarter Balanced Assessment scores, cumulative grade point average, social emotional learning scores, college/career readiness indicators, and post-secondary degree attainment.  These strong relationships have been demonstrated over multiple years and in a variety of school districts. Approximately 250,000 students have been scored using the ASI.

 The ASI has been effectively used for data analysis, program and intervention design and evaluation, educational research, and in the early identification of students who will benefit from additional academic support. Multiple papers have been presented on the ASI at the American Educational Research Association (AERA) and the California Educational Research Association (CERA) annual meetings. In 2014 a paper on the ASI earned the Outstanding Paper Award at CERA.


Theoretical Framework
The ASI is based on the theoretical framework of the ACE study (Adverse Childhood Experience Study) conducted by Fellitti in 1998. In this paper they saw a strong relationship between greater levels of childhood trauma and increasingly negative health outcomes.  A second framework is that of Response to Intervention (RTI). In the RTI model we provide increasing levels of support along with increasing need. A third contributor to the ASI framework is the concept of Educational Debt by Dr. Ladson-Billings. Educational debt, as it relates to the ASI, is a measure of our obligation to support each student. The ASI provides a quantitative measure of this level of debt.

Background
The Academic Support Index was developed to address a number of questions that we were facing at Berkeley High School (BHS) during the 2011-12 school year.  The school had yet to recover from an article in the East Bay Express in the spring of 2009 that had caused deep fractures in the school community. There was a significant amount of internal conflict regarding the comparison of academic outcomes by small learning community (SLC).  There were accusations of grade inflation and debates about which SLC had the most challenging students, which SLCs were being effective, and how the school should allocate resources. There was significant reliance on anecdotal stories about both success and failure rather than a data informed method for evaluating the efficacy of our overall school design, our SLCs, and our interventions.  

The ASI was developed to try to address these needs. We needed to be able to answer the following questions in a manner that was both valid and reliable in order to determine if we were moving in the right direction:

  • How can we tell if our programs or interventions are actually making a difference in student outcomes?
  • Is it possible to predict which students might struggle academically so we can provide them with appropriate support in advance?
  • How can we target our limited resources to the students most in need?
  • How can we talk about the achievement gap without contributing to stereotype threat?​
  • Overall, how can we become more effective with our outcomes and more efficient with our resources?


Critical to being able to answer those questions was the ability to be able to make “apples to apples” comparisons.  None of our learning communities or interventions could be said to have similar compositions of students. On the other hand, all of them had similar types of students, just in different proportions. The ability to see how similar students performed in different programs would be critical to identifying areas of success.

The general practice in education has been to look at academic outcomes by demographic “buckets” such as race/ethnicity, gender, disability or English learner status, etc. This practice is based on three significant fallacies. The first is that it assumes a level of homogeneity that just doesn’t exist.  For example, within the Hispanic/Latino population 44% of the families have college degrees and 34% have a high school diploma or less. Misleading conclusions will be made if you do not control for those differences when analyzing student data.

A second flaw is disproportionality.  Many of the factors that we know interfere with students’ academic performance are not equally distributed across all groups and are highly concentrated among students of color. For example, being socioeconomically disadvantaged (SED) has a significant impact on academic performance, but in Berkeley, only 8% of White students are SED compared to 55% of Black or African American students and 54% of Hispanic/Latino students (2016).  Similarly, students who are still learning English are concentrated in the Hispanic/Latino population and students with disabilities are highly concentrated amongst Black or African American students. Presenting academic data without controlling for these significant contributors to academic difficulties implies a causal relationship between race and outcomes and presents a narrative that is harmful to students and not particularly useful to teachers.

A third flaw in looking at outcomes by buckets is that students do not exist in singular buckets, but in the intersection of many different buckets.  These buckets include disability status, household education level, gender, race/ethnicity, your housing situation, primary language, etc. Some of these buckets can act as tailwinds and strongly correlate to higher academic performance (high parent education level, native English speaker or bilingual upon entering school) and some correlate to greater challenges, or headwinds.

The ASI was designed to address the additive nature of the factors that make academic success more difficult.  For example, a student with a disability who is a member of the cultural majority and whose parents are college graduates is typically not the same as a student with a disability who is a cultural minority whose parents only graduated from high school.  While they may have a disability in common, their access to resources and support at home is very likely significantly different. The ASI doesn’t just identify students who may benefit from additional support, it also tells us the degree of support that he or she may need.


Methodology
In order to better understand the relationship between students’ characteristics and academic outcomes I analyzed the relationship between cumulative grade point averages (GPA) and demographic subcategories.   I established rules for quantifying the headwinds in advance of the analysis. The rules were that if the 95% confidence interval (CI) overlapped with a GPA of 2.50 for a demographic subcategory (ie male vs. female) then that subcategory would earn 2 ASI points. A GPA of 2.50 was chosen because it is the midpoint for Cal State University eligibility. (A 95% Confidence Interval tells us that the true average is somewhere within that range.) If the CI did not overlap with a 2.50 GPA but was significantly below that of other(s) then it would earn 1 ASI point.  A student’s ASI is the sum of all of the headwinds that he or she faces.

In the example below, the subcategories of “High School Graduate” and “Not a High School Graduate” both have confidence intervals that overlap the 2.50 GPA target (lower limits (LCL) below 2.50 GPA and higher confidence intervals (HCL) above 2.50 GPA).  The “Some College” subcategory earns 1 ASI point as it does not cross the 2.50 threshold but is still below that of lower level of “College Graduate”, “Graduate School/Post-Graduate”, and “Decline to State. A table summarizing the analysis of each of the available demographic fields can be found at the end of this document.

Scoring Students on the ASI and Analysis
Once the ASI scoring system was developed, all BUSD students K-12 were scored and the relationship between their ASI and academic outcomes was analyzed.  The scoring itself is very simple and it takes less than 1 hour to score all ten-thousand plus BUSD students. The scoring is done approximately five times a year.

The analysis of the ASI and student outcomes is generally done in three different ways. Initially I calculate the correlation coefficient using linear regression.  This looks at the overall relationship between students’ performance and their ASI. A high correlation coefficient (greater than 0.90) suggests that there is a strong relationship between the ASI and the outcome being analyzed. The table below includes some examples.

Strong correlations between the ASI and outcomes have been found over multiple years and across a variety of schools and districts. A paper on this topic was recently submitted to the American Educational Research Association.  This paper demonstrated the strength of the relationship between the ASI and outcomes over several years of data and across three different districts including BUSD, a small rural district in Vermont, and a large urban district of over 40,000 students.

The second method used for analyzing the ASI looks to see if student performance is differentiated by their ASI scores (see chart below).  Where the confidence intervals (95% CI) represented by the bars on the graph below do not overlap we can say the at the ASI was able to differentiate student outcomes.   This is valuable in that it helps us use a student’s ASI to predict academic performance with a greater level of confidence.

A third method of analysis is to look to see if particular ASI clusters perform similarly in regards to meeting outcomes such as Proficiency on the Smarter Balanced Assessments.  In the table below you can see a very specific pattern where students with an ASI of 0 to 2 tend to meet or exceed proficiency on the SBA ELA whereas students with an ASI of 3 or higher tend to score below proficiency.  On the chart the horizontal bar represents proficiency with ASI 0-2 and ASI 3+ next to it for each grade level. There are a number of benefits to analyzing data through clusters. One is that it simplifies the long-term monitoring of district performance. Another is that given the historical consistency of these performance patterns, schools can identify students for support as early as the first day of school. While not all students with an ASI of three or higher fail to meet standards, way too many do.  By providing additional resources and monitoring early on, we can begin to interrupt this predictability.

Papers and Presentations
The Academic Support Index and its applications have been presented at a number of competitive educational research venues including the American Educational Research Association and the California Educational Research Association.  Titles and brief descriptions of these papers can be found at the end of this section. A paper on the ASI won the “Outstanding Paper Award” in 2014.

A chapter for the upcoming Cambridge Handbook of Applied School Psychology was submitted this past summer and a number of papers are currently being prepared for submission to peer reviewed journals on the ASI.

The ASI has been presented at a variety of other conferences including the California State Community College Academic Academy and Illuminate Users Conferences.  Locally, I have presented to a number of school Site Councils, parents’ organizations, the Berkeley City College Black Student Union, and annually to classes at Berkeley High.

Papers presented at peer reviewed conferences:

  • Building and Utilizing an Academic Support Index, AERA 2015 and CERA 2014
    • An introduction to the ASI and its theoretical framework
  • Identifying Students for Transition Support, CERA, 2015    
    • An evaluation of our protocol for identifying students for support during the transition from middle to high school. The protocol was able to identify 88% of the non-special education students who had 3 or Ds or Fs at the first marking period up to 4 months prior to the first day of school.
  • Boosting Test Performance for At-Risk Students, CERA, 2015
    • An evaluation of how we used the ASI to increase the passing rate for ASI 3+ on the CAHSEE.  While the intervention used ASI to identify students, the benefits accrued to disproportionately to students of color. After multiple years of 1% gains in the passing rate for African American students, we had a 13% gain in one year.  The intervention group had a 98% passing rate. The historical average passing rate for ASI 3+ was 63%.
  • A Comparison of the Local Control Funding Formula and the Academic Support Index in Predicting Academically Underperforming Students, CERA 2016
    • This paper demonstrated that the ASI 3+ was up to 50% better than “Unduplicated” at predicting which students would underperform. The ASI 3+ method identified up to 90% of underperforming students while at the same time having fewer false positives, and most importantly, significantly fewer false negatives. A false negative can result in a student not receiving support. The Unduplicated method missed up to 3 times more students than ASI 3+.


  • Using the STARS Protocol for Identifying Students At-Risk During the Transition to High School, AERA 2017
    • A  multiyear evaluation of the method for identifying struggling students both during the transition from middle to high school as well as from elementary to middle school.  
  • Revisiting the Academic Support Index: A Validation Study Using Data from  Rural, Semi-urban, and Urban School Districts, AREA 2018 
    • An analysis of the ASI over three very different districts demonstrated that the ASI is a valid and reliable tool for analyzing student data both within and across schools and over multiple years of data.


Current and Potential ASI Applications
While the ASI was initially developed for Berkeley High School, we have been scoring all BUSD students since the 2013-14 school year.  Because we calculate a student’s ASI as soon as he or she enters school, we can immediately identify students who may benefit from early monitoring and/or intervention. For example, we know that a kindergartner with an ASI of three or higher is seven times less likely to meet the reading target on TCRWP and twelve times less likely to meet standards on SBA reading at the end of third grade.  Knowing that, we can begin offering a variety of supports (extra time with text for example) as early as the first day of kindergarten. The full potential of this application of the ASI has yet to be exploited.

At the district and school level, the ASI is used to analyze data in a way that allows us to make apples to apples comparisons. Without this ability, analysis by site and grade level is confounded by variance in student populations making it difficult to rigorously identify areas of success.  

All assessments in Illuminate can be be disaggregated by the ASI clusters of 0 to 2 and 3+ to help teachers monitor both overall achievement as well as equity goals.  Last year the middle school math teachers used the historical performance by ASI 3+ students to identify standards where the students had struggled. They then used this information to modify their instruction going forward to improve the overall performance of these students. The results were a significant improvement for ASI 3+ students on all assessments with an increase of 22% of ASI 3+ students achieving mastery on one 8th grade assessment.

Berkeley High School has been the leader in using the ASI.  The BHS Single Plan for Student Achievement monitors student performance both overall and through the equity lens of ASI 3+.   A multi-year focus during professional development on the improvement of writing for ASI 3+ students has payed off with an average of 30% increase in the number of ASI 3+ students writing at mastery by the end of 10th grade over the past four years.  The ASI is also being used to track the enrollment of ASI 3+ students in higher level math classes with gains seen last year over the previous year. The intervention team at BHS has been using the ASI along with a transition rubric to identify students for support through the LCAP funds.  Several papers on this have been presented on conferences and the protocol is being replicated at a number of other schools in southern California. This same protocol is now being used for students transitioning from elementary to middle school.

The ASI has several benefits over other methods for identifying potentially struggling students. One is that, unlike Unduplicated (students who are either English Learners, served by foster care, or receive free or reduced lunch), the ASI is on a continuum and is able to both identify students and provide a measure of the degree of support a student may need. Another benefit of the ASI is the ability to identify students with significantly greater precision resulting in fewer false positives and fewer false negatives making more efficient use of district resources.  Middle and high school staff have been using the ASI to help identify students for support over the past several years.

Summary
There is still significant untapped potential for using the ASI in BUSD in evaluation, data analysis, and intervention design.  Because students within ASI bands perform very similarly year over year it is possible to make greater use of the ASI to predict how students will do in the future. This gives us lead time to interrupt that predictability.  Additionally, the ASI is significantly more precise than Unduplicated in predicting academic performance helping the district to be more efficient and effective with its resources. Because academic headwinds are disproportionately experienced by students of color, the benefits of using the ASI can directly impact our efforts to reduce the achievement gap.

Please feel free to email me with any questions:

davestevens@berkeley.net

DavidStevens@AcademicSupportIndex.com


You can read more about the Academic Support Index at: https://academicsupportindex.blogspot.com