Before you can understand where you are going, you have to know where you are.  Knowledge of past and present performance is critical for understanding what your system needs in order to move towards its aspiration.  Specifically, you will need to identify the most significant barriers your system will face with respect to achieving its aspiration.  To attain this information, your system will need to:

• Determine the data most indicative of performance
• Organize available data for analysis
• Analyze the data to identify performance patterns

Once systems collect this information, they are better able to pinpoint where change is needed, and are then prepared to begin the planning process.

Context and Challenge

As the Deputy Commissioner of the Massachusetts Department of Elementary and Secondary Education (MA DESE), Dr. Karla Baehr was determined to build a culture of data-driven decision-making throughout the school districts of Massachusetts.  However, like many states, Massachusetts had too many disparate data systems and it therefore took considerable time and effort to obtain a comprehensive picture of a district or school.

With this fact in mind, the MA DESE created the District Analysis and Review Tool (DART), a system that enables policymakers, parents, and the public to assess the effectiveness and improvement of all public schools and districts; hold school leaders accountable for that performance and improvement; and identify where intervention is needed.

While many stakeholders needed a quick and easy resource that provided clear, concise, and thoughtful district data, there were several challenges in implementing the system, including:

• Determining what data points would convey the “health” of a school/district
• Presenting the data effectively

Key Actions

Formulating a List of Key Indicators

To get an overall sense of the question “What do we care about?” Dr. Baehr began using her stakeholder meetings (for example, with Urban Superintendent Network, partner organizations, etc.) to determine what information beyond No Child Left Behind (NCLB) data was vital to the “health” of a school district.  This effort was expanded to stakeholder groups from around the Commonwealth who identified a range of high priority indicators.  The list was narrowed to 30 data elements, which cover a broad range of district and school information, including demographics, assessment data, post-secondary data, educator data, and financial data.  The data indicators, which are linked to a broader strategic framework defining the characteristics of effective districts, demonstrate trends in district performance and feed into public reporting, district self-improvement, and the district accountability system.[1]

Presenting the Data

It is well documented that poor data presentation can lead to erroneous conclusions and obscure the facts of what the data is intended to reveal.  To address this issue, MA DESE turned to the work of two experts in the presentation of informational graphics, Edward Tufte and Stephen Few.  Adapting Tufte and Few’s techniques, MA DESE was able to present district comparisons and multiple years of data in a simple, easy-to-read format (see Exhibit 1 below).

Each district or school receives a list of comparable districts or schools based on student enrollment and demographics.  Additionally, any district or school can be selected for comparison.  Also, on each chart, the selected district is always represented by an orange line, the Commonwealth by a gray line and your comparable district by a dark charcoal line.  Reports are formatted in 8.5” x 11” for easy printing to a printer or to Adobe .pdf format.

Results

In March, 2010 DART debuted with great reviews.  During the rollout of the DART professional development, the implementation team consistently heard from school, district and community personnel alike, that they would begin using the tool immediately.  The tool has assisted schools and districts in identifying performance patterns in student achievement, which informs their strategic planning.  They have also begun to conduct performance analysis to understand why the patterns exist.

Additionally, the schools and districts have provided suggestions for enhancement to the tool, and in October 2011 the new release of the tool will allow districts to make comparisons using additional indicators.  DART is not only becoming an analytical tool; it has allowed MA DESE to build a better rapport with the districts.

The DART has also received national attention, as several state education agencies have inquired about creating a similar tool.

For More Information

Deliverology 101, Section 2A

DART website: http://www.doe.mass.edu/sda/dart/