What are Comparison Groups and how does the NYC Department of Education calculate them?
Below is a screenshot from a School Quality Snapshot.
What are Comparison Groups?
The Comparison Group helps answer an important question: How well is the school helping its students grow and improve? Just looking at how high or low a school’s overall results are doesn’t answer that question because it doesn’t consider where the students started or the challenges they face.
That’s where the Comparison Group comes in. By looking at the results of similar students with similar starting points and challenges, it estimates how the students at the school would have done if they had gone to other schools. You could also think of it as a large, hypothetical school made up of similar students from across the city. By comparing a school’s result to its Comparison Group, you can see the school’s impact on its students.
How are Comparison Groups created?
To understand how a Comparison Group is made, let’s meet Anna, who currently attends a public school in New York City.
This is Anna’s school and all her classmates. Anna is currently attending an elementary school in New York City.
Anna has many characteristics that make her a beautiful member of the NYC public school system. Anna is currently in fourth grade, her family is eligible for public assistance, and she received a 2.3 on her ELA state test and a 2.5 on her math state test. The full list of characteristics that the DOE uses to find similar students can be found at the bottom of this page.
Each school in the city serves a different population of students. But many schools also have students with similar characteristics to Anna.
The DOE finds fifty students from other schools throughout the city with the most similar characteristics to Anna. In Anna's case, these students are all fourth graders in families that are eligible for public assistance (or in temporary housing), and received an ELA score near 2.3, and a Math score near 2.5 in third grade. (Note that Anna’s peer group can include students of different races and genders because we do not restrict the peer group to students of the same race or gender as the main student.)
Using the same method, the NYC DOE finds peer groups for each student in Anna’s school.
The peer groups for all the students in Anna's school are combined to form the Comparison Group for Anna’s school. This Comparison Group is composed of students from all around the city that are not in Anna's school.
If Anna's school has a hundred students, then the Comparison Group will include five thousand students (100 x 50 = 5000) that are similar to the students at Anna’s school.
For metrics that cover a subset of the main school’s students, the Comparison Group result is based on the peers to those students. For example, graduation rate only covers students in a high school’s graduating class. The Comparison Group result for this metric is based on the peers to the students in the graduating class.
As another example, the graduation rate for Asian students only covers Asian students in a high school’s graduating class. The Comparison Group result for this metric is based on the peers to the Asian students in the graduating class. As described above, the peers of Asian students can include students of different races. The peers of the Asian students will have similar prior test scores, disability status, and economic need—but are not necessarily of the same race (or gender).
This is how the NYC Department of Education creates a Comparison Group for every school. By taking into account the student population served by the school and finding similar students, the Comparison Group helps to show the school's effectiveness at helping its students improve. The Comparison Group is displayed for many of the metrics in the School Quality Snapshot and other reports and should help the user understand the impact of the school on its students.