Understanding What Matters in the School Choice Process

Cheerful Mixed Ethnicity Asian teenage girl sitting with saxophone while looking away. Happy female student is learning woodwind instrument in classroom. She is with friends at high school.

Problem

A key question defies simple answers: What do stakeholders value in their schools?

Understanding what students, parents, and teachers value in education and schooling has been a long-standing question for educators and policymakers. Answering this question would inform understanding of how parents decide on what school to send their children to, how teachers experience their work environments, and what matters to students in their school. There is a large literature on the sources of information parents use to make schooling choices and young adults use when choosing careers such as teaching. In both cases, people tend to rely on their social networks, especially family and friends. However, when choosing schools, families also visit websites of individual schools and sites created by state departments of education and nonprofit organizations that allow comparisons of schools using standardized metrics.

A relatively recent development greatly extends the information available to all families and, in effect, creates a vast public network that goes beyond individual social networks. Product websites now regularly allow consumers to rate and review products. The most widely known user review platform for schools is Great Schools. When typing a school name into any major search engine, the Great Schools webpage for that school is often one of the first links listed.

The Great Schools site integrates standardized data on test scores, student demographics, and other information from federal and other data sources with more nuanced and subjective information from users. Almost every school in the country has at least one review on the platform.

It is important to understand the content of these reviews as more students and families use them to gauge the quality and characteristics of schools.

Solution

NORC is analyzing five million words of text from the Great Schools review website.

NORC at the University of Chicago has partnered with the Institute of Education Sciences-funded National Center for Research on Education Access and Choice (REACH) to analyze all text reviews from the Great Schools website. We apply machine learning, specifically Natural Language Processing, to the 600,000 Great Schools reviews of more than 84,000 public, charter, and private schools in the country. Encompassing more than five million words of text, our initial analysis pre-specified eight broad topics and 27 sub-topics.

While Great Schools user reviews are self-selected and more challenging to analyze, they offer an unusual quantity of data that would be difficult to collect through any other means. For example, it is difficult to imagine the survey or interview design that would yield five million words of text about actual school experiences. NLP allows us to quickly and accurately analyze a massive corpus of text, enabling us to compare schools along a wide variety of dimensions. We combine the Great Schools data with the National Longitudinal School Database (NLSD), which integrates the federal Common Core of Data (CCD) with Census data at the school Census track, allowing us to test how the reviews vary according to school demographics, school sector (traditional public, charter, and private schools), and user type (families and educators).

Result

Findings from this enormous text analysis will shed light on factors in school selection.

This project will delineate who is most likely to post user reviews of schools. Are these representative of all students, parents, and educators? While the number of reviews is very large, the share of people who provide user reviews is very small, i.e., the equivalent of the “response rate” is low and likely self-selected. One purpose of this study is to compare the distribution of star ratings to other random samples as a check on whether people with more positive or negative views are more likely to post reviews.

Second, this project will address which aspects of schooling are most commonly raised in the Great Schools reviews of schools. Unlike surveys, user review data are unstructured, and reviewers can write whatever they wish, making it difficult to compare one review to another. We address the first problem using NLP, which allows analysis of individual words and word patterns. Our initial analyses create some structure by pre-specifying broad topics and sub-topics that we expect people to write about (each with its own associated keywords). Others have used this technique to analyze text data of other kinds of user reviews.

Lastly, this project will answer the broader question: What do user reviews tell us about what families want in their schools?