Research In Progress
Kim, B. H. What’s in a Letter? Using Natural Language Processing to Investigate the Prevalence of Linguistic Biases in Teacher Letters of Recommendation for Postsecondary Applications. Analytic plan available upon request. Work generously supported by the National Academy of Education and Spencer Foundation Dissertation Fellowship.
While scholars have already uncovered many ways that low-income, first-generation-to-college, and racial/ethnic minority students are systematically disadvantaged across the postsecondary application portfolio – from standardized tests to advanced course-taking opportunities – we know almost nothing about whether teacher letters of recommendation advance or impede these students’ college aspirations. This blind spot is especially concerning given mounting evidence that recommendation letters in other contexts can contain biased language, that teachers can form biased perceptions of their students’ abilities, and that narrative application components more generally may contribute to racial discrimination in selective college admissions. Meanwhile, institutions continue to move away from standardized test scores – positioning recommendations to be even more prominent going forward. In this study, I conduct the first system-wide, large-scale text analysis of teacher recommendation letters in postsecondary applications. With application and recommendation data from 2 million students, 500,000 teachers, and 400 postsecondary institutions, I examine the prevalence of “linguistic biases” within these letters: whether students are described by teachers in systematically different ways across racial/ethnic, gender, and socioeconomic groups. By combining rigorous econometric frameworks with sophisticated natural language processing (NLP) and text mining techniques, I analyze variation in letter characteristics at unprecedented scale and fidelity while accounting for salient confounding factors like student academic and extracurricular qualifications. It is paramount that we better understand the role of these letters in ameliorating or exacerbating inequity, and these analyses provide urgent insights for college admissions practices, affirmative action litigation, and text-as-data methodologies for education research.
Kim, B. H., Castleman, B. L., & Song, Y. New Strategies to Support Career Entry for Community College Graduates: Augmenting Intensive Career Advising Services with a Novel Job Recommendation Algorithm and Machine Learning. Analytic plan available upon request. Work generously supported by the Ascendium Education Group.
Individuals with college degrees experience better average labor market outcomes than non-degree holders, especially in times of economic downturn; the labor market premia associated with a college degree is particularly stark in the midst of the economic fallout from the COVID-19 health crisis. But that said, research continues to show large and meaningful differences in the mid-career earnings of college students from higher- and lower-income families (e.g., Chetty et al., 2017). Such disparities in economic well-being among students completing college and even graduating from the same college with the same GPA raise a fundamental question: Are investments to increase degree attainment among lower-income students sufficient to narrow longer-run economic inequality, or are investments to ameliorate the barriers that graduates encounter after college also necessary to ensure positive labor market outcomes and upward economic mobility? Much like how policy and practice coalesced around the importance of supporting historically underserved students through the complex decision-making process of college applications, we anticipate similar interventions will be necessary to support these same students through the complex job market for college graduates. In this project, we develop a large-scale, intrusive career advising intervention for community college students: one that leverages rich longitudinal workforce data and machine learning/predictive analytics to guide students towards available jobs relevant to their degree and with a track record of high-paying wages for similarly qualified graduates. By pairing these job recommendations with intensive career advising, we also provide students explicit support in navigating the nuances and informal expectations of the college graduate job market. At this time, we are currently assessing viability of this algorithmic approach, in addition to important examinations of potential for algorithmic bias within such a recommendation system.
Rodriguez-Segura, D., & Kim, B. H. (2021). The Last Mile in School Access: Mapping Education Deserts in Developing Countries. Development Engineering, 100064. Open-source codebase available via GitHub. Open-source article available at https://doi.org/10.1016/j.deveng.2021.100064
With recent advances in high-resolution satellite imagery and machine vision algorithms, fine-grain geospatial data on population are now widely available: kilometer-by-kilometer, worldwide. In this paper, we showcase how researchers and policymakers in developing countries can leverage these novel data to precisely identify “education deserts” – localized areas where families lack physical access to education – at unprecedented scale, detail, and cost-effectiveness. We demonstrate how these analyses could valuably inform educational access initiatives like school construction and transportation investments, and outline a variety of analytic extensions to gain deeper insight into the state of school access across a given country. We conduct a proof-of-concept analysis in the context of Guatemala, which has historically struggled with educational access, as a demonstration of the utility, viability, and flexibility of our proposed approach. We find that the vast majority of Guatemalan population lives within 3 km of a public primary school, indicating a generally low incidence of distance as a barrier to education in that context. However, we still identify concentrated pockets of population for whom the distance to school remains prohibitive, revealing important geographic variation within the strong countrywide average. Finally, we show how even a small number of optimally-placed schools in these areas, using a simple algorithm we develop, could substantially reduce the incidence of “education deserts” in this context. We make our entire codebase available to the public – fully free, open-source, heavily documented, and designed for broad use – allowing analysts across contexts to easily replicate our proposed analyses for other countries, educational levels, and public goods more generally.
Manuscripts Under Review
Kim, B. H., Bird, K. A., & Castleman, B. L. (revise and resubmit at Education Finance and Policy). Crossing the Finish Line but Losing the Race? Socioeconomic Inequalities in the Labor Market Trajectories of Community College Graduates. Working paper available upon request.
Researchers and policymakers have made important strides in identifying and reducing the impact of socioeconomic barriers to student success in the margins of college application, attendance, persistence, and completion. Yet we currently know little about the extent to which challenges associated with the job search and early labor market transitions differentially affect lower socioeconomic status (SES) college graduates. Past work suggests that lower-SES students experience lower earnings after graduation on average, but this work does not distinguish whether these disparities are the result of well-established factors like differential sorting to colleges and programs or due to barriers lower-SES graduates encounter during the college-to-career transition. We compare the post-graduation earnings and employment outcomes of higher- and lower-SES community college graduates within the same college, program, and graduation year, while controlling for a wide range of additional covariates related to graduates’ demographic, education, and employment backgrounds. Even after controlling for these measures, lower-SES graduates earn about $500 (5%) less per quarter than their higher-SES peers one year after graduation. When we examine this disparity by program, the difference is negligible among Nursing graduates, but even larger for non-Nursing graduates at $626 (7%) per quarter. Among non-Nursing graduates, the disparity shows no signs of closing seven years from graduation. These results highlight the importance of bringing greater attention to the college-to-career transition and the potential value in developing and testing strategies to support lower-SES students in the post-graduation job search and application process.
Kim, B. H., Meyer, K., & Choe, A. (2021). Using Natural Language Processing to Investigate Treatment Variation in Education: Evidence from a Large-Scale College Advising Field Experiment. Working paper available upon request.
Interactive, text message-based advising programs have become an increasingly common strategy to support college access and success for underrepresented student populations. Because text conversations between students and advisors are flexible and responsive to student input, students engaged in advising interventions of this kind may experience different treatments from one another. Given the unstructured, textual nature of these interactions, it has historically been difficult to characterize this variation. In this paper, we revisit data from a large-scale text advising experiment designed to improve college completion and measure treatment variation using automated text analysis techniques. We examine text interactions between a student and their advisor using natural language processing to quantify variation in the intensity (e.g., number and length of student replies), tone (e.g., positivity/negativity), and topics (e.g., financial aid) of messages. Our results reveal substantial variation in sentiment and topics discussed among student- or advisor-initiated messages (e.g., non-scheduled), but little variation in the scheduled messages. These findings highlight the potential for treatment variability to increase as advising models encourage greater personalization or advisor agency, and demonstrate the importance of measuring such treatment variation to better understand program implementation across sites and students.
Castleman, B. L., Bird, K. A., & Kim, B. H. (2019). Pathways to Success: Analyzing Program-Level Heterogeneity in Labor Market Outcomes for a State Community College System. Working paper available upon request.
Despite a significant body of evidence demonstrating program-level heterogeneity in the wage returns to a community college degree, we currently know little about the extent of program-level heterogeneity in non-wage labor market outcomes for community college graduates. We build on an existing literature by investigating the degree of institution- and program-level heterogeneity across several measures of employability, employment stability, and earnings for graduates of a large state community college system. We further examine whether the relative performance of colleges and programs are sensitive to the specific labor market metric we consider. Our descriptive results indicate substantial changes in the rank ordering of institutions or programs based on the labor market metric we employ. These findings demonstrate the importance of considering–and potentially increasing public sharing of– a broader range of labor market outcomes when assessing community college institutions or program returns.
Technical Reports & Policy Briefs
Kim, B. H. (2021). Supporting Students at Any Cost? Examining the Dynamics of Teacher Out-of-Pocket Spending, Student Demographics, and Teacher Autonomy. Research report available here, replication code and pre-analysis plan available via GitHub.
Nearly every public school teacher in the country regularly spends their own personal funds to purchase classroom supplies, with amounts ranging from tens of dollars to well over a thousand each year. Past descriptive work on the subject suggests that teachers are often attempting to support students in ways their pre-existing school budgets either can’t or won’t, indicating that higher teacher out-of-pocket spending may be a useful proxy to understand the degree of student need otherwise going unmet in our classrooms. In this report, I explore this link further by examining the relationship between teacher out-of-pocket spending, student race/ethnicity, and self-reported teacher autonomy over classroom instruction and materials, with data from the NCES Schools and Staffing Survey. I find that as the share of racial/ethnic minority students in a school increases, teacher spending also increases, and this relationship holds when accounting for factors like school urbanicity, teacher experience, and interactions with the share of students qualifying for free and reduced-price lunch. For example, teachers in schools with 75-100% racial/ethnic minority students spend about $130 more per year than peer teachers in schools with 0-24% racial/ethnic minority students – an approximately 31% difference. Indeed, the results offer suggestive evidence that the link between student race/ethnicity and teacher spending is more influential than the well-studied link between student poverty and teacher spending. I also show that higher levels of teacher autonomy are negatively associated with teacher spending at a comparable magnitude, independent of student demographics – in other words, that higher levels of teacher autonomy over classroom supplies predicts substantially lower teacher spending. Altogether, these results offer additional evidence that teacher spending may represent a useful proxy for unmet student need, and that teachers in schools with greater shares of racial/ethnic minority students and lower autonomy may struggle the most to deliver the high-quality instruction they strive towards.
Kim, B. H. (2020). Assessing the Role of Class Size Restrictions in Mitigating Community College Student COVID-19 Exposure through Student Network Analysis. Internal research report unavailable to public; please reach out for more information.
At the outset of the COVID-19 pandemic, higher education institution leaders were faced with difficult decisions about whether and how to operate their academic programs safely given incomplete information on the dangers, infectiousness, and transmission vectors of coronavirus at the time. As leaders look ahead to future semesters, class size restrictions are an increasingly attractive policy option for institutions seeking to maintain some level of in-person instruction. Building on work by Cornwell and Weeden (2020), I use network analysis with students’ course-taking patterns to assess the extent to which varying in-person class size restrictions meaningfully reduces the connectedness of students’ in-person interaction networks for a large state community college network at the campus-by-campus level. I moreover investigate the extent to which varying class size restrictions impacts students’ ability to attend any in-person classes at all, with specific interest in how students of various demographics are differentially forced entirely online by such class size restriction regimes. My main conclusion is that class size limitations would need to be far more aggressive in this context — allowing in-person meetings only for classes of approximately 25 students or fewer — than most of the popular policies being considered to meaningfully reduce students’ exposure to one another through coursework. I also find no evidence of concerning patterns in students being forced entirely online along several salient lines of equity: race/ethnicity, sex, and first-generation status. In other words, class size limitations seem to impact students’ course-taking modality similarly regardless of their demographics on these dimensions.
Kim, B. H., & Castleman, B. L. (2020). Can Predictive Analytics Improve the Efficiency of High-Cost Interventions? Evidence from an Intensive College Advising Program. Internal research report unavailable to public; please reach out for more information.
Education leaders seeking to improve equity in their institutions are often caught in an intractable bind: evidence-based interventions that successfully support improved outcomes among historically underserved students are often logistically intensive to implement and extremely expensive on a per-pupil basis, and budgets often don’t permit broad access to these programs as a result. While “nudge” style interventions were slated to help leaders at least partially resolve this cost-effectiveness quandary, recent evidence has found that maintaining the efficacy of these interventions is difficult as programs scale. Advances in predictive analytics and machine learning techniques now offer another possible solution: implement these costly interventions, but target them only to those students who stand to most benefit from them. In this report, we evaluate the effectiveness of one such intervention at a large public university system that provided more intensive college advising resources to students predicted to be at higher risk of drop out by a machine learning algorithm. Because students were assigned a continuous risk score and then categorized into discrete risk groups (e.g., highest risk, high risk, etc.) based on strict score thresholds, we employ a regression discontinuity design to evaluate the effectiveness of these additional intensive advising supports at each risk group threshold. Our results reveal that the added intervention supports did not improve student completion, credit accumulation, or grade point accumulation, at any of the investigated thresholds, but we often cannot rule out the presence of meaningful effect sizes due to low precision. Examinations of program take-up measures around each threshold (e.g., number of advising meetings attended) reveals that these null effects are likely driven by the fact that students above each cut off tended not to utilize the additional services available to them.
Kim, B. H. & Castleman, B. L. (2019). Can Earnings Outcomes Drive Student Enrollment to High-Earnings Community College Programs? The Impact of Integrating Earnings Data into a Popular Search Engine Platform. Internal research report unavailable to public; please reach out for more information.
Enrollment and re-enrollment into postsecondary education can be a highly complex and difficult process for prospective students despite the consistently high economic returns to postsecondary degrees. Even with broad federal efforts to simplify the relevant information (e.g., the U.S. Department of Education’s College Scorecard) and provide additional college advising supports to historically underserved student populations, enrollment among adults without postsecondary degrees remains low. Through a novel partnership with a large state-wide community college network and a popular search engine platform, we develop a data tool that presents search engine users with the mid-career earnings of graduates from high earnings programs at their local community college, alongside relevant links for more information on enrollment, when users search for related terms. Using a differences-in-differences design, we compare the applications and enrollment for “target” programs at participating institutions against similar programs at non-participating institutions to examine whether the roll-out of this data tool impacted public interest in these programs. In brief, we find that the data tool did not produce detectable effects on either application volume nor enrollment rates, driven largely by a relatively diffuse treatment timing window and low general precision given the noisiness and idiosyncratic nature of application and enrollment rates over time.
Kim, B. H. (2019). Pathways to Success: Improving the Transparency of Student Outcomes in a Large State Community College System. Internal research report unavailable to public; please reach out for more information.
One of the most consequential decisions for a community college student is their choice of major: the best causal evidence we have available shows that, depending on the major, an associate’s degree can increase the yearly earnings of graduates by as much as 103% and as little as 0%. Unfortunately, research also demonstrates that students severely misjudge the earnings associated with different majors, and detailed post-graduation employment outcomes by major tend to be difficult to obtain or otherwise inaccessible to prospective students – these factors combined making it difficult for students to make informed decisions about the critical question of their major. In this analysis, I present four policy alternatives that a large state community college partner could implement to address this lack of outcomes transparency within its network and improve the extent to which students are able to incorporate these valuable data into their enrollment decisions. I go on to estimate the possible repercussions of each of these alternatives in terms of their implementation costs, their utility to students, and their likely influence on student major decisions. I conclude by offering a specific recommendation to expand college advising services with these data, and offer explicit recommendations on how this recommendation could be implemented.
Kim, B. H., & Castleman, B. L. (2018). Exploring Heterogeneous Treatment Effects with Causal Forests: Evidence from a Large-Scale College Advising Nudge Experiment. Internal research report unavailable to public; please reach out for more information.
In the context of policy research, policymakers are often interested not just in whether a policy works, but for whom it works, more specifically. Moreover, commonly reported average treatment effects can often mask meaningful differences by individual demographics that fundamentally change the value proposition of an intervention measured to be either effective of ineffective on average. Even so, examining these heterogeneous treatment effects can be complicated without strong theoretical priors — especially if interaction effects are likely — given the need to explore many possibilities and thus the potential for finding spurious relationships just by statistical chance. Causal forests (Wager & Athey, 2018) present an attractive potential solution to this problem by leveraging machine learning methods to estimate these heterogeneous treatment effects in an iterative, but principled manner. In this report, I revisit the results of a large-scale college advising nudge experiment that previously found precise null average effects across multiple years of data and tens of thousands of subjects. Using an implementation of causal forests, I explore the extent to which these null effects potentially mask heterogeneous treatment effects across a rich array of student demographic data. In sum, I find that the average null effects were estimated to be broadly applicable across student subgroups, and that heterogeneous treatment effects are unlikely to play a factor in this context.
Kim, B. H., Castleman, B. L., & Song, Y. (2018). Do Coaching Styles Matter for Principal Improvement? An Application of Natural Language Processing Methods to a Principal Improvement Coaching Intervention. Internal research report unavailable to public; please reach out for more information.
K-12 principals are positioned as critical change-making agents within schools, and the policy research realm is only just beginning to explore their impact on student outcomes, teacher outcomes, and broader community outcomes. As the wave of high-profile teacher quality improvement policies continues in earnest, policymakers are increasingly looking for effective strategies to similarly improve the quality of their principals. Yet to date, we know relatively little about what interventions — if any — can support the development of principals. As part of an impact evaluation for a multi-site principal coaching program, we examine the extent to which specific coaching session content varies meaningfully with eventual principal improvement on a series of principal quality measures. We leverage a natural language processing technique known as word vector cluster analysis to analyze the detailed session notes written by coaches and measure how often coaches focused on varying skills, topics, and leadership frameworks. Our analysis reveals relatively little variation in coaching session content overall across coach-principal pairings, and similarly a lack of relationship between these measures and eventual principal quality outcomes. We conclude by offering recommendations for future applications of these automated content analysis techniques and coaching interventions of this style.