Work in Progress: Aligning What We Want With What We Seek: Increasing Comprehensive Review in the Graduate Admissions Process

To improve the identification of students likely to complete a graduate degree, we sought to change our process and the way we evaluate students for admission. For this, we changed the application and review process to include an assessment of applicants’ non-cognitive variables based on Sedlacek’s work and the Fisk-Vanderbilt MS-PhD Bridge Program admissions model [1-3]. In 2016, our Materials Science and Engineering (MSE) Department piloted the incorporation of these changes. Consequently, MSE increased the enrollment of women and underrepresented minority students by 19% and 57%, respectively and we will be evaluating the impact of the changes on performance metrics and completion rates over time. Ahead of this application cycle, we have made changes to the applications of more of our graduate programs in the college. Most recently, we were asked to create a plan for the university. As changes continue, we will evaluate outcomes and impact within the college and across the university over time, which we hope will inform best practices for improving completion of graduate degrees and diversity of graduate programs.


Introduction
For decades, graduate programs have evaluated students for admission based in considerable part on the standardized graduate record examination (GRE) and grade point average (GPA). These metrics, however, have limited correlation to success in graduate school. Moreover, based on a 2008 Council of Graduate Schools study, only about 50% of PhD students in science complete their degree in seven years and about 50% of all PhDs regardless of discipline complete in eight years [4]. In fact, fewer people earned PhDs between 2008 and 2016, compared to the preceding eight years [5]. Acceptance to graduate programs is extremely competitive; so competitive that, according to the US News and World Reports 2019 Best Graduate School Rankings in Engineering, the average acceptance rate for the top 20 engineering schools ranged from 9% -35.5% [6]. Similarly, the average GRE score for this same group ranged from 164 -168 points [6]. The maximum possible score is 170 points.
Because science programs tend to fund their PhD students, these data suggest that the ways in which we evaluate students for admissions may waste millions in research and university funding and leave room for improvement. Furthermore, the GRE has been shown to be biased against women and underrepresented minorities [2], which may serve to repress the diversity many graduate programs seek. To improve the current model, we decided to include factors, which are markers of success in graduate school for all students, independent of background. These factors, called non-cognitive variables, can provide programs with additional criteria with which to better assess all students [7].
Prior to 2016, the primary criteria for evaluating graduate applicants in our MSE department were numerical scores, particularly GPA and GRE scores. At the time the student population was 67% domestic and 33% international, the latter predominantly from South and East Asia.
Females represented 20.1% of the student body and underrepresented minorities, as defined by NSF, represented 3.6%.
At this time the Graduate Studies Committee in MSE began to question that approach following a seminar by Dr. Keivan Stassun, the founding Director of the Fisk-Vanderbilt MS to PhD Bridge Program, where he showed data highlighting the disadvantages GRE scores had on graduate student admission and the success associated with their alternative approach. Additionally, Figure 1, adapted from data in the seminal study by Kuncel and Hezlett [8], shows that, while GRE correlates well to first year GPA, its correlation to other graduate academic performance and completion metrics are weak. As a consequence, the value of standardized testing is questioned [9,10]. The correlation of GRE to publication citations, degree completion, and research productivity are reported as 23%, 22%, and 11%, respectively. These numbers can be converted to the respective "odds ratios" (the odds of above-average success for aboveaverage test scorers to the odds of above-average success for below-average scorers) of 2.4, 2.3, and 1.5 [8]. As an example, an odds ratio of 1.5 means that for every 1.5 successful applicants correctly identified by the GRE score, 1 successful applicant is missed. Thus, while the GRE is a valid indicator that students can maintain the minimum required GPA in their coursework (for our university, it's 3.0 [11]), additional criteria are needed to identify the applicants most likely to be successful in their research and complete their degree.

Figure. 1.
Correlation between standardized test scores and student success in graduate school, adapted based on [8,9].
Kuncel and Hezlett [8] report that standardized tests could not be proven to be biased based on race, ethnicity, and gender identity. However, other work questions these results [3], and a study by Miller and Stassun strongly suggests that such biases do exist, leading to their conclusion that standardized tests as a main admission criterion would suppress diversity in graduate school [2].
With this information, The Ohio State University MSE Department followed the Fisk-Vanderbilt comprehensive admission approach of identifying students with an outstanding "growth mind set" rather than outstanding test taking abilities. The concept of growth mindset follows Dweck's definition [12] and distinguishes between the fixed and growth mindsets. According to Dweck's writing, a person with the fully developed fixed mindset tends to have a static intelligence, avoid challenges, gives up when facing obstacles, does not see the point of making an effort, deflects criticism, and feels threatened by the success of others. In contrast, students with a growth mindset tended to embrace challenges, were improved by the obstacles they face, worked hard, learned from criticism, and celebrated the success of others.

Comprehensive Admission Process
Our comprehensive admission process has two steps. First, in order to ensure that students will be successful in their coursework, applicants are pre-selected by GPA and GRE to identify students with a high likelihood of maintaining the minimum required 3.0 GPA. From this group students with a 3.6 GPA or higher (or the international equivalent) are selected. This is because a large number of our university level fellowships require a minimum 3.6 GPA as one of the eligibility criteria [13]. These fellowships are a funding mechanism the university awards admitted applicants on a competitive basis to help programs recruit talented students to our programs. Programs can nominate students for consideration only if they meet the eligibility criteria.
These students are then evaluated for admission by the Graduate Studies Committee, a group of faculty charged with overseeing graduate education for the program, based on the extent to which they exhibit a growth mindset. This determination is made based on the application materials and rubric in Table 1. The rubric was developed by Stassun et al. for the Fisk-Vanderbilt program to identify students with a growth mindset [1,3]. The rubric is not revealed to students. The process starts by evaluating the statement of purpose and diversity statement. The threetiered rubric makes it relatively easy to assess the seven attributes. Careful evaluation of the letters takes some time but they tend to address all these points in some form. Conversely, applicants who did not put in a serious effort, or are squarely in the low-score category are sorted out. This allows us to sort out the lower end quickly, which can be a considerable fraction. In the next step, we perform phone or video interviews, ideally for all remaining candidates. We match the applicants to potential advisors mentioned in their application, or faculty who are interested in specific students. The junior faculty in the department are especially active in this respect.
Lastly, those applicants identified as having a growth mindset based on their applications are interviewed using the rubric. While it may be possible to "fake" a growth mind in a letter with the help of materials on the internet (see e.g. [14][15][16]), the mindset of most applicants can usually be determined after a few minutes of conversation. Ideally, all interviews should follow the same set of questions, however, this is not always done. The suggested applicant interview protocol follows the template from Stassun et al. [3], asking about high points (e.g. "What are you most proud of accomplishing during your college experience?"), low points and obstacles; and research experience in and out of lab and classes. Because the growth mindset can be identified quickly, interviews tend to take less than ten minutes. After the interview round, applicants are ranked for admission.

Results
MSE changed their admission criteria in 2016 and is currently in only its third year using this comprehensive admission process. Nevertheless, the increase in student representation is promising. Between 2016 and 2018 the number of women and underrepresented minorities increased in the department by 19% and 57%, respectively ( Table 2). Since the change, the department has obtained the highest number of university fellowship awards obtained in the past twelve years ( Figure 2). The number of awards has increased for both types of fellowships, the University Fellowships, which target specific academic metrics [13], and Graduate Enrichment Fellowships [17], which target specific academic metrics and students who will enhance the diversity of the university graduate student body. Since fellowship panels usually consist of experienced educators, the higher fraction of students with a proven growth mindset and the department's ability to write a more compelling nomination letter based on that mindset may have contributed to the increased number of fellowships. Moreover, the ranking of the MSE Graduate Program has improved over the past four years moving from No. 20 to 14, which was one of the five highlights of the year in the 2019 Best Materials Engineering Graduate Program rankings by U.S. News (2016-2019) [18]. While this this may not be correlated with the changed admission process during the same timeframe, it is at least a fortuitous and encouraging coincidence. Since the department is only three years into the change, it is still too early to assess the longterm outcomes of the change, specifically regarding completion rates, publication and citation numbers, or quantifiable research productivity. We will also be monitoring student progress to identify whether anyone struggles or whether students enrolled in the PhD program leave with or switch to the MS program. Instances of either situation will be investigated and where appropriate students will be provided with the additional support they need to be successful. Thus, developing valid outcome metrics is the next step in our process.

Conclusion
In summary, we have described here our efforts within the University's College of Engineering to change our process for evaluating students for admission. As a consequence, we have increased diversity and improved the identification of students likely to successfully complete a graduate degree. For the identification of graduate school success, we adopted the view that a growth mindset provides a greater chance to be successful in graduate school than a fixed mindset, as first defined by Dweck [12]. For this, we changed the application and review process to include an assessment of applicants' non-cognitive variables based on Sedlacek's work [7] and the Fisk-Vanderbilt MS-PhD Bridge Program admissions model [1][2][3]. In 2016, our MSE Department piloted the incorporation of these changes in its graduate program. As a result the enrollment of women and underrepresented minority students increased by 19% and 57%, respectively and we will be evaluating the impact on degree completion and research productivity over time.

Future Work
After MSE's piloting efforts, the College of Engineering worked with other engineering programs to adopt a more comprehensive admission philosophy. Over the past year the college committee of Graduate Studies Chairs, led by the Assistant Dean of Graduate Programs, has developed prompts for the statement of purpose and a recommendation writer assessment tool that allows for evaluation of the growth mindset similarly to what was done in MSE. The prompts request information on how students respond to particular situations. The recommendation writer assessment tool asks writer to rate student's demonstration of certain behaviors and characteristics associated with success in graduate school on a Likert scale. As of autumn 2018, the graduate student application for many of our engineering programs has been updated to incorporate these prompts and the assessment tool. The next phase will be to monitor and measure the outcomes of these changes longitudinally with respect to completion rates, publication and citation numbers, and student body composition. The comprehensive review effort has gained traction and the first author has been asked to lead a university taskforce, on which the co-author has agreed to participate, to create a written comprehensive review in graduate admissions plan for the university. Our goal is to create a plan that is adjustable based on the needs and desired outcomes of each program.