Evaluation & Assessment

With funds from the U.S. Department of Education, the National Science Foundation, and the State of Michigan’s Office of Equity, UROP has been engaged in a longitudinal assessment of the impact of the program on student retention, academic performance, engagement, and pursuit of graduate and professional education. UROP employs a multi-method approach to assessment and evaluation. Our early research efforts were quantitative in focus – surveys research and retention studies. We added qualitative research to our repertoire to increase our ability to detect benefits to UROP participation not identified by our more traditional quantitative approaches. The following is a summary of some of these studies and our findings.

UROP Retention Study

Given the main goal for UROP is to increase undergraduate student retention and academic achievement, we investigated whether, and to what degree, UROP is having those intended influences. The sample of 1,280 students consists of African American, Latino/a, and White UROP student matched to non-UROP students who applied to the program but who were not accepted. We obtained retention data from the university’s Office of the Registrar. It included demographic information (race and gender), term and year of entry, term and year of most recent active enrollment, current enrollment status, and grade point average for each term, cumulative grade point average, and enrollment status by term for each student. We defined retention as students’ persistence through graduation, and attrition as students’ departure from the University of Michigan.

Retention Study Findings

Retention effects were strongest for African American students and for sophomores rather than first-year students.

• UROP has a significant positive effect on male African-American participant’s degree completion. 75.3% degree completion for UROP students compared to 56.3% for control and 57.2% for at large African American males.
• African American students whose academic performance was below the median for their race/ethnic group appeared to benefit the most from UROP participation.
• White students appear to benefit from UROP participation, but not as strongly as African American students. (attrition rate for White UROP students compared to White controls is approximately 1:2).
• UROP does not appear to have an influence on Latino/a retention rates.
• UROP participation increases degree completion rates for male African American, White, and students of color, but not for male Hispanic participants or female UROP participants.

Focus Group Study

Group interviews with UROP students, non-UROP students, and students participating in another retention program (RP), were intended to identify the impact of UROP on students not otherwise identified via our other research projects. The focus group study allowed us to identify the influence of UROP from the perspective of the students. Students were interviewed in groups, with UROP students in one group, non-UROP students in another, and RP students in another. . Data were analyzed using a grounded theory method that consisted of multiple readings of the focus group interview transcripts to identify themes.

Focus Group Findings

Students discuss their undergraduate experiences in three distinct manners:

1. Proactive — Tendency to initiate activity; anticipate problems before they arise, and act before acted upon, and to seek out help from individuals.
2. Reactive — Respond to others’ actions, and feeling at the whim of the college environment.
3. Inactive — Non-interaction with the environment (not attending classes, not doing homework).

• UROP students make 58% of the proactive comments.
• UROP students are more likely to discuss anticipating future events, such as looking for jobs or going to graduate school.
• UROP students are more likely to initiate activity with people than non-UROP students are and see people (faculty, staff, etc.) as positive influences on their academic experiences.
• Non-UROP students make 68% of the reactive/inactive comments.
• Non-UROP students are more likely to react to people and see people as barriers to their learning experiences than UROP students.
• Non-UROP students make 69% of the reactive/inactive comments about people.
• Underrepresented minority students value both the research and programmatic aspects of UROP whereas the White and Asian students value the research more strongly than the programmatic components (peer groups, advising, etc.)

UROP Alumni Survey

The Alumni Survey was designed to identify UROP effects beyond graduation and to examine whether and to what degree UROP students differed on post-graduate education, experiences, and career pathways. The Alumni Survey sample was derived by matching UROP students to 2-4 non-UROP students. We matched UROP students to the controls on high school grade-point-average, SAT scores, ACT scores, intended major, race/ethnicity, and where possible, high school type. Control students were students who had applied to UROP but who were rejected. We had a 58.55% survey return rate for a sample of 291 students. The survey asked students about their post-graduate experiences, including graduate school, career choice, and satisfaction with current job.

Alumni Study Findings

• Students who participate in undergraduate research (UROP or other research) are significantly more likely to pursue post-graduate education than control students.
• UROP students are significantly more likely to pursue medical, law, or Ph.D. degrees than control students.
• Students who participate in undergraduate research are significantly more likely to be engaged in post-graduate research activity compared to control students.
• Students who participate in undergraduate research are significantly more likely to utilize an undergraduate faculty member for a job recommendation than control students.
• There are no differences or interaction by race/ethnicity on post-graduate education pursuit, post-graduate research activity, or job recommendation usage.

Synthesis of Research Findings

Our assessment and evaluation efforts suggest a positive benefit of UROP participation on student’s retention and academic achievement, pursuit of post-graduate education, post-graduate research activity, behavioral orientation (active participation in their educational experiences), and development as researchers. However, the findings are rooted in on differences by race/ethnicity, gender, and incoming personality characteristics.

Conclusions

UROP does influence students’ academic achievement, retention, behavior, and post-graduate educational and professional activities – all intended goals for the program.

Findings from several of the research projects suggest a strong connection between UROP participation and proactive behavior, with UROP students discussing how they actively interact with the academic environment.

Findings indicate that African American students’ retention and academic achievement does benefit from a program designed to integrate students into one of the core goals of higher education – research and the pursuit of knowledge. In addition, findings indicated that UROP extends its effect beyond the undergraduate experience by retaining students in the educational pipeline after graduation.

Male UROP participants benefit more strongly from involvement in undergraduate research in terms of higher degree completion rates than male non-participants.

The research is strengthened by the use of control students who applied to UROP, but who were not accepted. Hence, the controls should be similar to UROP students to whom they are compared on characteristics, such as motivation and academic commitment, that could be related to the outcomes specified.

Future research is needed to determine whether UROP facilitates proactive behavior or strengthens those behaviors already present.

Publications

Nagda, B.A., Gregerman, S.R., Jonides, J., von Hippel, William, Lerner, J.S., (1998). Undergraduate Student-Faculty Research Partnerships Affect Student Retention. The Review of Higher Education, Volume 22,pp.55-72.

Gregerman, S.R., (1999). Improving the Academic Success of Diverse Students Through Undergraduate Research. Council on Undergraduate Research Quarterly. December 1999.

Hathaway, R. S., Nagda, B.A., Gregerman, S.R., The Relationship of Undergraduate Research Participation to Graduate and Professional Educational Pursuit, An Empirical Study, Journal of College Student Development, (2002)

Gregerman, Sandra R., "The Role of Undergraduate Research in Student Retention, Academic Engagement, and the Pursuit of Graduate Education," Evidence on Promising Practices in Undergraduate Science, Commissioned Paper, Technology, Engineering, and Mathematics (STEM) Education Workshop 2 October 13-14, 2008 

Steiner, J. (2016). Engaging Students in Community-Based Research in Detroit. Council on Undergraduate Research Quarterly Summer 2016, 36(3). (Peer-Reviewed)