Tracking Alumni Outcomes: A Crucial Endeavor for Modern Higher Education
The landscape of higher education is undergoing a significant transformation, with institutions increasingly recognizing the imperative to track and disseminate information about their alumni's career outcomes. This shift, spurred by decades of advocacy and evolving societal expectations, is crucial for demonstrating institutional value, informing curriculum development, and guiding current students toward successful professional lives. Organizations such as Rescuing Biomedical Research and Future of Research have been instrumental in bringing this issue to the forefront of graduate education discussions. The second Future of Biomedical Graduate and Postdoctoral Training conference, for instance, explicitly included the collection of career outcomes data in its final recommendations (Hitchcock et al., 2017). More recently, the formation of the Coalition for Next Generation Life Science (CNGLS), comprising 48 institutions, signifies a collective commitment to the ongoing collection and dissemination of career data for both graduate and postdoctoral alumni. This article delves into the development and implementation of a methodology for collecting, examining, and reporting graduate and postdoctoral career outcomes, drawing heavily on the experiences and practices of the University of California, San Francisco (UCSF).
The Imperative for Alumni Outcome Tracking
For too long, a significant gap has existed in understanding the post-graduation trajectories of university alumni. This "dearth of data," particularly concerning postdocs (National Academy of Sciences, National Academy of Engineering, and Institute of Medicine, 2014), has limited the ability of institutions to fully grasp the impact of their educational programs. Your graduates provide the school’s most important public track record of employability and career success, which can motivate new students to consider applying to your institution. The persistent challenge lies in incomplete data collection-only certain alumni consistently report their outcomes, leaving most institutions with limited visibility into their graduates’ professional performance and career trajectories.
Beyond institutional pride and alumni networking, the drive to track outcomes is also influenced by external pressures. With President Obama’s emphasis on increasing the economic value of higher education, institutions may also face a regulatory requirement to track graduates’ employment and earnings. This necessitates a move beyond traditional methods, which often rely on self-reported data and suffer from low response rates. Almost all alumni data is self-reported; there are no federal databases that include both educational and economic data on an individual level, and the creation of such a federal database is forbidden by privacy regulations. Without access to such verifiable data, institutions have to rely on surveys and social networks to track graduate outcomes; both sources contain self-reported data.
Developing a Robust Data Collection Methodology: The UCSF Experience
The University of California, San Francisco (UCSF) has undertaken a comprehensive effort to address this data deficit. Here, we describe the development and implementation of a methodology for collecting, examining, and reporting graduate and postdoctoral career outcomes data at UCSF. As a service to the community, we describe and share all tools we have developed, and we provide calculations of the time and resources required to accomplish both retrospective and annual data collection and reporting.
UCSF has developed and is maintaining two distinct datasets: one for PhD alumni, initiated in 1996, and one for postdoctoral alumni, with data collection starting in 2011. For PhD alumni, a record is created for each student as he or she matriculates to the program. The postdoctoral alumni dataset includes every postdoctoral scholar who left the institution since 2011. In both cases, the datasets incorporate all available demographic information, previous education, program and degree information, and job titles and employers. Data is transferred from the student information system (for PhD alumni, via application programming interface [API]) and the Office of Institutional Research (for postdoctoral alumni, based on Human Resources records).
Read also: Your Guide to Track and Field Scholarships
Figure 1: Overview of Data Flow for PhD Alumni Career Outcomes. Postdoctoral outcomes data flow is similar, except where noted. Basic demographic and degree information is transferred to REDCap from the student information system; postdoctoral data comes from Human Resources. Annually, staff administer a one-time survey requesting current employment information from PhD alumni and conduct online searches for those who do not return a survey (postdocs and PhDs). Employment data are recorded in REDCap.
Choosing the Right Data Management System
The selection of an appropriate system for data collection, management, curation, and archiving is paramount. UCSF considered multiple platforms, including Microsoft Excel, Microsoft Access, Smartsheets, Salesforce, and REDCap. The critical requirement was a database that could offer both flexibility and stability. Flexibility is essential because while the breadth of alumni careers and organizations can be anticipated, unforeseen scenarios will inevitably arise, necessitating adjustments to data fields. Stability is equally important to prevent data corruption as these adjustments are made.
Ultimately, REDCap, developed by Vanderbilt University, was chosen. It is free, open-source, and possesses all the required features. REDCap's "data access groups" feature is particularly valuable, allowing administrators to grant different users varying levels of access to specific subsets of data, including view-only permissions. This granular control is crucial for adhering to regulations like the Federal Education Rights and Privacy Act (FERPA) while enabling various campus stakeholders to access relevant data without compromising its integrity. Furthermore, REDCap's integrated survey function and its ability to generate exportable reports (as .CSV files) streamline the data collection and analysis process. The platform's API capabilities were leveraged to directly update student demographic and enrollment data from UCSF's student information system. UCSF has made its REDCap data dictionaries for graduate student and postdoc outcomes databases available as Supplemental Material S3 and S4, respectively.
Phases of Data Collection: Retrospective and Ongoing
The data collection effort at UCSF was structured into two distinct phases: retrospective and ongoing.
Retrospective Data Collection: This phase involves identifying past positions of alumni who graduated previously. It is significantly more labor-intensive than ongoing collection. UCSF began retrospective data collection in 2017, initially relying entirely on Internet searches, as previously described (Silva et al., 2016). The aim was to record one position per year for up to 15 years after leaving the institution, capturing an annual snapshot around June-August. To simplify the database structure and due to unreliably available information, start and end dates for positions were forgone. It's important to note that very briefly held positions might occasionally be missed. Author E.A.S. observed that when logged into her account as the searcher, Google was more likely to return the correct individual as the top result, highlighting the impact of personalized search experiences.
Read also: Legacy of UGA Track and Field
Ongoing Data Collection: Launched in 2018, this phase focuses on recording the most recent position for each alumnus. For the PhD alumni dataset, survey results were incorporated into the data collection method. The high response rate achieved was attributed to two key factors: the brevity of the survey and an appeal to the cause. Respondents were assured that their data would be displayed anonymously in aggregate. A link to the public display of retrospective data was included, allowing prospective participants to see how their data would be utilized.
While results are publicly displayed in 5-year increments, annual data collection offers three significant advantages:
- Ease of Location and Update: It is easier to locate and update each individual annually through Internet searching.
- Nuanced Career Trajectories: Annual updates reveal more nuanced career trajectories, which are invaluable for student and postdoctoral services staff advising trainees on career exploration and decision-making. For example, individuals who are unemployed rarely self-identify as such in retrospective data. Alumni who left the institution more recently are generally easier to find, and their current positions are more readily discoverable than past roles.
Classification and Quality Assurance
To ensure consistency and comparability, UCSF employs a taxonomy developed collectively in 2017 by representatives of universities with NIH Broadening Experiences in Scientific Training awards, members of Rescuing Biomedical Research, and the founding institutions of the CNGLS. Classification terms are applied by UCSF staff, rather than the alumni themselves, to maintain uniformity. While most positions clearly fall into categories for career type and sector, many jobs do not fit neatly into a specific career category for job function.
To address potential inconsistencies, an audit process is implemented annually. After initial classification, a random subset of records is assigned for re-review by the coders who applied the classifications. In the retrospective phase, 200 individuals were assigned to each of three reviewers. Using a basic spreadsheet, each reviewer identified records that might require further examination and provided notes detailing any issues. This process not only identified a few errors but, more importantly, highlighted inconsistencies in coding that could be rectified in bulk. For instance, a discrepancy arose regarding the classification of fellowship programs that provide a direct pathway from graduate school to independent research, effectively bypassing the postdoctoral stage. The team had differing interpretations of whether to classify these as training positions or independent faculty-like positions ("faculty, tenure-track not applicable"). This audit process prompted clear classification decisions. A summary of these audits, encompassing both retrospective and ongoing studies, is provided in Table 2, listing the type of correction made in order of frequency. For PhD alumni, the most frequent inconsistency by far was in the classification of faculty as tenure-track.
Resource Allocation and Personnel
The scope and scale of this project demand significant staff time and well-defined roles. UCSF estimates the required time and outlines the roles and responsibilities of the primary personnel:
Read also: Comprehensive Biomedical Engineering Guide
- Project Sponsor: This individual makes overarching project decisions, directs data collection and analysis, and is the primary auditor. Typically, this role is filled by a dean, associate/assistant dean, or director of a relevant unit. At UCSF, this is author E.A.S.
- Project/Data Manager: This person documents project goals, communicates project status, tracks time and effort, identifies roles and responsibilities, and monitors project details. Secondary responsibilities include data collection, consolidation and management in REDCap, database administration, and data-quality audits and cleanup (author A.B.M.).
- Project Support Staff: These individuals are primarily responsible for searching for and documenting career outcomes in REDCap and classifying job titles and employers.
The data collection and classification for the 15-year retrospective study of PhD student alumni, undertaken in 2017, was completed within three months (June 15 to September 15). Throughout the remainder of 2017 and into 2018, a project sustainment plan was developed and implemented by the project manager, and the project was expanded to include retrospective data collection for the postdoc population. An update of all PhD and postdoc alumni outcomes was completed during the three summer months of 2018.
In Supplemental Material S7, UCSF provides a worksheet that estimates the resources required at other institutions for both retroactive data searches and annual updates of alumni outcomes. They found that it took an average of 10 minutes to complete data collection for each individual trainee in their retrospective study and an average of 5 minutes per trainee in their annual update. Many institutions report delaying commitment to such projects due to concerns about the resources required. Having successfully implemented systems for retrospective and ongoing data collection, UCSF shares its materials and resources to encourage other institutions to adopt similar practices.
Broader Implications and Recommendations for Institutions
Transparency in career outcomes for PhD students and graduates is an achievable goal and, arguably, a responsibility that universities must fulfill. The principle of "don't let the perfect be the enemy of the good" is crucial here. UCSF acknowledged from the outset that they would not be able to find all alumni or categorize every job title with absolute precision. They chose a repository with sufficient flexibility to allow for post hoc adjustments to the data.
Institutions considering implementing similar tracking systems should consider the following recommendations:
- Develop a Project Charter: A project charter establishes clear boundaries for the project's scope and scale, articulates personnel roles, and includes a timeline and description of project milestones. This document is critical for ensuring the project progresses at an acceptable pace and for preventing "mission creep"-unplanned expansions that can hinder completion. A charter was particularly important for UCSF's postdoc dataset, given the historical lack of data in this area.
- Collaborate with Campus Stakeholders: On any campus, career outcomes data may be collected and reported by various stakeholders with limited coordination. Collaboration offers opportunities to improve data quality and reduce overall institutional resources. For instance, graduate program staff and faculty often possess valuable, firsthand knowledge of current graduate positions due to maintained personal connections. While graduate programs may have reliable data, they might lack a robust platform for storing and analyzing it. Collaborating involves collecting accurate alumni information and providing a central platform with user support for data access. Similarly, T32 program directors are often required to report on trainee positions for extended periods. Centralized administration of these efforts can reduce the overall burden by minimizing duplicate efforts and leveraging data management expertise. Furthermore, a significant amount of data can be extracted from existing reports.
- Leverage Technology and Diverse Data Sources: Beyond traditional surveys, institutions can explore platforms designed to gather, measure, and report alumni outcomes within their existing student information ecosystem. Companies like Graduway and EverTrue offer alumni management software that can synchronize with social networks for more complete and current data. University Pages, launched in 2012, allows for specific alumni searches by graduation year, location, industry, and employer, greatly facilitating networking. Manual social network searches, taking approximately one minute per individual, can supplement survey data, increasing response rates and accuracy. While institutions have experimented with private social networks for alumni, these often compete with mainstream platforms. Partnering with third-party providers like iModules can yield better results. Customer Relationship Management (CRM) software is also widely used for managing alumni databases and supporting fundraising.
- Integrate Third-Party Data Enrichment: Edular's Student Relationship Management platform, for example, helps schools manage various operations, including career engagement and outcomes, on a unified foundation. Solutions offered by companies like CyberWarrior can help institutions analyze career data to enhance workforce programs and connect students to high-demand fields. Enriching alumni data with third-party labor market analytics and employer reporting can provide a more comprehensive picture. This can help identify graduates who may need additional career support, upskilling, or networking opportunities. Mapping graduate skills to employer demand can optimize workforce-aligned academic programs, and comparing program outcomes against national benchmarks can provide valuable context. AI-driven analytics can even forecast future hiring trends, allowing for proactive curriculum adjustments.
- Focus on Key Metrics and Reporting: Institutions should aim to track where graduates are working, how long they stay, and their career progression over time. Collecting job placement data for high-demand fields is particularly important. Customizable dashboards and reports can provide real-time insights to academic leadership, career services, and employers.
tags: #track #alumni #outcomes
