scholarly journals Applying Inter-rater Reliability to Improve Consistency in Classifying PhD Career Outcomes

2018 ◽  
Author(s):  
C. Abigail Stayart ◽  
Patrick D. Brandt ◽  
Abigail M. Brown ◽  
Tamara Hutto ◽  
Rebekah L. Layton ◽  
...  
Keyword(s):  
Guidance Document ◽  
Career Outcomes ◽  
Natural Evolution ◽  
Rater Reliability ◽  
Scientific Training ◽  
The Past ◽  
Global Workforce ◽  
Training Materials ◽  
Guidance Documents ◽  
Career Options

ABSTRACTIn the past year, there has been an exciting groundswell of national efforts to integrate multiple taxonomies for the transparent dissemination and analysis of PhD career outcomes. In this study, we leveraged the unique resources of the Broadening Experiences in Scientific Training Consortium to examine the reliability of the three-tiered Unified Career Outcomes Taxonomy (UCOT v.2017) that was collaboratively developed at a meeting convened by Rescuing Biomedical Research in August 2017. Using an amended version of the UCOT v.2017 (UCOT v.2017-rev1) and a new Supplementary Guidance document, we categorized over 570 PhD alumni records from three different universities. Utilizing Krippendorff’s alpha to measure the interrater reliability from nine different individuals, we determined moderate to robust reproducibility within the first two tiers of the taxonomy (Workforce Sector and Career Type); however, the reliability for the third tier (Job Function) did not meet established standards. The team identified significant sources of error, revised category definitions, improved coder training materials and processes, and tested for improved reliability through coding 219 PhD alumni records using the revised taxonomy, UCOT v.2017-rev2. Our results revealed that the changes introduced in UCOT v.2017-rev2 improved inter-rater reliability in all three tiers, and either met or exceeded the acceptable standards for reliability. A final set of clarifications were made to UCOT v.2017-rev2, resulting in UCOT v.2018 and a Finalized Guidance document. Our findings underscore the importance of carefully developing guidance documents to aid coders in the reliable and consistent categorization of alumni career outcomes. We propose periodic assessment of the UCOT v.2018 to address the natural evolution of PhD careers in the global workforce. Ultimately, we hope that UCOT v.2018 will aid in the classification and dissemination of alumni career outcomes that is essential to educating trainees, institutions, and agencies about the diversity of career options for PhDs, and therein empower all PhDs to pursue the careers of their choice.

scholarly journals Applying inter-rater reliability to improve consistency in classifying PhD career outcomes

F1000Research ◽  
2020 ◽  
Vol 9 ◽  
pp. 8 ◽  
Author(s):  
C. Abigail Stayart ◽  
Patrick D. Brandt ◽  
Abigail M. Brown ◽  
Tamara Dahl ◽  
Rebekah L. Layton ◽  
...  
Keyword(s):  
Iterative Refinement ◽  
Professional Organizations ◽  
Guidance Document ◽  
Career Outcomes ◽  
Rater Reliability ◽  
Job Functions ◽  
Global Workforce ◽  
Report Data ◽  
Guidance Documents ◽  
Labeling System

Background: There has been a groundswell of national support for transparent tracking and dissemination of PhD career outcomes. In 2017, individuals from multiple institutions and professional organizations met to create the Unified Career Outcomes Taxonomy (UCOT 2017), a three-tiered taxonomy to help institutions uniformly classify career outcomes of PhD graduates. Early adopters of UCOT 2017, noted ambiguity in some categories of the career taxonomy, raising questions about its consistent application within and across institutions. Methods: To test and evaluate the consistency of UCOT 2017, we calculated inter-rater reliability across two rounds of iterative refinement of the career taxonomy, classifying over 800 PhD alumni records via nine coders. Results: We identified areas of discordance in the taxonomy, and progressively refined UCOT 2017 and an accompanying Guidance Document to improve inter-rater reliability across all three tiers of the career taxonomy. However, differing interpretations of the classifications, especially for faculty classifications in the third tier, resulted in continued discordance among the coders. We addressed this discordance with clarifying language in the Guidance Document, and proposed the addition of a flag system for identification of the title, rank, and prefix of faculty members. This labeling system provides the additional benefit of highlighting the granularity and the intersectionality of faculty job functions, while maintaining the ability to sort by - and report data on - faculty and postdoctoral trainee roles, as is required by some national and federal reporting guidelines. We provide specific crosswalk guidance for how a user may choose to incorporate our suggestions while maintaining the ability to report in accordance with UCOT 2017. Conclusions: Our findings underscore the importance of detailed guidance documents, coder training, and periodic collaborative review of career outcomes taxonomies as PhD careers evolve in the global workforce. Implications for coder-training and use of novice coders are also discussed.

scholarly journals Applying inter-rater reliability to improve consistency in classifying PhD career outcomes

F1000Research ◽  
2020 ◽  
Vol 9 ◽  
pp. 8
Author(s):  
C. Abigail Stayart ◽  
Patrick D. Brandt ◽  
Abigail M. Brown ◽  
Tamara Dahl ◽  
Rebekah L. Layton ◽  
...  
Keyword(s):  
Iterative Refinement ◽  
Professional Organizations ◽  
Guidance Document ◽  
Career Outcomes ◽  
Rater Reliability ◽  
Job Functions ◽  
Global Workforce ◽  
Report Data ◽  
Guidance Documents ◽  
Labeling System

Background: There has been a groundswell of national support for transparent tracking and dissemination of PhD career outcomes. In 2017, individuals from multiple institutions and professional organizations met to create the Unified Career Outcomes Taxonomy (UCOT 2017), a three-tiered taxonomy to help institutions uniformly classify career outcomes of PhD graduates. Early adopters of UCOT 2017, noted ambiguity in some categories of the career taxonomy, raising questions about its consistent application within and across institutions. Methods: To test and evaluate the consistency of UCOT 2017, we calculated inter-rater reliability across two rounds of iterative refinement of the career taxonomy, classifying over 800 PhD alumni records via nine coders. Results: We identified areas of discordance in the taxonomy, and progressively refined UCOT 2017 and an accompanying Guidance Document to improve inter-rater reliability across all three tiers of the career taxonomy. However, differing interpretations of the classifications, especially for faculty classifications in the third tier, resulted in continued discordance among the coders. We addressed this discordance with clarifying language in the Guidance Document, and proposed the addition of a flag system for identification of the title, rank, and prefix of faculty members. This labeling system provides the additional benefit of highlighting the granularity and the intersectionality of faculty job functions, while maintaining the ability to sort by - and report data on - faculty and postdoctoral trainee roles, as is required by some national and federal reporting guidelines. We provide specific crosswalk guidance for how a user may choose to incorporate our suggestions while maintaining the ability to report in accordance with UCOT 2017. Conclusions: Our findings underscore the importance of detailed guidance documents, coder training, and periodic collaborative review of career outcomes taxonomies as PhD careers evolve in the global workforce. Implications for coder-training and use of novice coders are also discussed.

scholarly journals Becoming more transparent: Collecting and presenting data on biomedical Ph.D. alumni

2017 ◽  
Author(s):  
Christopher L Pickett ◽  
Shirley Tilghman
Keyword(s):  
Data Collection ◽  
American Medical ◽  
Biomedical Research ◽  
Training Programs ◽  
Career Outcomes ◽  
Scientific Training ◽  
American Universities ◽  
The Past ◽  
Career Trajectories ◽  
Association Of American Universities

For more than 20 years, panels of experts have recommended that universities collect and publish data on the career outcomes of Ph.D. students. However, little progress has been made. Over the past few years, a handful of universities, including those in the National Institutes of Health’s Broadening Experiences in Scientific Training consortium, and organizations, including the Association of American Universities and the Association of American Medical Colleges, launched projects to collect and publish data on biomedical Ph.D. alumni. Here, we describe the outcome of a meeting, convened by Rescuing Biomedical Research, of universities and associations working to improve the transparency of career outcomes data. We were able to achieve consensus on a set of common methods for alumni data collection and a unified taxonomy to describe the career trajectories of biomedical Ph.D.s. These materials can be used by any institution, with little or no modification, to begin data collection efforts on their Ph.D. alumni. These efforts represent an important step forward in addressing a recommendation that has been made for decades that will improve the ability of trainees to better plan for their careers and for universities to better tailor their training programs.

scholarly journals Becoming more transparent: Collecting and presenting data on biomedical Ph.D. alumni

2018 ◽  
Author(s):  
Christopher L Pickett ◽  
Shirley Tilghman
Keyword(s):  
Data Collection ◽  
American Medical ◽  
Biomedical Research ◽  
Training Programs ◽  
Career Outcomes ◽  
Scientific Training ◽  
American Universities ◽  
The Past ◽  
Career Trajectories ◽  
Association Of American Universities

For more than 20 years, panels of experts have recommended that universities collect and publish data on the career outcomes of Ph.D. students. However, little progress has been made. Over the past few years, a handful of universities, including those in the National Institutes of Health’s Broadening Experiences in Scientific Training consortium, and organizations, including the Association of American Universities and the Association of American Medical Colleges, launched projects to collect and publish data on biomedical Ph.D. alumni. Here, we describe the outcome of a meeting, convened by Rescuing Biomedical Research, of universities and associations working to improve the transparency of career outcomes data. We were able to achieve consensus on a set of common methods for alumni data collection and a unified taxonomy to describe the career trajectories of biomedical Ph.D.s. These materials can be used by any institution, with little or no modification, to begin data collection efforts on their Ph.D. alumni. These efforts represent an important step forward in addressing a recommendation that has been made for decades that will improve the ability of trainees to better plan for their careers and for universities to better tailor their training programs.

scholarly journals Becoming more transparent: Collecting and presenting data on biomedical Ph.D. alumni

2018 ◽  
Author(s):  
Christopher L Pickett ◽  
Shirley Tilghman
Keyword(s):  
Data Collection ◽  
American Medical ◽  
Biomedical Research ◽  
Training Programs ◽  
Career Outcomes ◽  
Scientific Training ◽  
American Universities ◽  
The Past ◽  
Career Trajectories ◽  
Association Of American Universities

For more than 20 years, panels of experts have recommended that universities collect and publish data on the career outcomes of Ph.D. students. However, little progress has been made. Over the past few years, a handful of universities, including those in the National Institutes of Health’s Broadening Experiences in Scientific Training consortium, and organizations, including the Association of American Universities and the Association of American Medical Colleges, launched projects to collect and publish data on biomedical Ph.D. alumni. Here, we describe the outcome of a meeting, convened by Rescuing Biomedical Research, of universities and associations working to improve the transparency of career outcomes data. We were able to achieve consensus on a set of common methods for alumni data collection and a unified taxonomy to describe the career trajectories of biomedical Ph.D.s. These materials can be used by any institution, with little or no modification, to begin data collection efforts on their Ph.D. alumni. These efforts represent an important step forward in addressing a recommendation that has been made for decades that will improve the ability of trainees to better plan for their careers and for universities to better tailor their training programs.

Hidden Politics? Assessing Lobbying Success During US Agency Guidance Development

2020 ◽  
Vol 30 (4) ◽  
pp. 548-562 ◽  
Author(s):  
Susan Webb Yackee
Keyword(s):  
Interest Groups ◽  
Development Process ◽  
Guidance Document ◽  
Public Attention ◽  
Policy Success ◽  
Government Documents ◽  
Us Government ◽  
Guidance Documents ◽  
The Relationship ◽  
Guidance Development

Abstract I test the proposition that interest groups achieve greater policy success when they lobby during the agency guidance document development process as opposed to the notice and comment process. Policymaking via guidance documents often receives lower levels of public attention, which provides greater flexibility to accommodate lobbying requests. I analyze the hypothesis during the creation of 41 rules by one US government agency—20 of which were promulgated using the notice and comment process and 21 via the guidance document process. I measure regulatory policy change using a content analysis of government documents and lobbying texts, and I also incorporate a telephone survey of interest groups who lobbied on these same rules. I find that interest groups perceive—and achieve—greater policy success when lobbying during the agency guidance process. The results yield new insights into the relationship between lobbying and regulatory policymaking.

scholarly journals International Council for Standardization in Haematology Recommendations for Hemostasis Critical Values, Tests, and Reporting

2019 ◽  
Vol 46 (04) ◽  
pp. 398-409 ◽  
Author(s):  
Robert C. Gosselin ◽  
Dorothy Adcock ◽  
Akbar Dorgalaleh ◽  
Emmanuel J. Favaloro ◽  
Giuseppe Lippi ◽  
...  
Keyword(s):  
Ad Hoc ◽  
Test List ◽  
Published Data ◽  
Guidance Document ◽  
International Council ◽  
Hospital Managers ◽  
Coagulation Tests ◽  
Good Laboratory ◽  
Guidance Documents ◽  
Institutional Stakeholders

AbstractThis guidance document was prepared on behalf of the International Council for Standardization in Haematology (ICSH), the aim of which is to provide hemostasis-related guidance documents for clinical laboratories. The current ICSH document was developed by an ad hoc committee, comprising an international collection of both clinical and laboratory experts. The purpose of this ICSH document is to provide laboratory guidance for (1) identifying hemostasis (coagulation) tests that have potential patient risk based on analysis, test result, and patient presentations, (2) critical result thresholds, (3) acceptable reporting and documenting mechanisms, and (4) developing laboratory policies. The basis for these recommendations was derived from published data, expert opinion, and good laboratory practice. The committee realizes that regional and local regulations, institutional stakeholders (e.g., physicians, laboratory personnel, hospital managers), and patient types (e.g., adults, pediatric, surgical) will be additional confounders for a given laboratory in generating a critical test list, critical value thresholds, and policy. Nevertheless, we expect this guidance document will be helpful as a framework for local practice.

scholarly journals The National Center for Advancing Translational Sciences’ Intramural Training Program and Fellow Career Outcomes

2020 ◽  
Vol 19 (4) ◽  
pp. ar51 ◽  
Author(s):  
Brittany Haynes ◽  
Kyle Brimacombe ◽  
Christy Hare ◽  
Jessica Faupel-Badger
Keyword(s):  
Training Program ◽  
Biomedical Research ◽  
Research Program ◽  
Early Career ◽  
Career Outcomes ◽  
Skill Sets ◽  
Intramural Research Program ◽  
Career Options

The translational scientist skill sets are consistent with those currently being emphasized in biomedical research to prepare trainees for various career options. The framework of the National Center for Advancing Translational Sciences intramural research program and the career outcomes of its alumni will be of interest to those involved in the career preparedness of early-career scientists.

scholarly journals Pirates as an Idea

2008 ◽  
Vol 1 (2) ◽  
pp. 8-15
Author(s):  
Umme Abida ◽  
Paul Hitchens ◽  
Bibi Laegaard ◽  
Sheldon Rozario
Keyword(s):  
21St Century ◽  
Natural Evolution ◽  
The Past ◽  
The Press ◽  
Long Time ◽  
Gps Devices ◽  
High Seas

At least 12 foreign ships are being held hostage in the waters off Eyl in the Nugal region, 300 nautical miles south of Africa's Horn, including a Ukrainian vessel, the MV Faina, loaded with 33 tanks and ammunition that was hijacked in September. The captured ships are being closely watched by hundreds of pirates aboard boats equipped with satellite phones and GPS devices. Hundreds mo re gunmen provide back-up on shore, where they incessantly chew the narcotic leaf ‘qat’ and dream of sharing in the huge ransoms that can run into millions of dollars. (Source: The Age) The above article is but one of many in the press in recent times reflecting how present day pirates operate. Pirates are still ruling the high seas, but historically what triggered their choice of lifestyle and why are they still active in the 21st century? Could actions in the past have eradicated the pirate industry a long time ago or is it just part of natural evolution?

scholarly journals A revised scoring scheme for the classification of socio-economic status in Nigeria

2021 ◽  
Vol 48 (1) ◽  
pp. 26-33
Author(s):  
Michael O. Ibadin ◽  
George O. Akpede
Keyword(s):  
Socioeconomic Status ◽  
Public Service ◽  
Economic Status ◽  
Rater Reliability ◽  
Comprehensive Review ◽  
Socio Economic Status ◽  
The Past ◽  
The University

Background: Researchers in medicine and related fields in Nigeria have usually made recourse to the instrument developed by Olusanya et al and Oyedeji in the past three-and-a half decades for determination of socioeconomic status (SES). Beside the question of their age, however, these instruments were purposive and might no longer be suitable because of the changes in the parameters on which they were based.Objective: To develop a robust but generic scheme that takes into consideration the changes in the nation’s socioeconomic space in the succeeding three and a half decades.Methods: A detailed and comprehensive review of the extant schemes was undertaken with a view to identifying their inherent weaknesses. The latter were then factored into the design of a new scheme taking into consideration the emergent restructuring of career positions in the civil/public service as well as the place of private and informal sectors of the economy. The new scheme was validated at the University of Beninand Irrua Specialist Teaching Hospitals.Results: The new scheme had a remarkably high Inter-rater reliability (r = 0.947, p<0.001), raterre- rater reliability (r = 0.984, p <0.001) and % agreement (with modified Oyedeji’s tool as standard) of 67% (K coefficient = 0.47, r = 0.71, p<0.001)Conclusion/Recommendation: The new scheme could be a viable tool for the assessment of SES of families and individuals, which not only takes into consideration current realities of the nation’s economy, but also is readily adaptable to meet foreseeable changes.

Sign in / Sign up

Export Citation Format

Share Document