scholarly journals Visual and Statistical Methods to Calculate Interrater Reliability for Time-Resolved Qualitative Data: Examples from a Screen Capture Study of Engineering Writing Patterns

2020 ◽  
Author(s):  
Manoj Malviya ◽  
Catherine Berdanier ◽  
Natascha Buswell
Keyword(s):  
Statistical Methods ◽  
Interrater Reliability ◽  
Qualitative Data ◽  
Time Resolved ◽  
Screen Capture ◽  
Engineering Writing

scholarly journals Visual and Statistical Methods to Calculate Interrater Reliability for Time-Resolved Qualitative Data: Examples from a Screen Capture Study of Engineering Writing Patterns

2019 ◽  
Author(s):  
Manoj Malviya ◽  
Catherine G P. Berdanier
Keyword(s):  
Interrater Reliability ◽  
Qualitative Data ◽  
Visual Methods ◽  
Time Resolved ◽  
Ongoing Research ◽  
New Methods ◽  
Screen Capture ◽  
Statistical Measures ◽  
Engineering Writing ◽  
Using Data

Traditionally, interrater reliability (IRR) is determined for easily defined events, such as deciding within which category a piece of qualitative data falls. However, for time-resolved or time-dependent observational data and other nontraditional data, complications arise due to the complexity of the data being interpreted and analyzed. In this paper, we present two promising new methods for calculating IRR based on visual representations of analyzed time-resolved data. We compare the IRR calculated using these two visual methods with five of the most common statistical measures for calculating IRR, finding excellent agreement between our new methods and existing statistical formulae. This methods development is exemplified using data for our ongoing research, in which we are working to analyze time-resolved engineering writing data recorded through screen capture technology. The process of developing methods of interrater reliability for our context can also be applied to other researchers who seek to analyze non-traditional data, such as those collected during eye-tracking, screen capture, or observational studies.

scholarly journals Qualitative Evaluation of the Project P.A.T.H.S.: An Integration of Findings Based on Program Implementers

2012 ◽  
Vol 2012 ◽  
pp. 1-14 ◽  
Author(s):  
Daniel T. L. Shek
Keyword(s):  
Hong Kong ◽  
Focus Groups ◽  
Interrater Reliability ◽  
Qualitative Data ◽  
Qualitative Evaluation ◽  
Data Analyses ◽  
Tier 1

An integration of the qualitative evaluation findings collected from program implementers conducting the Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes) in different years (n=177participants in 36 focus groups) was carried out. General qualitative data analyses utilizing intra and interrater reliability techniques were performed. Results showed that the descriptors used to describe the program and the metaphors named by the informants that could stand for the program were generally positive in nature. Program participants also perceived the program to be beneficial to the development of the students in different psychosocial domains. The present study further supports the effectiveness of the Tier 1 Program of the Project P.A.T.H.S. in Hong Kong based on the perspective of the program implementers.

scholarly journals Data Analysis Methods for Qualitative Research: Managing the Challenges of Coding, Interrater Reliability, and Thematic Analysis

2018 ◽  
Author(s):  
Michael Belotto
Keyword(s):  
Qualitative Research ◽  
Data Analysis ◽  
Thematic Analysis ◽  
Interrater Reliability ◽  
Qualitative Data ◽  
Sources Of Information ◽  
Qualitative Data Analysis ◽  
Qualitative Researcher ◽  
First Time ◽  
Data Analysis Methods

The purpose of this article is to provide an overview of some of the principles of data analysis used in qualitative research such as coding, interrater reliability, and thematic analysis. I focused on the challenges that I experienced as a first-time qualitative researcher during the course of my dissertation, in the hope that how I addressed those difficulties will better prepare other investigators planning endeavors into this area of research. One of the first challenges I encountered was the dearth of information regarding the details of qualitative data analysis. While my text books explained the general philosophies of the interpretive tradition and its theoretical groundings, I found few published studies where authors actually explained the details pertaining to exactly how they arrived at their findings. Some authors even confirmed my own experience that few published studies described processes such as coding and methods to evaluate interrater reliability. Herein, I share the sources of information that I did find and the methods that I used to address these challenges. I also discuss issues of trustworthiness and how matters of objectivity and reliability can be addressed within the naturalistic paradigm.

Using Qualitative Data for Estimating Construct Validity of Standardized Measures

1996 ◽  
Vol 4 (1) ◽  
pp. 59-74 ◽  
Author(s):  
Karen J. Aroian ◽  
Nancy Schappler-Morris
Keyword(s):  
Construct Validity ◽  
Validation Study ◽  
Interrater Reliability ◽  
Qualitative Data ◽  
Rating Scale ◽  
Construct Validation ◽  
Qualitative Interview ◽  
Interview Data ◽  
Standardized Measures ◽  
Observer Ratings

This paper describes efforts to develop a method of using qualitative interview data for investigating the construct validity of standardized measures. Illustration is provided through a multimethod validation study that examined the concurrence between observer assessments of interview data and a standardized paper and pencil measure–the Demands of Immigration Scale (DI). Specific aims of the multimethod validation study included (1) determining interrater reliability of the Demands of Immigration Observer Rating Scale (DI-ORS), which assessed respondents according to the interview data they provided and (2) establishing concurrence between observer assessments on the DI-ORS and the respondents’ self-ratings on the DI scale. Interrater reliability of the DI-ORS was achieved with 97.9% agreement. However, there were significant differences (p ≤ .05,2-tailed) or disagreement on 5 out of 6 possible occurrences for agreement or disagreement between observer ratings on the DI-ORS and respondents’ self ratings on the DI scale. The findings are informative for researchers who wish to use qualitative methods for construct validation.

scholarly journals Data Visualization for Time-Resolved Real-Time Engineering Writing Processes

2020 ◽  
Author(s):  
Catherine Berdanier ◽  
Natascha Buswell ◽  
Zixuan (Victoria) Zhao
Keyword(s):  
Real Time ◽  
Data Visualization ◽  
Time Resolved ◽  
Writing Processes ◽  
Engineering Writing

A Typology of Male Prisoners Making Near-Lethal Suicide Attempts

Crisis ◽  
2013 ◽  
Vol 34 (5) ◽  
pp. 335-347 ◽  
Author(s):  
Adrienne Rivlin ◽  
Robert Ferris ◽  
Lisa Marzano ◽  
Seena Fazel ◽  
Keith Hawton
Keyword(s):  
Suicide Risk ◽  
Interrater Reliability ◽  
Qualitative Data ◽  
Suicide Attempts ◽  
Psychotic Symptoms ◽  
Suicide Method ◽  
Forensic Institutions ◽  
Assist Suicide ◽  
Male Prisoners ◽  
Interview Transcript

Background: Prisoners are at high risk of suicide. Aims: This study aimed to develop a typology of prison suicide. Method: We interviewed 60 male prisoners who made near-lethal suicide attempts in prison to obtain quantitative and qualitative data regarding psychiatric, psychological, social, and criminological factors. We analyzed this information to develop a typology to classify suicidal prisoners and validated it by having a prison psychiatrist independently rate each interview transcript. Results: We developed a typology of five subgroups: attempts that (1) were due to a prisoner being unable to cope in prison, (2) were motivated by psychotic symptoms, (3) had instrumental motives, (4) were ”unexpected” by the prisoners themselves, and (5) were associated with withdrawal from drugs. The interrater reliability as measured by Cohen’s was good to excellent at 0.81 (p < .001), 95% CI (0.69, 0.93). Conclusion: With further validation in other samples, this typology may assist suicide prevention initiatives in prisons as well as other forensic institutions by informing the assessment and formulation of suicide risk.

scholarly journals The approach to the environmental issues in postgraduate nursing in Brazil

2021 ◽  
Vol 74 (1) ◽  
Author(s):  
Camila Henriques Nunes ◽  
Mario Sergio Michaliszyn ◽  
Eduardo Shimoda ◽  
Karla Rangel Ribeiro ◽  
Ronilson Gonçalves Rocha
Keyword(s):  
Content Analysis ◽  
Statistical Methods ◽  
Quantitative Data ◽  
Quantitative Research ◽  
Qualitative Data ◽  
Environmental Issues ◽  
Environmental Conservation ◽  
Cultural Aspects ◽  
Analysis Technique ◽  
Exploratory Approach

ABSTRACT Objectives: to analyze the insertion of environmental issues in the postgraduate stricto sensu in nursing in Brazil. Methods: quali-quantitative research, of exploratory approach, with documental research in the Sucupira Platform and in sites of the 112 courses stricto sensu in Nursing in Brazil, in 2018. We analyzed the curricular matrixes; course menus; teaching profile; dissertations and thesis (2004-2016); and bibliographic productions of the four-year evaluation 2013-2016. The analysis of quantitative data was performed by statistical methods; and qualitative data, by the Content Analysis Technique, outlined by Bardin. Results: the results demonstrate incipiency in the treatment of environmental issues. Regarding the theme, we highlight those that refer to socio-historical-cultural aspects in relation to the more direct environmental approach and facilitator of environmental conservation and consequent promotion of health. Final Considerations: The analysis allows identifying the fragility of the environmental approach in the analyzed programs, being necessary to improve the cross-sectionality of environmental issues.

Sign in / Sign up

Export Citation Format

Share Document