Are we testing for true reliability?

IEEE Software ◽  
1992 ◽  
Vol 9 (4) ◽  
pp. 21-27 ◽  
Author(s):  
D. Hamlet
Keyword(s):  
True Reliability

scholarly journals The Perception of Automation Reliability and Acceptance of Automated Advice

2022 ◽  
pp. 001872082110629
Author(s):  
Jack Hutchinson ◽  
Luke Strickland ◽  
Simon Farrell ◽  
Shayne Loft
Keyword(s):  
Cognitive Science ◽  
Critical Factors ◽  
Recency Effects ◽  
Positive Difference ◽  
Reliability Estimates ◽  
The Common ◽  
The Difference ◽  
Acceptance Rates ◽  
True Reliability

Objective Examine (1) the extent to which humans can accurately estimate automation reliability and calibrate to changes in reliability, and how this is impacted by the recent accuracy of automation; and (2) factors that impact the acceptance of automated advice, including true automation reliability, reliability perception, and the difference between an operator’s perception of automation reliability and perception of their own reliability. Background Existing evidence suggests humans can adapt to changes in automation reliability but generally underestimate reliability. Cognitive science indicates that humans heavily weight evidence from more recent experiences. Method Participants monitored the behavior of maritime vessels (contacts) in order to classify them, and then received advice from automation regarding classification. Participants were assigned to either an initially high (90%) or low (60%) automation reliability condition. After some time, reliability switched to 75% in both conditions. Results Participants initially underestimated automation reliability. After the change in true reliability, estimates in both conditions moved towards the common true reliability, but did not reach it. There were recency effects, with lower future reliability estimates immediately following incorrect automation advice. With lower initial reliability, automation acceptance rates tracked true reliability more closely than perceived reliability. A positive difference between participant assessments of the reliability of automation and their own reliability predicted greater automation acceptance. Conclusion Humans underestimate the reliability of automation, and we have demonstrated several critical factors that impact the perception of automation reliability and automation use. Application The findings have potential implications for training and adaptive human-automation teaming.

Reliability from Damaged Stress and Strength Data

1996 ◽  
Vol 46 (1-2) ◽  
pp. 135-142 ◽  
Author(s):  
S. P. Mukherjee ◽  
Sudhansu S. Maiti
Keyword(s):  
Strength Data ◽  
True Reliability

In case of stress-strength reliability R = P( X > Y), inference is made under various assumptions regarding the variables X and Y. In reality instead of observing X and Y one observes U and V which imply stress and strength subject to some sort of errors. In this article, procedures have been Indicated to estimate true reliability using U and V values under the assumption of exponentiality. Over and/or under-reporting has been treated generally as damage.

scholarly journals Peirce'siand Cohen'sκfor2×2Measures of Rater Reliability

2010 ◽  
Vol 2010 ◽  
pp. 1-10
Author(s):  
Beau Abar ◽  
Eric Loken
Keyword(s):  
Sample Size ◽  
Mixture Model ◽  
Gold Standard ◽  
Mixture Model Approach ◽  
Size Population ◽  
True Reliability ◽  
Better Than ◽  
Made In ◽  
True Science ◽  
Model Approach

This study examined a historical mixture model approach to the evaluation of ratings made in “gold standard” and two-rater2×2contingency tables. Peirce'siand the derivediaverage were discussed in relation to a widely used index of reliability in the behavioral sciences, Cohen'sκ. Sample size, population base rate of occurrence, the true “science of the method”, and guessing rates were manipulated across simulations. In “gold standard” situations, Peirce'sitended to recover the true reliability of ratings as well as better thanκ. In two-rater situations,iavetended to recover the true reliability as well as better thanκin most situations. The empirical utility and potential theoretical benefits of mixture model methods in estimating reliability are discussed, as are the associations between theistatistics and other modern mixture model approaches.

An Examination of the True Reliability of Lower Limb Stiffness Measures During Overground Hopping

2016 ◽  
Vol 32 (3) ◽  
pp. 278-286 ◽  
Author(s):  
David Diggin ◽  
Ross Anderson ◽  
Andrew J. Harrison
Keyword(s):  
Lower Limb ◽  
Joint Stiffness ◽  
Time Of Day ◽  
Testing Time ◽  
Attractor State ◽  
Tendon Stiffness ◽  
Consistency Results ◽  
Limb Stiffness ◽  
Measurement Artifact ◽  
True Reliability

Evidence suggests reports describing the reliability of leg-spring (kleg) and joint stiffness (kjoint) measures are contaminated by artifacts originating from digital filtering procedures. In addition, the intraday reliability of kleg and kjoint requires investigation. This study examined the effects of experimental procedures on the inter- and intraday reliability of kleg and kjoint. Thirty-two participants completed 2 trials of single-legged hopping at 1.5, 2.2, and 3.0 Hz at the same time of day across 3 days. On the final test day a fourth experimental bout took place 6 hours before or after participants’ typical testing time. Kinematic and kinetic data were collected throughout. Stiffness was calculated using models of kleg and kjoint. Classifications of measurement agreement were established using thresholds for absolute and relative reliability statistics. Results illustrated that kleg and kankle exhibited strong agreement. In contrast, kknee and khip demonstrated weak-to-moderate consistency. Results suggest limits in kjoint reliability persist despite employment of appropriate filtering procedures. Furthermore, diurnal fluctuations in lower-limb muscle-tendon stiffness exhibit little effect on intraday reliability. The present findings support the existence of kleg as an attractor state during hopping, achieved through fluctuations in kjoint variables. Limits to kjoint reliability appear to represent biological function rather than measurement artifact.

MRC sum-score in the ICU: Good reliability does not necessarily reflect “true reliability”

2012 ◽  
Vol 45 (5) ◽  
pp. 767-768 ◽  
Author(s):  
Els Karla Vanhoutte ◽  
Catharina Gerritdina Faber ◽  
Ingemar Sergio José Merkies ◽  
Keyword(s):  
Good Reliability ◽  
Sum Score ◽  
True Reliability

scholarly journals Human Behavioral Response to Fluctuating Automation Reliability

2021 ◽  
Author(s):  
Jack Hutchinson ◽  
Simon Farrell ◽  
Luke Joseph Gough Strickland ◽  
Shayne Loft
Keyword(s):  
Behavioral Response ◽  
High Reliability ◽  
Human Perception ◽  
Dynamic Environments ◽  
Classification Task ◽  
Reliability Estimates ◽  
Acceptance Rates ◽  
True Reliability ◽  
Classification Reliability ◽  
Over Time

Human perception of automation reliability and automation acceptance behaviours are key to effective human-automation teaming. This study examined factors that impact perceptions of automation reliability over time and the acceptance of automated advice. Participants completed a maritime vessel classification task in which they classified vessels (contacts) with the assistance of automation. In Experiment 1 automation reliability successively switched from high to low (or vice versa). In Experiment 2 automation reliability decreased by varying magnitudes before returning to high. Participants did not initially calibrate to true reliability and experiencing low automation reliability reduced future reliability estimates when experiencing subsequent high reliability. Automation acceptance was predicted by positive differences between participants perception of automation reliability and confidence in their own classification reliability. Experiencing low automation reliability caused perceptions of reliability and automation acceptance rates to diverge. These findings have important implications for training and adaptive human-automation teaming in complex and dynamic environments.

True Reliability Versus Inspection Efficiency

1969 ◽  
Vol R-18 (4) ◽  
pp. 201-203 ◽  
Author(s):  
R. A. Blais
Keyword(s):  
True Reliability
Sign in / Sign up

Export Citation Format

Share Document