scholarly journals Reliability testing of a portfolio assessment tool for postgraduate family medicine training in South Africa

2013 ◽  
Vol 5 (1) ◽  
Author(s):  
Louis Jenkins ◽  
Bob Mash ◽  
Anselme Derese
Keyword(s):  
South Africa ◽  
Family Medicine ◽  
Assessment Tool ◽  
Portfolio Assessment ◽  
Validity And Reliability ◽  
Rater Reliability ◽  
Retest Reliability ◽  
Competency Based ◽  
The Difference ◽  
Test Retest Reliability

Background: Competency-based education and the validity and reliability of workplace-based assessment of postgraduate trainees have received increasing attention worldwide.Family medicine was recognised as a speciality in South Africa six years ago and a satisfactory portfolio of learning is a prerequisite to sit the national exit exam. A massive scaling up of the number of family physicians is needed in order to meet the health needs of the country.Aim: The aim of this study was to develop a reliable, robust and feasible portfolio assessment tool (PAT) for South Africa.Methods: Six raters each rated nine portfolios from the Stellenbosch University programme, using the PAT, to test for inter-rater reliability. This rating was repeated three months later to determine test–retest reliability. Following initial analysis and feedback the PAT was modified and the inter-rater reliability again assessed on nine new portfolios. An acceptable intra-classcorrelation was considered to be > 0.80.Results: The total score was found to be reliable, with a coefficient of 0.92. For test–retest reliability, the difference in mean total score was 1.7%, which was not statistically significant. Amongst the subsections, only assessment of the educational meetings and the logbook showed reliability coefficients > 0.80.Conclusion: This was the first attempt to develop a reliable, robust and feasible national portfolio assessment tool to assess postgraduate family medicine training in the South African context. The tool was reliable for the total score, but the low reliability of several sections in the PAT helped us to develop 12 recommendations regarding the use of the portfolio, the design of the PAT and the training of raters.

A Novel Tool for the Assessment of Sport Climbers’ Movement Performance

2020 ◽  
Vol 15 (6) ◽  
pp. 795-800 ◽  
Author(s):  
Nicola Taylor ◽  
David Giles ◽  
Micha Panáčková ◽  
James Mitchell ◽  
Joel Chidley ◽  
...  
Keyword(s):  
Performance Assessment ◽  
Assessment Tool ◽  
Intraclass Correlation ◽  
Validity And Reliability ◽  
Consultation Process ◽  
Retest Reliability ◽  
Performance Factors ◽  
Sport Climbing ◽  
Movement Performance ◽  
Test Retest Reliability

Purpose: To assess the validity and reliability of a novel movement-performance assessment tool for climbing/sport climbing. Methods: First, salient climbing movement-performance factors were identified through an iterative consultation process with 10 expert climbing coaches; the resulting Climber’s Movement Performance Assessment Tool (CM-PAT) contained 14 items in 5 categories. Second, 61 intermediate to advanced climbers ascended a single route, which was video recorded. Subsequently, 4 experienced (>10 y coaching) coaches used the CM-PAT to observe and score the climbers’ performance. Interrater reliability and comparisons with existing measures of climbing performance (6-mo self-reported ability, success and failure, climbing pace [m·min−1], and geometric entropy) were made. Results: Intraclass correlation coefficient (2,k) for the 4 raters demonstrated excellent reliability (>.81) between observers and good to excellent test–retest reliability (.71–.91). Pearson correlations between self-reported ability and CM-PAT scores explained 61% of the variance in self-reported climbing performance compared with 16% for geometric entropy and 52% for climbing pace. Considering differences in successful and unsuccessful climbers, the CM-PAT (P < .0005; d = 2.14), geometric entropy (P = .014; d = 0.67), and pace (P < .0005; d = 1.88) were able to differentiate between groups. Conclusions: The CM-PAT is the first sport climbing performance observational instrument to be developed through a thorough iterative process with expert coaches. Excellent interrater and test–retest reliability and excellent agreement with self-reported ability and with existing quantitative measures of performance support its recommendation for use in coaching and research contexts. Notably, a key advantage over existing measures is the identification of coachable elements of performance.

scholarly journals Psychometric evaluation of the Chinese version of the Person-Centred Care Assessment Tool

BMJ Open ◽  
2020 ◽  
Vol 10 (7) ◽  
pp. e031580
Author(s):  
Cai Le ◽  
Ke Ma ◽  
Pingfen Tang ◽  
David Edvardsson ◽  
Lina Behm ◽  
...  
Keyword(s):  
Assessment Tool ◽  
Intraclass Correlation ◽  
Psychometric Evaluation ◽  
Quality Of Healthcare ◽  
Hospital Staff ◽  
Chinese Version ◽  
Validity And Reliability ◽  
Retest Reliability ◽  
Care Assessment ◽  
Test Retest Reliability

ObjectiveThis study aims to test a Chinese cross-cultural adaptation of the English version of the Person-Centred Care Assessment Tool (P-CAT) and evaluate its psychometric properties.DesignP-CAT was translated/back-translated using established procedures before the psychometric evaluation of the Chinese version was made.SettingTwo hospitals covering urban and suburban areas of Kunming in the Yunnan province of China.Participants152 female hospital staff completed the survey.Main outcome measure(s)Construct validity and reliability, including internal consistency and test–retest reliability, were assessed among a sample of hospital staff.ResultsThe factor analysis resulted in a two-component solution that consisted of two subscales. The corrected item-total correlations for all of the items ranged from 0.14 to 0.44, with six items not meeting the cut-off level for item-total correlation (>0.3). The Chinese P-CAT demonstrated strong reliability, with a Cronbach’s alpha of 0.91–0.94 for the scales and a test–retest reliability coefficient of 0.88 for the overall scale scores. The intraclass correlation was 0.92 (95% CI 0.90 to 0.95).ConclusionP-CAT appears to be a promising measure for evaluating staff perceptions of person-centredness in Chinese hospital environments. The results show that P-CAT can be a useful tool for improving the quality of healthcare in terms of person-centred care in the Chinese context.

scholarly journals Validity and Reliability of the Korean Version of the Apathy Evaluation Scale Short form for Patients with Dementia

2020 ◽  
Vol 27 (2) ◽  
pp. 164-175
Author(s):  
Young-Rim Choi ◽  
Ye-Na Lee ◽  
Eunhye Jeong ◽  
Sung Ok Chang
Keyword(s):  
Factor Analysis ◽  
Short Form ◽  
Republic Of Korea ◽  
World Health ◽  
Validity And Reliability ◽  
Rater Reliability ◽  
Retest Reliability ◽  
Korean Version ◽  
The Republic ◽  
Test Retest Reliability

Purpose: The purpose of this study was to evaluate the validity and reliability of the Korean version of the Apathy Evaluation Scale Short Form (K AES-10) for people with dementia. Methods: Participants in this study were 198 patients with dementia resident in two hospitals in the Republic of Korea. The AES-10 was translated into Korean through the translation and adaptation of instruments process developed by the World Health Organization. This process was carried out by three bilingual nursing professionals. Validity and reliability, such as construct validity, convergent validity, Cronbach's ⍺, inter-rater reliability, and test-retest reliability, were evaluated. IBM SPSS Statistics 25.0 and AMOS 26.0 programs were used for the data analysis. Results: A uni-dimensional model was proposed in the explanatory factor analysis; a single factor accounted for 68.0% of total variances. The confirmatory factor analysis supported the validity of the Korean version of the AES-10. The scale had a Cronbach's ⍺ of .95, inter-rater reliability of .66 (<i>p</i><.001), and a test-retest reliability of .81 (<i>p</i><.001). Conclusion: The findings show that the Korean version of the AES-10 is both valid and reliable for measuring apathy among people diagnosed with dementia in the Republic of Korea.

scholarly journals Assessing the Validity and Reliability of the Persian Version of the Interpersonal Problem Solving Skills Assessment Tool in Schizophrenia

2021 ◽  
Vol 22 (1) ◽  
pp. 118-131
Author(s):  
Milad Abolhasani ◽  
◽  
Ashraf Karbalaee Nouri ◽  
Enayatollah Bakhshi ◽  
Milad Abolhasani ◽  
...  
Keyword(s):  
Content Validity ◽  
Face Validity ◽  
Validity And Reliability ◽  
Alpha Coefficient ◽  
Rater Reliability ◽  
Problem Solving Skills ◽  
Retest Reliability ◽  
Persian Version ◽  
Interpersonal Problem Solving ◽  
Test Retest Reliability

Objective: This study aimed to translate the Assessment of Interpersonal Problem-Solving Skills (AIPSS) into Persian and to evaluate the validity and reliability of the Persian version of AIPSS to use for adults with schizophrenia. Materials & Methods: In this methodological study, the translation process was performed according to the International Quality of Life Assessment (IQOLA) protocol. The face validity of the translated AIPSS was determined based on the opinions of experts and The Content Validity Index (CVI) and Content Validity Ratio (CVR) were also calculated for each item. The Persian version of the test was performed on 52 patients with schizophrenia disorders at Tehran’s Razi Mental Hospital; they were selected using a convenience sampling method. Cronbach’s alpha coefficient was used to evaluate internal consistency. Inter-rater reliability was determined by the Intraclass Correlation Coefficient (ICC). A retest was complete on 15 patients with 2 weeks interval and ICC was used to determine the test-retest reliability. Results: Face validity was confirmed by the experts’ opinions. The Content Validity Ratio (CVR) and the Content Validity Index (CVI) were equal to one for all scenes. Cronbach's alpha coefficient for all scales was ranged between 0.511 and 0.821. The ICC in all scales were more than 0.98 for inter-rater reliability. In calculating test-retest reliability, the ICC for all scales ranged 0.733-0.893. Conclusion: Results show that the Persian version of AIPSS has acceptable face validity, content validity, internal consistency, inter-rater reliability test-retest reliability. Therefore, this instrument can be used in clinical fields and research studies to assess the social skills of Iranian patients with schizophrenia.

scholarly journals Validity and reliability of the English and translated Chinese versions of the Integrated Palliative care Outcome Scale (IPOS) in Singapore

2020 ◽  
Author(s):  
Victoria Long ◽  
Yin Bun Cheung ◽  
Debra Qu ◽  
Katherine Lim ◽  
Guozhang Lee ◽  
...  
Keyword(s):  
Correlation Coefficient ◽  
Internal Consistency ◽  
Validity And Reliability ◽  
Cronbach’S Alpha ◽  
Rater Reliability ◽  
Intra Class Correlation ◽  
Retest Reliability ◽  
Cronbach's Alpha ◽  
Test Retest Reliability ◽  
Intra Class Correlation Coefficient

Abstract Context: Measurement of patient-centred outcomes enables clinicians to focus on patient and family priorities and enables quality of palliative care to be assessed.Objectives: This study aimed to evaluate the validity and reliability of the English and translated Chinese IPOS among advanced cancer patients in Singapore.Methods: IPOS was forward and backward translated from English into Chinese. Structural validity was assessed by confirmatory factor analysis; known-group validity by comparing inpatients and community patients; construct validity by correlating IPOS with Edmonton Symptom Assessment System-revised (ESAS-r) and Functional Assessment of Cancer Therapy–General (FACT-G); internal consistency by Cronbach’s alpha; inter-rater reliability between patient and staff responses; test-retest reliability of patient responses between two timepoints.Results: 111 English-responding and 109 Chinese-responding patients participated. The three-factor structure (Physical Symptoms, Emotional Symptoms and Communication and Practical Issues) was confirmed with Comparative Fit Index and Tucker-Lewis-Index > 0.9 and Root Mean Square Error of Approximation < 0.08. Inpatients scored higher than outpatients as hypothesised. Construct validity (Pearson’s correlation coefficient, r≥|0.608|) was shown between the related subscales of IPOS and FACT-G and ESAS-r. Internal consistency was confirmed for total and subscale scores (Cronbach's alpha ≥ 0.84), except for the Communication and Practical Issues subscale (Cronbach’s alpha = 0.29–0.65). Inter-rater reliability (Intra-class correlation coefficient [ICC] ≤ 0.43) between patient and staff responses was insufficient. Test-retest reliability was confirmed with Intra-class correlation coefficient ICC = 0.80 (English) and 0.88 (Chinese) for IPOS Total.Conclusion: IPOS in English and Chinese showed good validity, good internal consistency, and good test-retest reliability, except for the Communication and Practical Issues subscale. There was poor inter-rater reliability between patients and staff.

scholarly journals Development and reliability of the Korean version of the Feeding Abilities Assessment

2019 ◽  
Vol 32 (1) ◽  
pp. 69-74
Author(s):  
Seul Gi Koo ◽  
Hae Yean Park ◽  
Jongbae Kim ◽  
Areum Han
Keyword(s):  
Correlation Coefficient ◽  
Internal Consistency ◽  
Content Validity ◽  
Assessment Tool ◽  
High Reliability ◽  
Intraclass Correlation ◽  
Rater Reliability ◽  
Retest Reliability ◽  
Korean Version ◽  
Test Retest Reliability

Objective The purpose of this study is to introduce a standardised assessment tool by verifying the reliability of the translated Korean version of the Feeding Abilities Assessment (K-FAA), which was developed to suit Korean culture. Methods The research subjects were 65 patients with dementia living in nursing homes. The K-FAA was completed by verifying the suitability of translation and reverse translation. The validity of the K-FAA was established through content validity, while its reliability was analysed based on internal consistency reliability for the items, test–retest reliability and inter-rater reliability. Results The content validity index determined, based on the assessment of professors, occupational therapists, and nurses, was more than .70. Cronbach’s α was more than .929, showing good internal consistency. A test–retest reliability of .884 was derived using Pearson’s correlation coefficient (p < .01), and an inter-rater reliability of .800 was derived using the kappa coefficients; intraclass correlation coefficient was .897, which also indicated good reliability. Conclusion The K-FAA was modified to fit the Korean domestic situation, and this assessment had high reliability. Therefore, K-FAA can evaluate the feeding ability of patients with dementia. Future studies should focus on providing evidence-based data to maintain or supplement the feeding ability of patients with dementia in Korea.

scholarly journals Inter-rater reliability and test-retest reliability of the Performance and Fitness (PERF-FIT) test battery for children: a test for motor skill related fitness

2021 ◽  
Vol 21 (1) ◽  
Author(s):  
Bouwien C. M. Smits-Engelsman ◽  
Eline Smit ◽  
Rosemary Xorlanyo Doe-Asinyo ◽  
Stella Elikplim Lawerteh ◽  
Wendy Aertssen ◽  
...  
Keyword(s):  
Low Income ◽  
Motor Skill ◽  
Assessment Tool ◽  
Test Battery ◽  
Good Test ◽  
Rater Reliability ◽  
Retest Reliability ◽  
Reliability Indices ◽  
Significant Difference ◽  
Test Retest Reliability

Abstract Background The Performance and Fitness (PERF-FIT) test battery for children is a recently developed, valid assessment tool for measuring motor skill-related physical fitness in 5 to 12-year-old children living in low-income settings. The aim of this study was to determine: (1) inter-rater reliability and (2) test-retest reliability of the PERF-FIT in children from 3 different countries (Ghana, South Africa and the Netherlands). Method For inter-rater reliability 29 children, (16 boys and 13 girls, 6–10 years) were scored by 2 raters simultaneously. For test–retest reliability 72 children, (33 boys and 39 girls, 5–12 years) performed the test twice, minimally 1 week and maximally 2 weeks apart. Relative and absolute reliability indices were calculated. ANOVA was used to examine differences between the three assessor teams in the three countries. Results The PERF-FIT demonstrated excellent inter-rater reliability (ICC, 0.99) and good test-retest reliability (ICC, ≥ 0.80) for 11 of the 12 tasks, with a poor ICC for the Jumping item, due to low spread in values. A significant difference between first and second test occasion was present on half of the items, but the differences were small (Cohen’s d 0.01–0.17), except for Stepping, Side jump and Bouncing and Catching (Cohen’s d 0.34, 0.41 and 0.33, respectively). Overall, measurement error, Limits of Agreement and Coefficient of Variation had acceptable levels to support clinical use. No systematic dissimilarities in error were found between first and second measurement between the three countries but for one item (Overhead throw). Conclusions The PERF-FIT can reliably measure motor skill related fitness in 5 to 12-year-old children in different settings and help clinicians monitor levels of fundamental motor skills (throwing, bouncing, catching, jumping, hopping and balance), power and agility.

scholarly journals Validity and Reliability of a Short Diet Questionnaire to Estimate Dietary Intake in Older Adults in a Subsample of the Canadian Longitudinal Study on Aging

Nutrients ◽  
2018 ◽  
Vol 10 (10) ◽  
pp. 1522 ◽  
Author(s):  
Anne Gilsing ◽  
Alexandra Mayhew ◽  
Hélène Payette ◽  
Bryna Shatenstein ◽  
Sharon Kirkpatrick ◽  
...  
Keyword(s):  
Longitudinal Study ◽  
Assessment Tool ◽  
Dietary Assessment ◽  
Intraclass Correlation ◽  
Correlation Coefficients ◽  
Food Groups ◽  
Validity And Reliability ◽  
Retest Reliability ◽  
System Usability Scale ◽  
Test Retest Reliability

This study assessed test-retest reliability and relative validity of the Short Diet Questionnaire (SDQ) and usability of an online 24 h recall among 232 participants (62 years ± 9.1; 49.6% female) from the Canadian Longitudinal Study on Aging (CLSA). Participants were asked to complete four 24 h dietary recalls (24HRs) using the Automated Self-Administered 24-h Dietary Assessment Tool (ASA24-Canada-2014), two SDQ administrations (prior to recalls one and four), and the System Usability Scale (SUS) for ASA24. For the SDQ administrations, Intraclass Correlation Coefficients ranged from 0.49 to 0.57 for nutrients and 0.35 to 0.72 for food groups. Mean intakes estimated from the SDQ were lower compared than those from the 24HRs. For nutrients, correlation coefficients were highest for fiber, calcium, and vitamin D (45–64 years: 0.59, 0.50, 0.51; >65 years: 0.29, 0.38, 0.49, p < 0.01); Kappas ranged from 0.14 to 0.37 in those 45–64 years and 0.17 to 0.32 in participants >65 years. Among the 70% who completed all recalls independently, the SUS indicated poor usability, though the majority reported feeling confident using ASA24. Overall, the SDQ captures intake with varying test-retest reliability and accuracy by nutrient and age. Further research is needed to inform use of a more comprehensive dietary measure in the CLSA.

scholarly journals Inter-Rater Agreement and Test-Retest Reliability of the Performance and Fitness (PERF-FIT) Test Battery for Children: A Test for Motor Skill Related Fitness.

2020 ◽  
Author(s):  
Bouwien Smits-Engelsman ◽  
Eline Smit ◽  
Rosemary Xorlanyo Doe-Asinyo ◽  
Stella Elikplim Lawerteh ◽  
Wendy Aertssen ◽  
...  
Keyword(s):  
Low Income ◽  
Motor Skill ◽  
Assessment Tool ◽  
Test Battery ◽  
Rater Agreement ◽  
Good Test ◽  
Rater Reliability ◽  
Retest Reliability ◽  
Reliability Indices ◽  
Test Retest Reliability

Abstract Background: The Performance and Fitness (PERF-FIT) test battery for children is a recently developed, valid assessment tool for measuring motor skill-related physical fitness in 5 to 12-year-old children living in low-income settings. The aim of this study was to determine: (1) inter-rater agreement and (2) test-retest reliability of the PERF-FIT in children from 3 different countries (Ghana, South Africa and the Netherlands). Method: For inter-rater reliability 29 children, (16 boys and 13 girls, 6-10 years) were scored by 2 raters simultaneously. For test–retest reliability 72 children, (33 boys and 39 girls, 5-12 years) performed the test twice, minimally one week and maximally two weeks apart. Relative and absolute reliability indices were calculated. ANOVA was used to examine differences between the three assessor teams in the three countries.Results: The PERF-FIT demonstrated excellent inter-rater reliability (ICC, 0.99) and good test-retest reliability (ICC, ≥ 0.80) for 11 of the 12 tasks. a poor ICC was found for the Jumping item only, due to low spread in values. Overall, measurement error, Limits of Agreement and Coefficient of Variation had acceptable levels to support clinical use. No systematic differences were found between first and second measurement between the three countries but for one item (Overhead throw). Conclusions: The PERF-FIT can reliably measure motor skill related fitness in 5 to 12-year-old children in different settings and help clinicians monitor levels of power and agility, and fundamental motor skills (throwing, bouncing, catching, jumping, hopping and balance).

scholarly journals Validity and reliability of the English and translated Chinese versions of the Integrated Palliative care Outcome Scale (IPOS) in Singapore

2021 ◽  
Vol 20 (1) ◽  
Author(s):  
Victoria Jane En Long ◽  
Yin Bun Cheung ◽  
Debra Qu ◽  
Katherine Lim ◽  
Guozhang Lee ◽  
...  
Keyword(s):  
Palliative Care ◽  
Correlation Coefficient ◽  
Internal Consistency ◽  
Validity And Reliability ◽  
Cronbach’S Alpha ◽  
Rater Reliability ◽  
Retest Reliability ◽  
Cronbach's Alpha ◽  
Palliative Care Outcome Scale ◽  
Test Retest Reliability

Abstract Context Measurement of patient-centred outcomes enables clinicians to focus on patient and family priorities and enables quality of palliative care to be assessed. Objectives This study aimed to evaluate the validity and reliability of the English and translated Chinese versions of the Integrated Palliative care Outcome Scale (IPOS) among advanced cancer patients in Singapore. Methods IPOS was forward and backward translated from English into Chinese. Structural validity was assessed by confirmatory factor analysis; known-group validity by comparing inpatients and community patients; construct validity by correlating IPOS with Edmonton Symptom Assessment System-revised (ESAS-r) and Functional Assessment of Cancer Therapy–General (FACT-G); internal consistency by Cronbach’s alpha; inter-rater reliability between patient and staff responses; test-retest reliability of patient responses between two timepoints. Results One hundred eleven English-responding and 109 Chinese-responding patients participated. The three-factor structure (Physical Symptoms, Emotional Symptoms and Communication and Practical Issues) was confirmed with Comparative Fit Index and Tucker-Lewis-Index > 0.9 and Root Mean Square Error of Approximation < 0.08. Inpatients scored higher than outpatients as hypothesised. Construct validity (Pearson’s correlation coefficient, r ≥ |0.608|) was shown between the related subscales of IPOS and FACT-G and ESAS-r. Internal consistency was confirmed for total and subscale scores (Cronbach’s alpha≥0.84), except for the Communication and Practical Issues subscale (Cronbach’s alpha = 0.29–0.65). Inter-rater reliability (Intra-class correlation coefficient [ICC] ≤ 0.43) between patient and staff responses was insufficient. Test-retest reliability was confirmed with Intra-class correlation coefficient ICC = 0.80 (English) and 0.88 (Chinese) for IPOS Total. Conclusion IPOS in English and Chinese showed good validity, good internal consistency, and good test-retest reliability, except for the Communication and Practical Issues subscale. There was poor inter-rater reliability between patients and staff.

Sign in / Sign up

Export Citation Format

Share Document