The Refractive Thinker®: Vol V: Strategy in Innovation; Ch 7: Assessing an Innovation: Student Outco

The Refractive Thinker: Vol V: Strategy in Innovation; Ch 7: Assessing an Innovation: Student Outcomes from Master's Degree Programs May 26,
Table of contents

Managing change in postgraduate medical education: The structure and content of postgraduate medical training in the UK are undergoing a major modernisation process. This will have a significant impact on the role of hospital consultants with educative responsibilities. The University of Manchester has conducted a qualitative study of the culture of medical education in the SHO grade. The study includes an exploration of hospital consultants' perceptions of the modernisation process, and its impact on their role.

Interviews were conducted with 28 consultants with varying education-related duties. There is widespread uncertainty about the nature of change to postgraduate medical education, particularly amongst front-line clinical educators with no additional education-management role. Even those with such roles e.

Medical Directors, Clinical and College Tutors display considerable levels of anxiety and confusion about the modernisation process. There is a strong sense that educational supervisors should have dedicated time to plan and deliver training. This should be supported with appropriate and sustained training for their educational role.

Hospital consultants are concerned about the impact of modernisation in postgraduate medical education on their own role. This is understandable given the many pressures on their time, although much of their uncertainty is a result of limited awareness about change combined with communication deficiencies from Government downwards. Development of the regional and local infrastructure that supports medical education is required. The majority of consultants are committed to the education of doctors in training, bur greater recognition and support of their role is necessary if goodwill is to be maintained.

Does portfolio contribute to the development of reflective skills? Questions about the utility of a portfolio as a method for the development and assessment of reflective skills are frequently raised in the literature. However, the literature shows few studies which report answers to these questions. The purpose of this presentation is to give more insight in the practical use of a reflective portfolio in medical undergraduate education. In our research, we were specifically interested in the conditions that promote the development of reflective skills.

We have interviewed teachers about their experiences with coaching and assessing students in keeping a portfolio. While doing this, we focussed on the teachers' perceptions of portfolio and reflection. We used grounded theory methodology to explore teacher perceptions in an open and broad way.

All mentors in our study agreed that the process of compiling and discussing a portfolio contributes to the development of reflective ability. The thinking activities that a student undertake while compiling his portfolio are essential for this effect. Factors which are decisive for the successful use of portfolio are: Standardized patients in a catalan medical school: Standardized patient, undergraduate, competencies Authors: Universitat Rovira i Virgili de Tarragona Summary: In the Faculty of Medicine of Universitat Rovira i Virgili started a project on competencies learning.

The participants in the project were students of the last courses of Medicine. Cases with standardized patients were the formative instrument. The competence components analysed were: An opinion questionnaire was undertaken by 50 participants. Through the questionnaire 18 different areas were evaluated, using a Likert scale, relating to logistics, organization, contents and learning impact. Results proved this project is feasible and well accepted, and is a good method to improve the learning process of medical students. A survey of cheating on tests among Catholic University of Chile medical students Keywords: Cheating is an unethical behavior.

In medical schools, this represents a recurrent problem, with a reported frequency close to 60 percent. To investigate cheating on tests, an anonymous questionnaire was distributed among 97 fourth-year medical students. Students were asked whether they have seen other students cheat and their attitudes about cheating on: They also were questioned on the reasons for, consequences of, and deterrents to cheating. Slightly more than half of the students expressed that cheating is not related to inappropriate behaviors with patient care.

Interestingly, a high percentage of students were in agreement in their responses and attitudes to cheating. It is remarkable that students perceive test cheating as unethical and having negative consequences. This constitutes the ground basis to develop a nurturing culture of Medicine, enhancing honesty, integrity, and professionalism. Acceptance of peer review by doctors in a Canadian community hospital was assessed using a post-review survey. In this program, practising doctors were systematically reviewed in the hospital using a multimodal review process.

They then filled out a survey regarding their impression and degree of satisfaction with the review. Possible reasons for this are discussed and proposals presented to gain acceptance, even with sceptical groups of doctors. The feasibility, reliability, and construct validity of a program director's supervisor's evaluation form for medical school graduates Keywords: We determined the feasibility, reliability and construct validity of a supervisor's survey for graduates of our institution. We prospectively sought feedback from Program Directors for our graduates during their first post-graduate year.

Surveys were sent out once yearly with up to 2 additional mailings. For this study, we reviewed all completed Program Director Evaluation Form surveys from Interns are rated on a scale in each of 18 items. Mean scores per item were calculated. Feasibility was estimated by survey response rate. Internal consistency was determined by calculating Cronbach's alpha and with exploratory factor analysis with varimax rotations.

Assuming that our graduates would show a spectrum of proficiency when compared to graduates from other schools, construct validity was determined by analyzing the range of scores, including the percent of scores below acceptable level 2 or 1, see below table. Cronbach's alpha was 0. Mean scores across items were 3. Our survey was feasible and had high internal consistency.

Factor analysis revealed two complimentary domains knowledge and professionalism , supporting the content validity. Analysis of range of scores supports the form's construct validity. The continual education has been necessarily accepted in the world, in this direction, the instructional needs and determining the priority of continuing education programs prepare the possibility of obtaining the desired quality. The forth-especial target of his research was the perception of the most important motivation to participate the program have evaluated, orderly, the review of information 2.

This programs have been completely successful ones, but it is recommended that we could obtain the further qualitative promotion of instructions by presenting the new scientific appreciative subjects, using the various methods in performing the instructional programs, also attending to coincidence of contents with occupational needs of GP and making reasons on them by setting forth the important questions. The effectiveness comparison of two educational methods on academic advisors Knowledge, Attitude, and Practice Keywords: Isfahan University of Medical Sciences Summary: The purpose of this study was to investigate the effect of two educational methods on workshop and having educational material the level of knowledge, attitude, and practice of Hamadan University of medical sciences.

In this study, participated in the pre-test Section before the intervention and participated in the experimental program. The AA in experimental program randomly divided in two groups. Data collection for knowledge, attitude, and practice was conducted by the valid and reliable questionnaires before educational program and after one academic semester prior to the program. This differences existed only between Group1 with Group 2 and pre-test group. This study shown that developing educational workshop program based on roll playing, group discussion, and group working and interaction could be affected to the behavior and attitude that result improving their skills and abilities.

Finding of this research may be able to be beneficial for developing educational program for AA of universities. Assessment of the intra-service rotations in anaesthesiology and reanimation: Hospital Germans Trias i Pujol Summary: Assessment of the intra-service rotations in Anaesthesiology and Reanimation: Hospital Germans Trias i Pujol. In order to improve the supervision of the trainee rotations, the anaesthetist in charge of each area will complete an evaluation form.

The change in methodology will improve the personal performance of the trainee. Improve the final result, reaching the stated objectives more successfully, through the identification of the strengths and weaknesses that need to be improved. Once the consultant has defined the objectives of their area, the evaluation form is completed at the halfway point and at the end of the period, both by the consultant and the resident independently. Both evaluation forms are compared and contrasted establishing the points to be improved and comparing the progress of the learner.

The evaluation include seven aptitude and five attitude criteria. Both are conducted in a qualitative way with a descriptive, non-numerical scale. To determine whether Rheumatologists adopt and adhere to the use of personal learning projects PLPs as a method of continuing professional development CPD and maintenance of certification following the introduction to the concept of PLPs and their utilization within a review workshop.

Rheumatologists attending a 2 day continuing education workshop were involved in a 30 minute interactive lecture outlining the concept of learning portfolios and how to use a PLP as a method of continuing education. Attending Rheumatologists filled out a pre and post-workshop evaluation questionnaire followed by the completion of a 3 month follow-up questionnaire.

Average awareness of CPD methods was 7. In the average number of PLP was reported at 5. Time constraints still remained the number one barrier for personal involvement with CPD, while the use of paper diaries remained the favoured PLP method of recording. There was an increase in Rheumatologists awareness and application of PLPs, which was sustained at the 3 month period.

Patient Satsifaction; Rheumatology Clinic Authors: Patients who had a diagnosis of chronic arthritis and had been seen in clinic on at least three prior occasions were asked to complete the Leeds Patient Satisfaction Questionnaire LPSQ once they had registered for the appointment. The attending rheumatologist and other clinic medical staff were not made aware of which patients had completed the questionnaire. All questionnaires were scored according to the guidelines of the Leeds Satisfaction Questionnaire, and were checked by two independent investigations to minimize arithmetical errors.

Descriptive statistics were calculated. Eighty-seven patients completed the questionnaire. The mean normalized Overall Satisfaction score, combining satisfaction rates across all subgroups, was The overall mean scores of the subgroups were Giving of information Empathy with the patient technical quality of competence Attitude towards the patient Access to the service and continuity of care General Satisfaction Conclusions: Patients appear to be very satisfied with the care they receive. Areas that could be improved in the future include patient education regarding clinic services, waiting times, and receiving urgent consultation if needed.

Residents play an important role in teaching medical students and there is a large number of contact hours among them. So developing teaching skills, being familiar with innovative teaching styles, knowing how to increase the educational efficacy, providing an educational spiral are the necessities of Residency Programs. To determine the effect of teaching skills workshop on the teaching role of residents. This is a Quasi- experimental study. A self-administered questionnaire was distributed among interns of pediatrics and internal medicine wards in 2 universities.

Then the randomized selected residents participated in an 8 hours workshop. There was a significant difference between the mean group ratings for all of the teaching skills characteristics in both universities. The overall teaching skills in Iran University and 5 categories of teaching skills in Kermanshah University except "Giving feedback", and "Professional characteristics", were increased. Overall teaching effectiveness of residents was increased after the workshop.

Increasing scores of skills after the workshop, reveal that training programs and teaching skills courses for residents should be performed as formal instructional residency programs. A needs assessment should be done to develop such a course. The dual roles of the global rating scale on a 30 station Objective Structured Clinical Examination for chiropractic radiologists: OSCE, borderline method, global rating scale Authors: The borderline candidate method was used to set the minimum performance level MPL and compared to the previously used modified Ebel method for ease of use and examiner confidence in the MPL.

Reliability Alpha for station totals, global scales, and combined were high. The correlation Pearson's to the sum of global scores and total checklist scores was. Feedback from examiners revealed that they unanimously supported the continued use of the global rating scale. The main reasons cited were 1 that they felt that the detailed checklist advantaged weaker candidates and that global rating scale allowed the examiners to award strong candidates who may not have got all the checklist points, and punish weaker candidates who got most of the checklist points but were very disorganized in their approach, and 2 they felt more confident in the MPL set by the global rating scale in comparison to the Ebel method.

A pilot project introduced the use of a videotaped structured interview to reduce costs in the admission process. Candidates were videotaped while being interviewed by a panel. The interviews were combined with letters of reference, letters of intent, a questionnaire, and college transcripts and scored by the panel and 4 Fellows spread across Canada. Those scoring the video-tapes responded to 2 questionnaires, how helpful each of the 5 processes were in evaluating the candidates, and satisfaction with the use of video-taped interviews.

Internal reliability of the admission instrument was estimated Cronbach's Alpha 0. An exploratory factor analysis was performed to determine what traits were being measured collapsing the 35 rating scales to 7 factors. The raters found the reference letters, the letter of intent and the interview most helpful. The contribution of standardized patients to error variance in candidate scores on a high stakes objective structured clinical examination Keywords: University of Calgary Summary: The purpose of this research project was to determine if differences in standardized patient SP performance contribute to error variance in candidate scores.

There were 50 examiners and SPs involved over five parallel tracks. The standard deviations of the logit measures for candidates, examiners, and SPs were. Some SPs more readily yield information to candidates than others. Physician performance, structural equation modeling, multi source feedback Authors: To empirically test a theoretical model of physician performance. Performance data for physicians derived from four sources self, patient, peer and co-worker ratings were tested within a structural equation modeling.

The physicians, selected using proportionate random sampling and stratified by urban and rural communities, had all been registered with the licensing body for more than five years and were generalist physicians from the disciplines of family medicine, obstetrics and gynecology, internal medicine, and pediatrics.

A four-factor structural model of physician performance was based on the four data sources and fit to the data using structural equation modeling methods. Conclusions - The model provides a multi-source and multi-dimensional approach to assessing physician performance, with peers, patients, and co-worker assessing the physician from their own unique perspective they also concur on several dimensions.

International medical graduates IMGs may provide services in Canada under a 'defined' license prior to the successful completion of Medical Council of Canada examinations. Study tested feasibility and psychometrics of a multi source degree evaluation. Mean ratings on all surveys were between 1 and 5. The mean self rating was marginally higher than the mean medical colleague 4. Most items performed well. Multisource evaluation is feasible as a monitoring tool. Parent evaluations of paediatric interview skills Keywords: University of Adelaide Summary: Concerns regarding the reliability of patient evaluations have limited their use in medical student learning.

A program was developed to obtain parent evaluations of student paediatric interview skills for feedback and to identify students at risk of poor performance in summative assessments. Students received their individual parent scores and de-identified class mean scores as feedback, and participants were surveyed regarding the program.

Parent evaluation scores were compared with student performance in faculty assessments of clinical interview skills. Parents supported the program and participating students valued parent feedback. Obtaining one 'Lowscore' was associated with increased risk of obtaining a faculty assessment score below the class mean OR 4. Parent evaluations provided useful feedback to students and identified one group of students at increased risk of weaker performance in summative assessments. Developing and Implementing an Educational Assessment program: Assessment program, Student educational outcomes Authors: An educational assessment program was recently developed and implemented at the Dn.

The objective of the program is to improve student educational outcomes. The educational assessment program has three major steps: Department faculty establish educational objectives based on institutional mission; determine and implement assessment procedures; and measure educational outcomes. Results of assessment procedures are used to improve educational outcomes. This presentation will discuss the challenges presented along each of the above steps.

In addition, methods of encouraging "faculty ownership" of the program and ways to encourage students to take assessment instruments seriously will be discussed. Mohammad Ali Sadighi Gilani, Dr. Ali Akbar Zeinaloo Institution: School of public health Tehran university of medical sciences and Educational development center. Educational assessment is the process of setting and providing evaluative-descriptive information on the value and significance of educational objectives, operation and the results, in order to direct the decisions, responses and information.

The purpose for decision-making, aimed in this definition is to select the best out of possible choices by using information related to efficiency and accuracy of each choice. Responding means the ability to offer a persuading report on educational activities, their reasons, expenditure and effects. Supervision on an educational program or activity is to answer to the following questions: The research general objective is to evaluate educational activities theoretical and practical of professor from learner's viewpoint during an educational semester, and to submit learner's impression in a confidential improve and promote the quality of education.

MGT Strategic Innovation and Ideas

Standard setting methods for OSCEs are controversial. The stations and cases were identical. All candidates were naive to the stations no repeating candidates. Examiners received training on the setting of MPLs prior to each examination. Examiners set the MPL for the station to which they were assigned. Between cycles for each examination noon and end of day , examiners were asked to identify the must know items on the 25 item rating form used for their station.

The MPLs for the examinations were a summation of the must know items. This study found on average a mark difference between the two standard setting procedures 3. This investigation provided evidence that examiner MPL decisions setting on OSCE are not stable from administration to administration and that alternative, more stable, methods should be used. While standardised tests of clinical skills, such as OSCEs Objective Structured Clinical Examinations , have become widely used to assess clinical competence, the method of setting the pass mark varies greatly and there is no agreed 'best' standard setting process.

There is a need for more quantitative evidence in this field. In our study, we compared the pass marks set for six OSCE stations using the Angoff method, for a graduating level examination, across five medical schools in the UK. The pass marks set for each of the six OSCE stations at the five medical schools differed significantly. In-depth analysis of the judges' scores on individual stations at each medical school and comparison of results across the schools will be presented and discussed.

These results have serious implications for the outcomes of graduating examinations, in that students with the same level of competency would pass at one medical school, but would fail at another, even when the test is identical. At Barts and the London School of Medicine, students at the end of the 3rd Year are required to pass a 20 station OSCE covering basic clinical and communication skills before they are able to proceed to the 4th Year of the course. The borderline group method for setting the passing score was introduced in The final pass mark for the overall OSCE was calculated as the mean of the pass marks set for each individual station.

The OSCE was fully compensatory, as each student's overall score was the summated mean of their individual station scores. This meant that poor performances on some stations could be compensated by highly scoring performances on other stations. A second passing criterion was therefore introduced: Analysis of the effects of introducing these 2 criteria on the numbers of students passing this 3rd Year OSCE will be presented and discussed.

University of Toronto Summary: Does a recently introduced CanMEDs-based monthly in-training evaluation form produce valid results? At the end of each training month, in-training evaluation forms are completed for all Core-Medicine residents. The form contains 36 items grouped by CanMed role. Each item, except for those associated with the Professional role, requires a rating of exceeds, meets, or does not meet expectations.

With the Professional role only the latter two options are used. We conducted an analysis of the results of all monthly in-training evaluation forms returned in the academic year. Aggregate scores for each role were calculated. With the exception of the Professionalism score, the internal consistencies of the role scores were all greater than 0. Analyses of these data demonstrated growth in role scores across the 3 pgy years, consistency in role scores within pgy years; and demonstrated that role scores discriminated among residents. There were observed differences between the ratings of roles and in the ratings of competencies within roles.

Continuous Nursing Education, Resources, Authors: Continuous Nursing Education is considered as a means of proper reply to the rapid changes in health care delivery and promoting professional standards of current practice among the nurses. This study was conducted to investigate the nurses' opinion about continuous nursing education, and to determine the educational resources as well as nurses' educational needs. This descriptive cross sectional study was conducted on the nursing units in Kermanshah-Iran in The members of target population were one hundred nurses who provided the direct patient care in their units.

The responses were measured by a single-item Likert scale. A total of one hundred questionnaires were studied. They had experience as the nurses for more than ten yrs They spent not only more than 48 hours in continuous nursing education in two past years, but also 5. On the whole, the motivation for their participation, obstacles in continuous nursing education and need to this educational programme were important from their viewpoints.

Offering the best value for continuous nursing education is an important subject. In order to increase nurses' motivation and to minimize the obstacles, we should improve this educational programme by increasing the personnel numbers in the hospital, decreasing the workload, participating in the continuous nursing education programmes and giving special time for these programmes.

Faculty Performance Evaluation Keywords: Performance evaluation of medical school faculty is difficult to conduct in an unbiased, consistent professional manner and subject to criticisms of subjectivity and invalidity. The University of Health Sciences has developed a model system which objectively identifies key responsibilities for faculty in the following domains: Each domain lists a goal major end results of job , objectives key measurable achievements and performance levels. The faculty member and supervisor determine the applicable major end results of the job.

Free Membership Registration to Download

Criteria for evaluation with clearly delineated performance expectations are presented. At the end of the evaluation period the faculty member is held accountable for performance, and ratings are conducted based upon evidence presented. This presentation will discuss the theory and principles leading to the development of the form and the criteria selected. Audience participation will be encouraged through small group activities. Prepares for assigned lectures. Assessment of common errors in clinical evaluation in view points of students Keywords: School of Nursing and Midwifery Summary: In this descriptive study of nursing,midwifery, operating room students were randomly selected.

Data was gathered using a questionnaire 2 weeks after the end of the clinical courses and receiving the course scores. There were relation between viewpoints of students with their grade point average. Revise in students evaluation methods based on course objectives, instructing the instructors on student performance evaluation methods is suggested are suggested. Medical licensure, dimensionality Authors: Medical licensure examinations in the U.


  1. Vilparis ou l’otage d’une éternelle controverse: (Roman) (French Edition).
  2. Rue de la Pompe: A Satiric Urban Fantasy.
  3. Body Language (Collins Need to Know?).
  4. Bribery and Extortion: Undermining Business, Governments, and Security (Praeger Security Internation.
  5. Why Men of Color Are an Endangered Species in America.
  6. leondumoulin.nl: Dr. Joseph W. T. Pugh: Books, Biography, Blogs, Audiobooks, Kindle!

Nevertheless, the common purpose and the integration of the three exams have hardly been operationalied. To operationalize the common measurement objective, the Comprehensive Osteopathic Medical Licensing Examination COMLEX uniquely requires all its three Level exams to have a common content outline while allowing different Levels emphasizing on different aspects of practice.

This common-outline design assumes a unidimensionality of the whole exam series. Factor structure of the three COMLEX exams was studied by treating the test plan categories as hypothetically distinct tests. Disattenuated correlations among the categories were examined and a principle component analysis of the disattenuated correlations was performed. An item-level factor analysis, allowing for the detection of factors unrelated to the test plan categories, was also performed.

The dimensionality of a subset of items common in all three levels of the examination was studied in relation to the dimensionality of non-linking items. Results of this study encourage the concept that the knowledge component of the medical competence, regardless its broadness, may be considered as a single construct when it is operationalized carefully. Should visual spatial perception tests affect residency choices Keywords: Visual spatial perception VSP tests have been statistically correlated with surgical techniques.

Because of that certain residencies have looked into the applicaton of these tests as a screening tool for acceptance into their programs. We proposed that medical students, themselves, might select their residency program dependent on what they perceive to be their innate level of skill in VSP. We used as a cohort the final year medical students at our university and asked them to fill our a 30 minute abbreviated bonafide VSP test and then asked them which residencies they were applying to, as well as their gender.

Statistical analysis was performed on the data obtained from 75 medical students who were equally divided along gender lines. Using a pre-ordained cut-off point, we showed that male students fared slightly higher than the female students. After categorizing and lumping the specialties along surgical versus medical lines, it was noted that female students who had a low score rarely applied to surgical specialties. Our results show that including VSP testing as part of an armamentarium of exams may be useful in advising students about their choice of residency selection.

It should not, however, be used as a criteria of exclusion. A "Consequential Validity" Study Keywords: Student ratings of instruction are widely employed in universities generally and medical schools particularly, across Canada and the United States e. Indeed, student ratings of instruction are one of the most thoroughly studied forms of personnel evaluation.

Most previous research has focused on psychometric properties such as reliability and validity of student ratings instruments as indicators of the quality of teaching and the overall effectiveness of instruction by individual instructors. These results have often been conflicting and contradictory Arreola, ; Kulik, ; reliability is generally adequate but evidence for validity is mixed.

To investigate the degree to which student rating information is useful i. The results of the present study indicate that although there is general acceptance of the use of student ratings by both instructors and students, there is greater evidence of "consequential validity" from instructors than from students. Quality criteria for portfolio assessment of undergraduate medical students Keywords: Aim of the study: Portfolios have gained wide acceptance as a learning and assessment tool.

Yet, little research has been reported on the validity of these portfolio-assessments. The issue is whether assessors are influenced by the lay-out and writing of the portfolio when making their judgment. The main research question in this study was: For this study, portfolios from undergraduate medical students were used. Based on in-depth interviews with assessors and literature, a scoring list with fifteen quality criteria was established.

The criteria of this list could be divided into two groups: Two researchers have scored a stratificated sample of fourty portfolios, using the scoring list. The inter-raterreliability was counted with Pearson product-moment correlation. The correlation between the quality criteria and earlier judgment was examined with a regression analysis.

Interraterreliability was acceptable with an average of 0, for the fifteen criteria. The strongest predictor of the end-judgment seemed to be the quality of the reflections. This study shows that reflective skills in portfolios can be assessed in a valid way. Further research must make clear if other competencies than reflection can also be assessed in a valid manner with portfolio.

Adverse events are a common occurrence in the hospital setting. It is recognized that the incidence of adverse events can be reduced through education in key competency areas. At Gold Coast Hospital, the intern orientation program includes a number of highly interactive skill stations which are based on clinical scenarios.

These have been identified as being critical to providing safe patient care, thus reducing the incidence of adverse events. Anecdotal evidence and feedback from previous years, suggests that the educational intervention provided through the skill stations at Gold Coast Hospital improves the skill level of interns in a number of core competency areas.

These core competencies have been identified as being essential competencies for interns in the provision of safe patient care. Through the skill stations, the interns are exposed to clinical situations that address the competency area in a non-threatening, safe, interactive environment. These educational sessions are provided immediately prior to the commencement of clinical duties and responsibilities. Skill stations have been a routine part of the education training program for junior doctors at Gold Coast Hospital.

The aim of our research has been to determine the benefit of this education program and the impact of the education on the level of competency, thereby insuring improved quality of patient care. Three three-hour educational modules were developed to facilitate family physician management of patients with Alzheimer's Disease. The modules covered diagnosis and pharmacotherapy; care of patients with mild to moderate dementia and late stage dementia. Teaching was done in small groups using interactive strategies case based learning, role playing.

Participants completed pre and post course 3 months assessments as well as commitment to change statements at the end of the course with a follow-up at 3 months. To assess knowledge, comfort with management, and level of care provided before and 3 months after the modules. To assess physician adherence to commitment to change statements 3 months after the modules. Paired sample t-tests were used to assess change in knowledge, comfort, and level of involvement in care.

Frequency counts assessed physician implementation of commitment to change statements. The modules had an impact on physician practice and their comfort managing patients with AD.

MGT475: Strategic Innovation and Ideas

Incognito simulated patients for formative and summative assessment Keywords: James Cook University Summary: To explore the acceptability and feasibility of using incognito simulated patients for formative and summative assessment in general practice and develop a code of practice for the process. The use of simulated patients is a well-established method for the training and assessment of medical students and doctors. In the Netherlands covert or incognito simulated patients are employed to assess the competence of general practitioners in the workplace.

In the UK incognito simulated patients have not been used for assessment, nor has the practice been established for training in consultation skills. Two incognito simulated patients carried out consultations with eleven pre-registration house officers in five general practice settings. The simulated patients gave their views on the doctors' consultation skills.

The doctors and the simulated patients were interviewed to explore their views on the acceptability of the exercise. The doctors did not object to the experience though they did have concerns. They valued receiving feedback on their skills. The simulated patients had varying views on the process. Ethical issues have been raised. There were some logistical problems in setting up the consultations. This appears to be a valuable additional method for training in consultation skills and assessment but a code of practice needs to be established so that doctors and simulated patients do not feel threatened by the process.

Consideration needs to be given to the patient scenarios, the use of feedback and the assessment process. To describe use of patient simulations for student, entry-level, and experienced pharmacists. Retrospective analysis of teaching, learning, and assessment strategies involving patient simulations by a university, a licensing examination body, and the regulatory authority in pharmacy.

At all levels examined, similar approaches to patient simulations are used. At all levels, training and monitoring of assessors is undertaken to optimize reliability of assessment. Patient simulations permeate all levels of pharmacy education and practice in Ontario. Collaboration between acdaemic, regulatory, and examining bodies ensures consistent, fair, and valid use of simulations for teaching and assessment purposes.

Balanced use of holistic global and analytical checklist scoring at all levels provides a consisten approach to use of simulations within pharmacy. Working and training as an intern: In recent years, the Medical Council has sought to enhance the quality of education and training of interns in Ireland. Among its initiatives has been the production of a generic job description, the introduction of a log book so that individuals can monitor their progress and the setting up of a national 'Network of Intern Coordinators and Tutors' to supervise and further develop intern training.

To evaluate the impact of these initiatives, the Medical Council undertook a postal survey of all Irish interns during In contrast to the experience of interns in other countries, the majority provided positive feedback on many aspects of their education and training, their work environment and their professional relationships. However, some problems were identified, including a lack of protected time for education, a lack of formal educational programmes, insufficient feedback on performance, and an unnecessarily stressful work environment. Although feedback on the internship experience in Ireland is generally positive, further work is necessary to address the problems identified.

Ireland now has the necessary structures in place to promote even better Intern education and training. Student technical skill compared to clinical decision-making and interpersonal skills Keywords: Clinical decision-making, competency assessment, interpersonal skills, information gathering, clinical competency, intern education, behavioral ratings Authors: Northwestern Health Sciences University Summary: Data from a clinical evaluation exercise entitled the Developmental Assessment DA were analyzed for rater reliability and association of subscales.

Summary of background data: The DA assesses third year chiropractic students in a minute, standardlized-patient encounter. And 3 are the subscales themselves associated to some degree? Confirmatory factor analysis provided evidence for two of the three subscales. Interpersonal skill variables were less correlated. The research showed strong empirical evidence that students were consistently skilled across the subscales; those highly skilled interpersonally were also skilled clinical thinkers and skilled information gatherers.

The DA instrument appears to reliably assess clinical competence and could be adapted for other health care curricula. The research also supports the use of behavioral ratings in clinical training. Philadelphia College of Osteopathic Medicine Summary: Beginning in September all osteopathic medical students in the USA will be required to take this SP examination in addition to their written board examinations. This presentation will discuss the interpersonal communication scores derived from the Global Patient Assessment GPA during three pilots test of this examination conducted at osteopathic medical schools.

The GPA is a six item rating scale of interpersonal communication skills. SPs complete the GPA along with data gathering checklists of history-taking and physical examination skills during a twelve station performance evaluation. Osteopathic physician raters evaluate the OMT and patient management parts. Dimensions rated on the GPA include active listening, eliciting information, giving information, empathy, respectfulness and professionalism. Score reliability using the GPA instrument has been fairly high.

For example, in the most recent evaluation of fourth year medical students during a twelve station examination, the generalizability coefficient was 0. The purpose of this study is to determine to what extent the phenomenon where low performers over-estimate their performance on exams and high performers under-estimate their performance can be attributed to regression effects. Results showed no statistically significant difference between subgroups 1 and 2 or 3 and 4, suggesting that regression effects do not account for the phenomenon of low performers over-estimating their performance and high performers under-estimating their performance.

A replication in another course yielded similar results. Many forms of Problem-Based Learning PBL , require students to assess their learning needs and set out a plan for meeting those needs. Recent studies have called into question whether low performing students can accurately assess their performance, finding that they tend to have substantially inflated perceptions of their capabilities. Determining whether these findings are real or an artifact of the research design is important to understanding the dynamic that underlies this finding. This study found that the tendency of the poorest performers to over-estimate their performance is not an artifact of the sampling design.

This study examined whether inconsistency between MCAT scores and undergraduate grade point averages GPA related to their predictive validity. We also created 3-dimensional plots with undergraduate GPA and MCAT overall scores on the two horizontal axes and each of the six criterion scores, in turn, on the vertical axis.

The largest correlation was. High MCAT-low GPAs divided the distribution into three different groups showing a more complex pattern involving high percentages of low Step 1 scores as well as high Step 1 scores. These results suggest that predicting medical school performance using MCAT and GPA may require more complex methods than have typically been used. In particular, linear regression may need to involve complex interaction terms in the model.

The Medical Council of New Zealand MCNZ defines a number of clinical and procedural skills that are expected of a doctor at the end of the first year in order to gain general registration. In this study, we survey a group of junior doctors at the beginning and end of their first postgraduate year to gauge their self-perceived experience with a variety of clinical skills and conditions.

At the end of the year there was a significant discrepancy between the skills expected by the MCNZ of those doctors and those skills actually attained as well as a small subset in which skills declined. A review of the literature compares New Zealand results with skill requirements of medical councils in other western countries. Evaluation of basic sciences knowledge at the end of the medical course: Faculty of Medicine, University of Colombo Summary: The adoption of a new medical curriculum resulted in a drastic reduction of teaching hours of basic sciences.

To assess the knowledge of basic sciences at the end of the medical course and to make comparisons between students who followed the traditional and new curricula. An instrument multiple choice question paper was developed with separate questions in basic sciences subjects. The question paper was administered after the final examination. Computer-automated corrections were carried out. Marks obtained for each subject was analyzed separately and comparisons were made between batches. Individual subject marks had a similar trend with better performance shown by batches II and III, except in microbiology and parasitology in which performance of batch II was poor when compared to batch I.

The reduction of traditional teaching lecture and practical hours of basic sciences in the new curriculum does not appear to adversely affect the basic sciences knowledge retained at the end of the medical course. University of Otago, New Zealand Summary: Validation should include subgroup score analysis. Gender is an important issue in a clinical examination as the gender of the candidate, patient and examiner s may influence the outcome. The effect of gender on results in an OSCE for 5th year students was studied.

Two examiners mark students independently on a checklist score and a global score. We analysed the OSCE station scores allowing for gender of candidate, patient and examiner s individually and interactions. Female students did significantly better on stations than male students with regard to global score 2. Stations with female patients had lower checklist scores than those with male patients for The global score for a male examiner pair 3.

The checklist scores for single gender examiner pairs were identical It is reassuring that there is no interaction between the gender of the candidate, examiner and patient. Female students scoring better and mixed examiner pairs marking lower require further study. Assessment by observed consultation: Validation of content Keywords: Validation of content should be more than the content being deemed relevant by experts. It should also include the way in which the scoring of this content is interpreted.

Included in this are 1 history taking, 2 clinical examination and 3 data interpretation stations. The consultation stations are graded on common scales for facets: These attributes are deemed relevant by faculty. Pass mark verification includes global and borderline score techniques.

If the result interpretations are valid, each of these facets should contribute to the overall decision. Logistic regression analyses for prediction of global and borderline scores were used to assess relative contribution to this decision. In , patient relationship is not to be marked on a common scale, but as a veto score and the grading scales for other facets are being extended.

This needs to be considered when developing and comparing marking schemes. Each OSCE consisted of 3 data and 3 observed consultation stations. Each data station produced a single grade A-F. Each consultation station included a grade for: Hence each student was awarded 12 grades. Satisfactory performance for a graduating student was graded C. These 12 marks were summated. As these were penultimate year students the faculty set the accumulated underperformance pass mark at By these methods the accumulated underperformance pass would be lowered from 12 to 6, the compensated pass mark raised from 36 to The process is being reviewed and developed.

Validation evidence may include multitrait multimethod matrices MTMM. This involves correlations between different attributes assessed in different ways. This includes 3 data interpretation patient summaries with common investigations and 3 consultation stations. The consultation stations are marked on: During respiratory and cardiology were most frequent subspecialities represented on stations.

A MTMM matrix was produced for these consultation and data interpretation results. This may be due to the fact that the data questions are described in the context of acute illness done to ensure coverage , where as the consultations are all with people with chronic problems. The positive correlation for data questions may be a true finding, as these questions assess common attributes consultation and management of acute illnesses irrespective of subspeciality. Context specificity may be apparent between acute and chronic presentations as well as subspecialities. Effect of a rotating modular curriculum on examination results Keywords: At the end of the year all students sit an end-of-year OSCE and written examination covering all specialities.

Does rotation affect OSCE outcome? The mean difference between examination scores for each block was calculated. This implies that the 2 OSCE's assess some common attributes. The change in OSCE performance through the year has many possible explanations. Does changing clinical module assessments affect outcome in year assessments? Following 3 years at a central campus, University of Otago students study at 3 clinical schools, rotating through various clinical blocks. This was chosen for several reasons: The consequence of changing the block assessment on year assessment was studied.

Passing scores were not constant over the three years. Although the change did not improve the OSCE score, the lack of a practise written examination did not adversely affect the students. Curriculum, teaching and assessment changes continue at all schools but we should try to ensure that no students are disadvantaged and that advantageous developments are shared.

Student perceptions on the content balance and relevance of curriculum Keywords: Leicester Warwick Medical School curriculum comprises Phase 1, campus based integrated biological, social and clinical science modules, and Phase 2, clinical community and hospital based teaching. Phase 1 module leaders, practising clinicians and senior Phase 2 students are able to perceive relevance of module content to clinical practice. Previous Warwick-based Phase 1 student general feedback suggested that there was an excess of 'sociology' with no relevance to clinical practice.

We explored this further. For each student the biological, clinical and social VAS scales are combined to form triangle, the centroid of which is used as a representation of the overall perception. All student results are plotted to represent class perceptions. Individuals marked the VAS consistently. There was considerable variation between modules for both the content and perceived relevance to Phase 2 and clinical practice.

Clinical content was seen as relevant, sociology was not. The mean VAS marks for the module shown was 4. For Phase 1 modules, rather than abandon student perceived irrelevant content, the relevance, especially "sociology", needs to be improved. Improving examiner consistency in an assessment of advanced life support ALS Keywords: Many UK healthcare postgraduate training programs require the successful completion of an ALS provider course.

Course assessments include observation of candidates dealing with cardio-pulmonary arrest using mannequins. These assessments have become high-stakes examinations. Previously examiner inconsistencies were demonstrated. The assessment process has been altered with performance criteria checklists and dual examiners. Cheryl is a very brilliant woman, teacher, coach, friend, and a leader. I was attracted to her because of her unselfishness, and a willing heart to not only share the knowledge she has with those around her, but also to empower them to look beyond themselves.

Cheryl, I thank you for everything, your wisdom, coaching, and for giving me a wing to fly on. You are an amazing strong woman and a great friend that shares all she has! Lucy Surhyel Newman Lagos, Nigeria. Cheryl developed a totally new approach to publishing for The Refractive Thinker. Cheryl is one of the most vibrant, intelligent, and creative individuals, I have ever worked with.

Laura Grandgenett Cassville, WI. The authors contributing to this series are some of the best critical thinkers in their field of study. Publishing is an expensive venture, time consuming, and most times heartbreaking just trying to get through the door. Who has an easy time with rejection of what may be potentially life changing theories?