Client Report
Background information
Traditionally, a medical student in the United States will submit their application information to the American Medical College Application Service (AMCAS). AMCAS will then verify and standardize transcripts, collect letters of recommendations and personal statements, collect personal information, and provide a means to document meaningful experiences. These applications are then made available to the medical schools an applicant applies to. If the applicant meets certain minimum requirements, he or she is sent a secondary application. The individual medical schools can then collect additional data using this secondary application. These applications are then reviewed and scored by pre-admissions committees in medical schools and invitations for interviews are sent to selected candidates. In 2011, the average medical school applicant submitted 14 applications but received less than 2 interviews (Koenig, et al., 2013). The prescreening of the applicants to determine who would be invited for an interview is a very important step in the process.
Schools have attempted to review applications holistically by considering what each applicant could have achieved if they had all had been on a level playing field without losing sight of what each applicant has achieved (Turner & Nicholson, 2011). Each medical school faces the issue that they receive about 4000 applications from highly qualified students, can conduct about 450 interviews and then have only a limited number of seats per class. With so many variables how can candidates be fairly evaluated? Admissions committee members who have a long tenure on the committee have developed a mental rubric of how they score applicants. Newer members of the admissions committee, however, are requesting a more standardized rubric that they can utilize until they become more experienced at a holistic review. The need to develop such a rubric and allow it to be utilized on the new WebAdmit software is the goal of this project.
The 2003 Supreme Court decision in Grutter v Bollinger stressed the importance of developing an admissions process that was mission-driven, evidenced based and introduced the concept of the educational benefit for all when the student body is diversified (Monroe, A., Quinn, E., Samuelson, W., Dunleavy, D. M., & Dowd, K. W., 2013). By 2008, 58% of medical schools included diversity as one of the primary goals in their admissions process (Monroe et al., 2013). In addition, the term diversity was expanded to include personal attributes, experiences, and demographics. Two conflicting goals have developed. One goal is to maintain diversity in the class and the second is to develop a method of screening that is standardized, defensible, and fair. The diversity of committee members allows for a diversity of screening methods that may help compensate somewhat for the subjectivity of the screens, but the Association of American Medical Colleges insists on even more standardization of the admissions process.
The MCAT and GPA are widely used screening tests for medical admissions and are reliable in differentiating applicants with high cognitive ability as well as predicting future success in cognitive endeavors (Ferguson & Madeley, 2002). In a study by Cariago-Lo et al. (1997) and cited in Salvatori (2001), the science GPA was found to be a good predictor of academic performance in medical students. Salvatori (2001) concluded after a review of the admissions literature that there is predictive validity of academic success in medical school with the MCAT score. There have also been concerns about the subjectivity of interviews and the lack of any data demonstrating a correlation between high interview scores and success in medical school (Turner & Nicholson, 2011). The use of trained interviewers and a more structured interview format may improve the reliability and validity of a selection interview (Salvatori (2001). Another area of dissension with regards to interviews is whether to provide the application data prior to the interview. Those who oppose it state that it increases the potential for bias (Koenig, et al., 2013). In a study conducted by Monroe et al. (2013) of medical school admissions staff by email, interview recommendations and letters of recommendation were rated most important in determining which applicants were admitted, with personal statements and community service increasing in importance compared to a study conducted in 1986. A study conducted by Ferguson et al. and cited in Turner & Nicholson (2011) stated that the personal statements and letters of references, which make up about 20-25% of a medical school application have a low predictive value for success and do not provide enough information to make any conclusions about a candidate. In addition, there is a concern about where content in the personal statement may have been borrowed from (Turner & Nicholson, 2011). Salvatori (2001) concludes that writing samples, whether personal statements or letters of reference, are extremely limited in their use as a reliable, predictive tool for assessing candidates.
Turner & Nicholson (2011) utilized three focus groups, recorded their discussions, and took copious notes to try to ascertain the thinking process of their reviewers. Some of their findings include: many reviewers select for medical work experience especially more than one type of work experience, but they are not consistent on what is defined as the minimum amount of work experience needed to show a good understanding of the medical field. Reviewers place high value on the ability to reflect on both work experience and commitment to study medicine. Many reviewers reject applicants for poor letters of evaluation even if the reviewer needs to read between the lines to get this information. The reviewers commonly used personal statements to select for applicants who were team players in sports or other activities. A lack of consistency in the approach to evaluating applicants is clear.
The general conclusions that can be ascertained from these reviews are that there is little evidence that any one method is superior to another when developing an admissions review process. It behooves each medical school to look carefully at their mission statement and develop a set of admissions criteria standardized in a rubric format to provide some consistency in choosing applicants who meet the mission statement criteria. With these concerns in mind the goal is to develop a rubric that could be used by more inexperienced admissions committee members when scoring applicants for medical school. The lack of a current rubric is a concern added to the fact that the admissions committee will be required to score candidates using the new admissions software, Webadmit, next cycle.
A one-day workshop setting will be utilized and will be scheduled at the beginning of the next admissions cycle to mine the collective unwritten rubrics that seasoned committee members use to develop a written rubric for use by new committee members with the end goal being universal usage. All members of the admissions committee will be strongly encouraged to attend. By the end of the workshop, it is anticipated that a rubric will be developed based on group consensus of criteria that best exemplify the mission statement. At the biweekly admissions committee meetings individual pieces of the rubric will be re-evaluated and reviewed by the committee. During the last quarter of the workshop, the Webadmit training session will be conducted and each admissions committee member will be required to demonstrate proficiency by scoring one applicant on the WebAdmit software utilizing the rubric.
Learning Environment
The proposed solution would be a combination of direct instruction and problem based learning in a learner centered environment. A learner centered environment focuses on the beliefs knowledge and goals of the students (Brown & Green, 2016). The purpose of this workshop is to utilize the backgrounds, biases, previous experiences, and preferences of all committee members to develop a rubric that can be utilized by the more inexperienced members of the committee. The purpose of selecting a learner-centered environment, according to Brown & Green (2016) is to allow learners to look at their preconceived ideas and work together to develop new ways of looking at the content. The hope is that by utilizing a learner centered environment there will be buy-in by the committee’s members since ultimately it will be their choice whether to use a standardized rubric or not. A possible concern with using this type of environment is that it is easy for the groups to lose track of the end goal and get sidetracked on contentious debates and disagreements on what they feel are the most critical attributes to include in the rubric. For this reason, admissions staff members will need to be constantly circulating throughout the room to redirect conversation without stifling creativity or discussion (Johnson and Johnson ,1994).
Health care practitioners have long been using a problem based method of working together. The purpose of these sessions in a health care forum, however, has historically been not to solve clinical problems but instead to generate a list of issues or ideas that would then be further explored (Yeo, 2005). There are several factors that may affect the success of using a problem based method of instruction. One factor is that the success of the endeavor is strongly dependent on the mindset of the participants. If the committee members do not feel this workshop setting and sequence of instruction is a worthwhile method to develop a scoring rubric, they are unlikely to invest in the process. A second major factor is the mix of the participants in each group. Inexperienced members must be encouraged to share their views and members who feel very strongly about their point-of-view must be kept in check, so all can feel empowered to participate (Yeo, 2005). For this reason, the facilitator will have the very important role of motivating all participants and redistributing power when necessary (Yeo, 2005).
Type of Teaching
Since a problem based method of instruction will be used in an environment where the only clear goal is to develop a scoring rubric the teaching method would best be described as indirect-teaching in an open-ended learning environment (Brown & Green, 2016). The purpose of the workshop is to allow an encourage the admissions committee members to thoroughly evaluate the prescreening process taking the time to carefully align criteria with the mission statement of the school. The mission statement provides what Brown and Green (2016) call the big picture. Brown and Green (2016) state that the open-ended learning environment is ideal when the instructor wants to encourage divergent thinking and considering multiple perspectives. Thus, it clearly fits with the design of the workshop. The workshop will provide the opportunity to refine the rubric more carefully as the committee members work through the assignments.
Hannafin et al. (1997) stress that the content that must be covered should be embedded in the task and not interpreted by an outside source such as an instructor or facilitator. This would be key in this workshop because outside interpretation of the attributes of the applicants would lead to a decrease in diversification of thought. The facilitator should have an indirect role where he/she provides guidance and facilitates discussion (Brown & Green, 2016). Brown and Green (2016) stress that no environment should go completely to one extreme or the other. A lack of background information, history of past admissions policies, and statistics would make it difficult for participants to fully explore the mission statement and meet the goals of the workshop. For this reason, mini sessions were held during the biweekly admissions committee meetings to present this information.
Instructional Activities
1. Problem based learning will be utilized in small groups to help develop a rubric for evaluating candidates. Problem based learning provides the opportunity for an open-ended discussion in which all members will contribute a perspective as well as their experiences. This andragogy classically provides the opportunity for group members to discuss a problem and then construct a list of objectives or ideas that will require further research (Hannafin et al., 1997). Problem based learning as an andragogy has a long history in medical schools with the defining characteristic being that it is centered around an ambiguous and complex problem (Ungaretti et al., 2015). Defining characteristics that demonstrate that an applicant is a best candidate for the medical school is clearly a complex problem. Funke (1991) defined a complex problem as having four features: intratransparency, polytely, situational complexity and time-delayed effects. Developing a rubric that is comprehensive and agreeable to all has many goals that could be mutually exclusive, variables that may not be known, as well as an inability to predict future effects of current decisions (Funke, 1991).
The admissions committee has been looking at various aspects of the admissions policy over the last year. The biweekly discussions have provided an impetus to provide data to the committee on both the characteristics and outcomes of the admissions process in the school as well as an exploration of the process at the other 141 allopathic medical schools in the United States. According to Hannafin et al. (1999) when using an open-ended learning environment, it is important to provide learners with the background information and tools they will need to be successful.
These small groups in the workshop will be a mix of new and more experienced committee members preselected before the workshop. After each rubric has been developed a composite rubric will be developed incorporating the most consistent attributes found in each small group rubric. Comparing the similarities and differences will provide topics for debate which will allow different points of view to be considered. Brown & Green (2016) suggest the use of a graphic organizer when comparing similarities and differences. Each leader of a small group would add their attributes to the categories of the rubric with the end goal of developing a single over-arching rubric. This would provide transparency showing that the inputs from all groups were utilized. This is important according to Fernandes and Simon (1999) who state that a person identifies with a particular group and evaluates the alternatives in terms of the consequences to his or her group. They go on to say that the more complex the problem is the more the resources used to solve the problem come from the individual and not from the problem itself (Fernandes & Simon, 1999). Careful selection of group members must be made in order to maintain as much balance as possible between members who would be likely to identify with one particular group or another. The focus needs to be on the criteria found in applicant admissions not on the biases and experiences each committee member brings to the workshop.
2. After the consensus rubric is developed the group will work through a candidate and score the applicant together taking care to document evidence from the application that supports the score on the rubric. According to Katz and Vinker (2014) research studies have questioned both the reliability and the predictive validity of the standard admission process of looking at metrics first and then using an exclusionary essay and interview. They conducted a study of 300 applicants utilizing rubrics to evaluate a number of different tasks which were subsequently evaluated with a scoring rubric. They stress the need for training as well as subsequent review sessions to help decrease inter-rater variability. Hack (2015) designed a study using a rubric to evaluate qualitative traits in students with subsequent follow-up. The motivation was to demonstrate that rubrics provide more transparency and consistency in evaluation.
Working through a rubric as a committee group provides a cooperative learning experience while scoring a rubric for a candidate. Each small group would be given a candidate who has already been pre-screened and scored by more than one experienced member of a past committee. The goal is for each member of the group to feel that they can successfully and accurately score a medical student application. The mix of experienced and inexperienced committee members will provide a scaffold for the inexperienced committee members. As the small groups work through the rubric and score the candidate they will be asked to make notes on their thinking. At the completion of the activity, group members will be given the scores that the experienced member had previously rated them. All groups would then be encouraged to develop a short presentation where they briefly describe the candidate, provide the scores that were given by the experienced members in the past, and provide their scores. They will then defend any discrepancies between the scores. This will provide the opportunity for the rubric to be analyzed and any changes be made before it has been put into practice.
According to Johnson and Johnson (1994) a cooperative learning environment is key to allow participants to manage the conflict which will invariably occur during this workshop. They feel that a cooperative learning environment promotes a greater ability to achieve, as well as more positive relationships. Since the committee will be working together throughout the year and this workshop is likely to bring out strong opinions, a collaborative, cooperative environment must be stressed and maintained.
3. The Association of American Medical Colleges (AAMC) has stressed the need to take a more holistic look at applicants. To assess non-cognitive attributes effectively and fairly there is a push for a more standardized, transparent, legally defensible method of scoring candidates. On one extreme of assessment is the unstructured interview and on the other end is the Mini-Multiple Interviews. This approach of using a scoring rubric while still maintaining some of the flexibility of an interview would seem to best suit the needs of this school. In a study conducted by Donnon, and Paolucci, (2008) utilizing a rubric scored, semi-structured interview increased the inter-rater reliability due to the increased objectivity of the scoring, but the predictive validity still needs further research. In this workshop the rubric will be utilized to score the factual information in the application, not the interview, but the goal of decreasing inter-rater reliability and increasing standardization remains the same. WebAdmit provides the ability to develop elaborate scoring systems. A future goal could be to develop rubrics that score both the factual information in the application as well as the interview.
In this workshop an experienced member will be paired with an inexperienced member. The inexperienced member will talk through their evaluation of a candidate and will receive feedback from the experienced member. Having a senior member of the committee listen to the thinking of a newer member as he or she evaluates the candidate will provide a type of just-in-time teaching. According to Sun, Martinez and Seli (2014) the purpose of just-in-time teaching is to maximize class time for discussions between peers, minimize lecture and put non-class time to better use for students. They analyzed the use of clickers before and in class to determine whether knowing student’s misconceptions beforehand and then tailoring lecture accordingly increased student self-efficacy and engagement. During the break, a short quiz could be given to all committee members to determine areas of weakness in understanding the categories rated on the rubric.
The senior committee member could provide any clarifications that are identified on the quiz as well as answer any questions the more junior member has. Brown & Green (2016) state that just-in-time teaching provides a more direct instructional method in the open-ended teaching environment. Clark (2016) feels that just in time teaching is perceived by medical students as less monotonous and more interactive than traditional teaching. Faculty members tend to be most satisfied with a group workshop setting that allows brainstorming and individual mentoring (Leh, 2005). The ability for the new reviewer to ask questions and get immediate feedback as he/she is working on the candidate review provides context for the advice as well as making it feel most relevant. Hosting a single workshop and then revisiting key components at the biweekly admissions committee meetings would fully maximize the time available as well as keep the content from feeling overwhelming.
4. A short presentation will then be given on accessing the Webadmit software and step-by-step instruction sheets will be distributed. Each member of the committee will then access Webadmit and score an applicant before leaving the workshop. A direct teaching method will be needed here because the software is new to committee members. The clearest, most time effective method possible will be utilized. Brown and Green (2016) advocate using this direct teaching method when the goal is to deliver specific information in a predetermined order. The instructor will provide a step-by-step demonstration first on how to log into Webadmit and then go through the steps in the pre-admission screening. This will allow committee members to utilize both the visual and auditory channels without overwhelming either in the hopes of decreasing cognitive load (Pass & Sweller, 2014). Care will be taken so that the visual images do not contain the text spoken by the instructor and the instruction manual is not passed out until the presentation is complete to prevent redundancy of information (Mayer, 2014). After the presentation committee members will be given written instructions and will be encouraged to utilize these instructions to access the Webadmit system and do a pre-screening assessment of a candidate. This could be done with a “fake” candidate as a simulation or with a real candidate. The information technology staff as well as the admissions staff will move throughout the room providing just-in-time teaching so that committee members are forced to try to work through the screening on their own, but frustration is avoided.
Summary
The purpose of this workshop was to allow experienced and inexperienced admissions committee members the opportunity to collaborate and develop a rubric that will be utilized to screen medical school applicants as part of the preadmissions screening process. The problem that the school has been dealing with is the need to develop a standardized, transparent, defensible screening procedure. The planned workshop will take place in a daylong session at the beginning of the next admissions season and will be supplemented with content presented in the biweekly admissions committee meetings. A stepwise approach will be taken of developing a rubric, utilizing this rubric to check that it meets the goals stated by the committee, collaborative practice using the rubric with newer committee members being mentored by more experienced mentors and then finally demonstrating the ability to use the rubric effectively on the new Webadmit software.
References
Brown, A. H., & Green, T. D. (2016). The essentials of instructional design (3rd edition). New York, NY: Routledge.
Cariaga-Lo, L.D., Enarson, C.E., Crandall, S.J., Zaccaro, D.J. & Richards, B.E. (1997). Cognitive and noncognitive predictors of academic difficulty and attrition. Academic Medicine 72 (October Supplement): S69–S71.
Chih-Yuan Sun, J., Martinez, B., & Seli, H. (2014). Just-in-time or plenty-of-time teaching? Different electronic feedback devices and their effect on student engagement. Journal of Educational Technology & Society, 17(2), 234-244.
Clark, K. R. (2016). Teaching techniques. Just-in-time teaching. Radiologic Technology, 87(4), 465-467.
Donnon, T., & Paolucci, E. O. (2008). A generalizability study of the medical judgment vignettes interview to assess students' noncognitive attributes for medical school. BMC Medical Education, 8:58.
Ferguson, E., & Madeley, L. (2002). Factors associated with success in medical school:Systematic review of the literature. British Medical Journal, 324, 952–957.
Fernandes, R., & Simon, H. A. (1999). A study of how individuals solve complex and ill- structured problems. Policy Sciences, 32(3), 225-245.
Funke, J. (1991). Solving complex problems: Exploration and control of complex systems. In R. Sternberg and P. Frensch (Eds.) Complex problem solving - Principles and mechanisms. Hillsdale. New Jersey: Lawrence Eribaum Associates, pp. 185-222.
Hack, C. (2015). Analytical rubrics in higher education: A repository of empirical data. British Journal of Educational Technology, 46(5), 924-927.
Hannafin, M., Hannafin, K., Land, S., & Oliver, K. (1997). Grounded practice and the design of constructivist learning environments. Educational Technology Research and Development, 45(3), 101-117.
Hannafin, M., Land, S., & Oliver, K. (1999). Open learning environment: Foundations, methods, and models. In C.M. Reigeluth (Ed.) Instructional design theories and models: A new paradigm of instructional theory. Mahwah, N.J: Lawrence Erlbaum Associates.
Johnson, D. W., & Johnson, R. T. (1994). Constructive conflict in the schools. Journal of Social Issues, 50(1), 117-37.
Katz, S., & Vinker, S. (2014). New non-cognitive procedures for medical applicant selection: a qualitative analysis in one school. BMC Medical Education, 14: 237.
Koenig, T. W., Parrish, S. K., Terregino, C. A., Williams, J. P., Dunleavy, D. M., & Volsch, J. M. (2013). Core personal competencies important to entering students’ success in medical school. Academic Medicine, 88(5), 603-613.
Mayer, R. E. (2014). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed.) (p. 43-71). New York, NY: Cambridge University Press.
Monroe, A., Quinn, E., Samuelson, W., Dunleavy, D. M., & Dowd, K. W. (2013). An overview of the medical school admission process and use of applicant data in decision making. Academic Medicine, 88(5), 672-681.
Leh, A. C. (2005). Lessons learned from service learning and reverse mentoring in faculty development: A Case Study in Technology Training. Journal of Technology & Teacher Education, 13(1), 25-41.
Ungaretti, T., Thompson, K. R., Miller, A., & Peterson, T. O. (2015). Problem-based learning: Lessons from medical education and challenges for management education. Academy of Management Learning & Education, 14(2), 173-186.
Paas, F., & Sweller, J. (2014). Implications of cognitive load theory for multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed.) (p. 27-42). New York, NY: Cambridge University Press.
Salvatori, P. (2001). Reliability and Validity of Admissions Tools Used to Select Students for the Health Professions. Advances in Health Sciences Education, 6(2), 159-175.
Turner, R., & Nicholson, S. (2011). Reasons selectors give for accepting and rejecting medical applicants before interview. Medical Education, 45(3), 298-307.
Yeo, R. (2005). Problem-based learning: Lessons for administrators, educators, and learners. International Journal of Educational Management, 19(7), 541-551.
Traditionally, a medical student in the United States will submit their application information to the American Medical College Application Service (AMCAS). AMCAS will then verify and standardize transcripts, collect letters of recommendations and personal statements, collect personal information, and provide a means to document meaningful experiences. These applications are then made available to the medical schools an applicant applies to. If the applicant meets certain minimum requirements, he or she is sent a secondary application. The individual medical schools can then collect additional data using this secondary application. These applications are then reviewed and scored by pre-admissions committees in medical schools and invitations for interviews are sent to selected candidates. In 2011, the average medical school applicant submitted 14 applications but received less than 2 interviews (Koenig, et al., 2013). The prescreening of the applicants to determine who would be invited for an interview is a very important step in the process.
Schools have attempted to review applications holistically by considering what each applicant could have achieved if they had all had been on a level playing field without losing sight of what each applicant has achieved (Turner & Nicholson, 2011). Each medical school faces the issue that they receive about 4000 applications from highly qualified students, can conduct about 450 interviews and then have only a limited number of seats per class. With so many variables how can candidates be fairly evaluated? Admissions committee members who have a long tenure on the committee have developed a mental rubric of how they score applicants. Newer members of the admissions committee, however, are requesting a more standardized rubric that they can utilize until they become more experienced at a holistic review. The need to develop such a rubric and allow it to be utilized on the new WebAdmit software is the goal of this project.
The 2003 Supreme Court decision in Grutter v Bollinger stressed the importance of developing an admissions process that was mission-driven, evidenced based and introduced the concept of the educational benefit for all when the student body is diversified (Monroe, A., Quinn, E., Samuelson, W., Dunleavy, D. M., & Dowd, K. W., 2013). By 2008, 58% of medical schools included diversity as one of the primary goals in their admissions process (Monroe et al., 2013). In addition, the term diversity was expanded to include personal attributes, experiences, and demographics. Two conflicting goals have developed. One goal is to maintain diversity in the class and the second is to develop a method of screening that is standardized, defensible, and fair. The diversity of committee members allows for a diversity of screening methods that may help compensate somewhat for the subjectivity of the screens, but the Association of American Medical Colleges insists on even more standardization of the admissions process.
The MCAT and GPA are widely used screening tests for medical admissions and are reliable in differentiating applicants with high cognitive ability as well as predicting future success in cognitive endeavors (Ferguson & Madeley, 2002). In a study by Cariago-Lo et al. (1997) and cited in Salvatori (2001), the science GPA was found to be a good predictor of academic performance in medical students. Salvatori (2001) concluded after a review of the admissions literature that there is predictive validity of academic success in medical school with the MCAT score. There have also been concerns about the subjectivity of interviews and the lack of any data demonstrating a correlation between high interview scores and success in medical school (Turner & Nicholson, 2011). The use of trained interviewers and a more structured interview format may improve the reliability and validity of a selection interview (Salvatori (2001). Another area of dissension with regards to interviews is whether to provide the application data prior to the interview. Those who oppose it state that it increases the potential for bias (Koenig, et al., 2013). In a study conducted by Monroe et al. (2013) of medical school admissions staff by email, interview recommendations and letters of recommendation were rated most important in determining which applicants were admitted, with personal statements and community service increasing in importance compared to a study conducted in 1986. A study conducted by Ferguson et al. and cited in Turner & Nicholson (2011) stated that the personal statements and letters of references, which make up about 20-25% of a medical school application have a low predictive value for success and do not provide enough information to make any conclusions about a candidate. In addition, there is a concern about where content in the personal statement may have been borrowed from (Turner & Nicholson, 2011). Salvatori (2001) concludes that writing samples, whether personal statements or letters of reference, are extremely limited in their use as a reliable, predictive tool for assessing candidates.
Turner & Nicholson (2011) utilized three focus groups, recorded their discussions, and took copious notes to try to ascertain the thinking process of their reviewers. Some of their findings include: many reviewers select for medical work experience especially more than one type of work experience, but they are not consistent on what is defined as the minimum amount of work experience needed to show a good understanding of the medical field. Reviewers place high value on the ability to reflect on both work experience and commitment to study medicine. Many reviewers reject applicants for poor letters of evaluation even if the reviewer needs to read between the lines to get this information. The reviewers commonly used personal statements to select for applicants who were team players in sports or other activities. A lack of consistency in the approach to evaluating applicants is clear.
The general conclusions that can be ascertained from these reviews are that there is little evidence that any one method is superior to another when developing an admissions review process. It behooves each medical school to look carefully at their mission statement and develop a set of admissions criteria standardized in a rubric format to provide some consistency in choosing applicants who meet the mission statement criteria. With these concerns in mind the goal is to develop a rubric that could be used by more inexperienced admissions committee members when scoring applicants for medical school. The lack of a current rubric is a concern added to the fact that the admissions committee will be required to score candidates using the new admissions software, Webadmit, next cycle.
A one-day workshop setting will be utilized and will be scheduled at the beginning of the next admissions cycle to mine the collective unwritten rubrics that seasoned committee members use to develop a written rubric for use by new committee members with the end goal being universal usage. All members of the admissions committee will be strongly encouraged to attend. By the end of the workshop, it is anticipated that a rubric will be developed based on group consensus of criteria that best exemplify the mission statement. At the biweekly admissions committee meetings individual pieces of the rubric will be re-evaluated and reviewed by the committee. During the last quarter of the workshop, the Webadmit training session will be conducted and each admissions committee member will be required to demonstrate proficiency by scoring one applicant on the WebAdmit software utilizing the rubric.
Learning Environment
The proposed solution would be a combination of direct instruction and problem based learning in a learner centered environment. A learner centered environment focuses on the beliefs knowledge and goals of the students (Brown & Green, 2016). The purpose of this workshop is to utilize the backgrounds, biases, previous experiences, and preferences of all committee members to develop a rubric that can be utilized by the more inexperienced members of the committee. The purpose of selecting a learner-centered environment, according to Brown & Green (2016) is to allow learners to look at their preconceived ideas and work together to develop new ways of looking at the content. The hope is that by utilizing a learner centered environment there will be buy-in by the committee’s members since ultimately it will be their choice whether to use a standardized rubric or not. A possible concern with using this type of environment is that it is easy for the groups to lose track of the end goal and get sidetracked on contentious debates and disagreements on what they feel are the most critical attributes to include in the rubric. For this reason, admissions staff members will need to be constantly circulating throughout the room to redirect conversation without stifling creativity or discussion (Johnson and Johnson ,1994).
Health care practitioners have long been using a problem based method of working together. The purpose of these sessions in a health care forum, however, has historically been not to solve clinical problems but instead to generate a list of issues or ideas that would then be further explored (Yeo, 2005). There are several factors that may affect the success of using a problem based method of instruction. One factor is that the success of the endeavor is strongly dependent on the mindset of the participants. If the committee members do not feel this workshop setting and sequence of instruction is a worthwhile method to develop a scoring rubric, they are unlikely to invest in the process. A second major factor is the mix of the participants in each group. Inexperienced members must be encouraged to share their views and members who feel very strongly about their point-of-view must be kept in check, so all can feel empowered to participate (Yeo, 2005). For this reason, the facilitator will have the very important role of motivating all participants and redistributing power when necessary (Yeo, 2005).
Type of Teaching
Since a problem based method of instruction will be used in an environment where the only clear goal is to develop a scoring rubric the teaching method would best be described as indirect-teaching in an open-ended learning environment (Brown & Green, 2016). The purpose of the workshop is to allow an encourage the admissions committee members to thoroughly evaluate the prescreening process taking the time to carefully align criteria with the mission statement of the school. The mission statement provides what Brown and Green (2016) call the big picture. Brown and Green (2016) state that the open-ended learning environment is ideal when the instructor wants to encourage divergent thinking and considering multiple perspectives. Thus, it clearly fits with the design of the workshop. The workshop will provide the opportunity to refine the rubric more carefully as the committee members work through the assignments.
Hannafin et al. (1997) stress that the content that must be covered should be embedded in the task and not interpreted by an outside source such as an instructor or facilitator. This would be key in this workshop because outside interpretation of the attributes of the applicants would lead to a decrease in diversification of thought. The facilitator should have an indirect role where he/she provides guidance and facilitates discussion (Brown & Green, 2016). Brown and Green (2016) stress that no environment should go completely to one extreme or the other. A lack of background information, history of past admissions policies, and statistics would make it difficult for participants to fully explore the mission statement and meet the goals of the workshop. For this reason, mini sessions were held during the biweekly admissions committee meetings to present this information.
Instructional Activities
1. Problem based learning will be utilized in small groups to help develop a rubric for evaluating candidates. Problem based learning provides the opportunity for an open-ended discussion in which all members will contribute a perspective as well as their experiences. This andragogy classically provides the opportunity for group members to discuss a problem and then construct a list of objectives or ideas that will require further research (Hannafin et al., 1997). Problem based learning as an andragogy has a long history in medical schools with the defining characteristic being that it is centered around an ambiguous and complex problem (Ungaretti et al., 2015). Defining characteristics that demonstrate that an applicant is a best candidate for the medical school is clearly a complex problem. Funke (1991) defined a complex problem as having four features: intratransparency, polytely, situational complexity and time-delayed effects. Developing a rubric that is comprehensive and agreeable to all has many goals that could be mutually exclusive, variables that may not be known, as well as an inability to predict future effects of current decisions (Funke, 1991).
The admissions committee has been looking at various aspects of the admissions policy over the last year. The biweekly discussions have provided an impetus to provide data to the committee on both the characteristics and outcomes of the admissions process in the school as well as an exploration of the process at the other 141 allopathic medical schools in the United States. According to Hannafin et al. (1999) when using an open-ended learning environment, it is important to provide learners with the background information and tools they will need to be successful.
These small groups in the workshop will be a mix of new and more experienced committee members preselected before the workshop. After each rubric has been developed a composite rubric will be developed incorporating the most consistent attributes found in each small group rubric. Comparing the similarities and differences will provide topics for debate which will allow different points of view to be considered. Brown & Green (2016) suggest the use of a graphic organizer when comparing similarities and differences. Each leader of a small group would add their attributes to the categories of the rubric with the end goal of developing a single over-arching rubric. This would provide transparency showing that the inputs from all groups were utilized. This is important according to Fernandes and Simon (1999) who state that a person identifies with a particular group and evaluates the alternatives in terms of the consequences to his or her group. They go on to say that the more complex the problem is the more the resources used to solve the problem come from the individual and not from the problem itself (Fernandes & Simon, 1999). Careful selection of group members must be made in order to maintain as much balance as possible between members who would be likely to identify with one particular group or another. The focus needs to be on the criteria found in applicant admissions not on the biases and experiences each committee member brings to the workshop.
2. After the consensus rubric is developed the group will work through a candidate and score the applicant together taking care to document evidence from the application that supports the score on the rubric. According to Katz and Vinker (2014) research studies have questioned both the reliability and the predictive validity of the standard admission process of looking at metrics first and then using an exclusionary essay and interview. They conducted a study of 300 applicants utilizing rubrics to evaluate a number of different tasks which were subsequently evaluated with a scoring rubric. They stress the need for training as well as subsequent review sessions to help decrease inter-rater variability. Hack (2015) designed a study using a rubric to evaluate qualitative traits in students with subsequent follow-up. The motivation was to demonstrate that rubrics provide more transparency and consistency in evaluation.
Working through a rubric as a committee group provides a cooperative learning experience while scoring a rubric for a candidate. Each small group would be given a candidate who has already been pre-screened and scored by more than one experienced member of a past committee. The goal is for each member of the group to feel that they can successfully and accurately score a medical student application. The mix of experienced and inexperienced committee members will provide a scaffold for the inexperienced committee members. As the small groups work through the rubric and score the candidate they will be asked to make notes on their thinking. At the completion of the activity, group members will be given the scores that the experienced member had previously rated them. All groups would then be encouraged to develop a short presentation where they briefly describe the candidate, provide the scores that were given by the experienced members in the past, and provide their scores. They will then defend any discrepancies between the scores. This will provide the opportunity for the rubric to be analyzed and any changes be made before it has been put into practice.
According to Johnson and Johnson (1994) a cooperative learning environment is key to allow participants to manage the conflict which will invariably occur during this workshop. They feel that a cooperative learning environment promotes a greater ability to achieve, as well as more positive relationships. Since the committee will be working together throughout the year and this workshop is likely to bring out strong opinions, a collaborative, cooperative environment must be stressed and maintained.
3. The Association of American Medical Colleges (AAMC) has stressed the need to take a more holistic look at applicants. To assess non-cognitive attributes effectively and fairly there is a push for a more standardized, transparent, legally defensible method of scoring candidates. On one extreme of assessment is the unstructured interview and on the other end is the Mini-Multiple Interviews. This approach of using a scoring rubric while still maintaining some of the flexibility of an interview would seem to best suit the needs of this school. In a study conducted by Donnon, and Paolucci, (2008) utilizing a rubric scored, semi-structured interview increased the inter-rater reliability due to the increased objectivity of the scoring, but the predictive validity still needs further research. In this workshop the rubric will be utilized to score the factual information in the application, not the interview, but the goal of decreasing inter-rater reliability and increasing standardization remains the same. WebAdmit provides the ability to develop elaborate scoring systems. A future goal could be to develop rubrics that score both the factual information in the application as well as the interview.
In this workshop an experienced member will be paired with an inexperienced member. The inexperienced member will talk through their evaluation of a candidate and will receive feedback from the experienced member. Having a senior member of the committee listen to the thinking of a newer member as he or she evaluates the candidate will provide a type of just-in-time teaching. According to Sun, Martinez and Seli (2014) the purpose of just-in-time teaching is to maximize class time for discussions between peers, minimize lecture and put non-class time to better use for students. They analyzed the use of clickers before and in class to determine whether knowing student’s misconceptions beforehand and then tailoring lecture accordingly increased student self-efficacy and engagement. During the break, a short quiz could be given to all committee members to determine areas of weakness in understanding the categories rated on the rubric.
The senior committee member could provide any clarifications that are identified on the quiz as well as answer any questions the more junior member has. Brown & Green (2016) state that just-in-time teaching provides a more direct instructional method in the open-ended teaching environment. Clark (2016) feels that just in time teaching is perceived by medical students as less monotonous and more interactive than traditional teaching. Faculty members tend to be most satisfied with a group workshop setting that allows brainstorming and individual mentoring (Leh, 2005). The ability for the new reviewer to ask questions and get immediate feedback as he/she is working on the candidate review provides context for the advice as well as making it feel most relevant. Hosting a single workshop and then revisiting key components at the biweekly admissions committee meetings would fully maximize the time available as well as keep the content from feeling overwhelming.
4. A short presentation will then be given on accessing the Webadmit software and step-by-step instruction sheets will be distributed. Each member of the committee will then access Webadmit and score an applicant before leaving the workshop. A direct teaching method will be needed here because the software is new to committee members. The clearest, most time effective method possible will be utilized. Brown and Green (2016) advocate using this direct teaching method when the goal is to deliver specific information in a predetermined order. The instructor will provide a step-by-step demonstration first on how to log into Webadmit and then go through the steps in the pre-admission screening. This will allow committee members to utilize both the visual and auditory channels without overwhelming either in the hopes of decreasing cognitive load (Pass & Sweller, 2014). Care will be taken so that the visual images do not contain the text spoken by the instructor and the instruction manual is not passed out until the presentation is complete to prevent redundancy of information (Mayer, 2014). After the presentation committee members will be given written instructions and will be encouraged to utilize these instructions to access the Webadmit system and do a pre-screening assessment of a candidate. This could be done with a “fake” candidate as a simulation or with a real candidate. The information technology staff as well as the admissions staff will move throughout the room providing just-in-time teaching so that committee members are forced to try to work through the screening on their own, but frustration is avoided.
Summary
The purpose of this workshop was to allow experienced and inexperienced admissions committee members the opportunity to collaborate and develop a rubric that will be utilized to screen medical school applicants as part of the preadmissions screening process. The problem that the school has been dealing with is the need to develop a standardized, transparent, defensible screening procedure. The planned workshop will take place in a daylong session at the beginning of the next admissions season and will be supplemented with content presented in the biweekly admissions committee meetings. A stepwise approach will be taken of developing a rubric, utilizing this rubric to check that it meets the goals stated by the committee, collaborative practice using the rubric with newer committee members being mentored by more experienced mentors and then finally demonstrating the ability to use the rubric effectively on the new Webadmit software.
References
Brown, A. H., & Green, T. D. (2016). The essentials of instructional design (3rd edition). New York, NY: Routledge.
Cariaga-Lo, L.D., Enarson, C.E., Crandall, S.J., Zaccaro, D.J. & Richards, B.E. (1997). Cognitive and noncognitive predictors of academic difficulty and attrition. Academic Medicine 72 (October Supplement): S69–S71.
Chih-Yuan Sun, J., Martinez, B., & Seli, H. (2014). Just-in-time or plenty-of-time teaching? Different electronic feedback devices and their effect on student engagement. Journal of Educational Technology & Society, 17(2), 234-244.
Clark, K. R. (2016). Teaching techniques. Just-in-time teaching. Radiologic Technology, 87(4), 465-467.
Donnon, T., & Paolucci, E. O. (2008). A generalizability study of the medical judgment vignettes interview to assess students' noncognitive attributes for medical school. BMC Medical Education, 8:58.
Ferguson, E., & Madeley, L. (2002). Factors associated with success in medical school:Systematic review of the literature. British Medical Journal, 324, 952–957.
Fernandes, R., & Simon, H. A. (1999). A study of how individuals solve complex and ill- structured problems. Policy Sciences, 32(3), 225-245.
Funke, J. (1991). Solving complex problems: Exploration and control of complex systems. In R. Sternberg and P. Frensch (Eds.) Complex problem solving - Principles and mechanisms. Hillsdale. New Jersey: Lawrence Eribaum Associates, pp. 185-222.
Hack, C. (2015). Analytical rubrics in higher education: A repository of empirical data. British Journal of Educational Technology, 46(5), 924-927.
Hannafin, M., Hannafin, K., Land, S., & Oliver, K. (1997). Grounded practice and the design of constructivist learning environments. Educational Technology Research and Development, 45(3), 101-117.
Hannafin, M., Land, S., & Oliver, K. (1999). Open learning environment: Foundations, methods, and models. In C.M. Reigeluth (Ed.) Instructional design theories and models: A new paradigm of instructional theory. Mahwah, N.J: Lawrence Erlbaum Associates.
Johnson, D. W., & Johnson, R. T. (1994). Constructive conflict in the schools. Journal of Social Issues, 50(1), 117-37.
Katz, S., & Vinker, S. (2014). New non-cognitive procedures for medical applicant selection: a qualitative analysis in one school. BMC Medical Education, 14: 237.
Koenig, T. W., Parrish, S. K., Terregino, C. A., Williams, J. P., Dunleavy, D. M., & Volsch, J. M. (2013). Core personal competencies important to entering students’ success in medical school. Academic Medicine, 88(5), 603-613.
Mayer, R. E. (2014). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed.) (p. 43-71). New York, NY: Cambridge University Press.
Monroe, A., Quinn, E., Samuelson, W., Dunleavy, D. M., & Dowd, K. W. (2013). An overview of the medical school admission process and use of applicant data in decision making. Academic Medicine, 88(5), 672-681.
Leh, A. C. (2005). Lessons learned from service learning and reverse mentoring in faculty development: A Case Study in Technology Training. Journal of Technology & Teacher Education, 13(1), 25-41.
Ungaretti, T., Thompson, K. R., Miller, A., & Peterson, T. O. (2015). Problem-based learning: Lessons from medical education and challenges for management education. Academy of Management Learning & Education, 14(2), 173-186.
Paas, F., & Sweller, J. (2014). Implications of cognitive load theory for multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed.) (p. 27-42). New York, NY: Cambridge University Press.
Salvatori, P. (2001). Reliability and Validity of Admissions Tools Used to Select Students for the Health Professions. Advances in Health Sciences Education, 6(2), 159-175.
Turner, R., & Nicholson, S. (2011). Reasons selectors give for accepting and rejecting medical applicants before interview. Medical Education, 45(3), 298-307.
Yeo, R. (2005). Problem-based learning: Lessons for administrators, educators, and learners. International Journal of Educational Management, 19(7), 541-551.
