Експертнi висновки
У відповідних підрозділах міститься
аналітична інформація у вигляді експертних
звітів професорів Міріам Фрідман Бен-Девід та
Єна Харта.
The report of the professor Miriam Friedman Ben-David
DEVELOPMENT OF AN INDEPENDENT TEST BOARD TO INTRODUCE A LICENSURE SYSTEM INTO THE
HEALTH SECTOR IN UKRAINE
Tacis BISTRO/SSTA project:
Development of an Independent Test Board to introduce a Licensure System into the Health
Sector in Ukraine.
Progress Report
Consultant’s Report
Professor Miriam Friedman Ben-David
Visit to Kyiv, February 5-10, 2000
This report was prepared with the financial assistance from the Commission of European
Communities. The views expressed herein are those of the Consultant and do not represent
any official view of the Commission.
Introduction
This progress report of the TACIS Bistro/SSTA supported project on the “Development of
an Independent Test Board to Introduce a Licensure System in the Health Sector in
Ukraine", is based on the visit of the EU Consultant (Professor Miriam Friedman
Ben-David) from the Centre for Medical Education in Dundee, to the Testing Board in Kyiv,
Ukraine. The visit took place from Saturday February 5th to Thursday February 10th and
consisted of four formal working days.
An outline of Professor Friedman’s Schedule during the visit is attached as Appendix A.
Background
In June 1999, The Testing Board (TB) an independent not for profit professional testing
organisation, was officially registered with the mandate to develop standardised tests for
the licensing of doctors, dentists and pharmacists.
The first mandatory licensing examination in dentistry was implemented for all 1,516
dentistry students who graduated in 1999. The “Krok 2 Dentistry” assesses the
dentistry graduates’ level of professional competence in clinical dentistry sciences.
Further mandatory implementation of Medical Licensing Examinations (MLE) are planned for
medical school graduates in the year 2001. It is projected that approximately 19,000
medical students will take the Basic Science examination “Krok 1” at the end of third
year and the Clinical science examination “Krok 2” at the end of the final year of
their undergraduate medical education.
The mandatory implementation of MLE in Medicine and Dentistry followed a 4 year
development and piloting of the Krok 1 & 2 Medicine and Krok 2 Dentistry.
Although the Testing Board is employing the most recent advanced testing technology
currently employed in the western world, absolute adaptation of western testing practice
might not work in Ukrainian conditions due to different social environment and
legislation. Therefore, the TB is exploring and investigating new testing technologies
while considering adaptation and modification of procedures which will match the Ukrainian
social and educational culture.
In the process of test development and its mandatory implementation the Testing Board
“Discovers” their “Consumers”. The medical school graduates as well as the medical
schools are the current recipients of the examination results An individual certificate
form for “Krok 2 Dentistry” was developed which documents the passing status of
examinees and contains general evaluation of the candidates’ professional competence as
well as useful diagnostic information.
The schools obtain school reports which include the mean score for the school, school
performance for each of the exam sub tests and national mean scores.
TB staff have developed a spider-web Macro analysis report which indicates in a graphic
form performance on the examination sub tests relative to a standard. The report
identifies Macro level knowledge gaps which are further investigated using a micro-level
analysis employing a psychometric analysis as well as flagging difficult topics. The
coding system applied to the examination items in the bank provide the general
classification (topics) as well as specific classification (sub tests) of the knowledge
gaps analysis.
The TB activities point in the direction of high quality test technology with the
incorporation of educational principles which will eventually institute educational reform
and will enhance teaching and learning practices.
Since the Krok 1 & 2 Medicine and Krok 2 Dentistry employ the most advanced concepts
of licensing examinations, it is hoped that eventually the MLE examinations will drive the
educational system.
A great deal of time is invested by TB staff to market the examinations, (CD-Rom,
workshops, publications and others). Further studies are planned to identify educational
and social components which could be influenced by the licensing procedures. Thus the TB
views itself as an organisation that introduces advanced testing technologies as well as
instigating reform in education.
My consultation visit focused mainly on the structure and communication lines in the
Testing Board as well as on issues of marketing the examinations to the medical schools,
and government officials. The cost involved in running the TB for the wide scope of its
activities was a major concern for the TB staff. At this junction, TB staff have an urgent
need to liase with top International testing experts to further validate their operations.
They are also interested in developing knowledge and understanding of educational
innovations in order to better function in their role of instigating educational reform in
Ukraine.
The report addresses the following activities as outlined in the Schedule of Activities
(Appendix B), developed and reported by Professor Ian Hart during his Consultant visit
November 9-14, 1999.
Activity AP-1- Four day Workshop in Kyiv with
Miriam Friedman to train staff in their roles in running an assessment centre.
Activity AP-2- Planning a Five day training on the management of assessment technology at
the Centre for Medical Education in Dundee for 6 Ukrainian staff.
Activity AP-3 To build a computer model of
(Sectorial Study #1) introduction and functioning of licensing examination (note: Slight
modification of the original statement in Appendix B).
Activity AP-4 Development of a business plan of the
(Sectorial Study #2) Testing Board and definition of an efficient and customer appropriate
model of MLE System funding under existing Ukrainian financial and social conditions
The progress Report is organised under the following headings:
1. Report on the Four day Workshops in Kyiv
February 5-10,2000.
2. Planning of the Five day Workshops at the Centre for Medical Education Dundee for
Testing Board staff.
3. Report on Sectorial studies # 1 & 2.
4. Report on a meeting with delegates of the Commission of European Communities in Kyiv.
1. Report on the Four day Workshops in Kyiv February 5-10, 2000
On Sunday February 6, 2000, Testing Board staff (Appendix C), met with Professor Friedman
to discuss the following topics:
1. Running the Testing Board: Public and Internal Policy
(Appendix D).
2. Running the Testing Board: TB divisions, their interactions, divisional products and
responsibilities
(Appendix E).
Policy and structure of Testing Boards in the USA were presented for discussions. The main
topics of concern were:
1. How to develop high quality examinations with low cost.
2. Who should pay for MLE – The Medical Schools? The medical students?
3. Seventy percent of medical students are “contract students” – i.e. they pay for
their medical education. For these students money is the only criteria for entering
medical school. MLE should serve as a gate keeper for preventing incompetent medical
students to obtain a license.
4. Pass/fail policies are needed to prevent the failing students from re-entering
undergraduate education.
5. Lack of guidelines and policies for Foreign Graduates who would like to be treated
differently than the Ukraine medical students.
6. What are the effective ways to approach medical schools and convince them to buy into
the MLE system?
How should resistance in schools be overcome?
7. Innovative methods of evaluating the medical school teachers are needed.
8. TB staff are interested in methods for identifying causes of poor and high performance
on MLE by medical school.
9. TB provides practice tests to the medical schools. TB may score the practice test for
fee and provide diagnosis reports to students.
10. How should the TB expand their testing services to provide for additional revenue,
which will allow development of new testing technology?
11. The advantages of establishing an International Committee for MLE, were explored.
On Sunday afternoon, the discussions were devoted to planning the Monday and Tuesday
(February 7 & 8) workshops. A number of representatives were invited from the Ministry
of Health, Medical Schools, Pharmaceutical, Dental and Nursing education, and programme
officers at the Delegation of European Communities in Ukraine (Appendix F).
The two day workshops were aimed at marketing MLE and obtain a wide support from the
participants for the mandatory implementation of MLE.
Monday February 7, 2000
A number of topics were presented by the Head of Board of Education and Medical Science of
MOH, Professor Yury Voronenko, Director of TB, Dr Irina Bulakh, Deputy Director, Dr Marina
Mrouga, visiting consultant, Professor Miriam Friedman and other representatives (Appendix
A).
The main presentations include:
- Medical Licensing examination in Ukraine: Current achievements.
Y.V. Voronenko (Appendix G)
- Prospective plan of Testing Board development.
I. Bulakh (Appendix H.1)
- Bistro project: Development of Independent Testing Board to introduce Licensure System
in Health Sector in Ukraine.
Marina Mrouga (Appendix H.2)
- Routes for Medical Licensure.
Miriam Friedman Ben-David (Appendix I)
- Innovation in Assessment.
Miriam Friedman Ben-David (Appendix J)
Main discussion topics:
1. Testing in the Nursing profession.
Ten thousand nurses are currently tested in the Ukraine employing a 600 item testing
booklets. The testing activity is endorsed by the Ministry of Health.
2. Expected outcomes of the Bistro project are:
a. Development of an efficient MLE system
b. Development of a strategic management plan
c. Structure of the Testing Board
3. Issues of accredetain were discussed. Currently four levels of higher education
organisations exist in Ukraine: Schools, Colleges, Institutes, University/Academy. All
levels are expected to maintain high standards of professionalism among faculty. Admission
policies as well as the number and the scope of subject matters should be evaluated. If
universities can not demonstrate expected standards they should not be qualified as a
university.
4. The implementation of Step 3 examination was discussed. It is impossible to prepare
quality standardised tests for each speciality. One approach may include the preparation
of tests for blocks of specialities.
5. Assessment of interns.
Plans are under way to establish online testing for interns. TB works closely with Soros
foundation to develop a distance learning course in methods of education and measurement
which is computer based. In this project TB will focus on general educational tendencies
not specifically related to the medical profession.
6. It is essential for TB members to participate in International Conferences and interact
with international experts such as the National Board of Medical Examiners (NBME), USA,
Centre for Medical Education(CME), Dundee, Scotland.
7. Standardised patients programmes were presented for the assessment of clinical skills.
Participants were asked to comment on the possibility of administering such an exam as
part of the MLE procedure. This examination, will add a more comprehensive assessment of
medical schools students in which knowledge as well as clinical skills and attitudes will
be tested.
Tuesday February 8, 2000
In the second workshop day a number of topics were further presented (Appendix A).
Computer support of Medical Licensing examination.
Yuri Bogachkov (Appendix K)
Modern tendencies in Medical Schools Assessment.
Miriam Friedman Ben-David
Central discussion issues included role of Educational Commission in the System of
Licensing Examinations
1. General information about Educational Commission
The Educational Commission will define the overall policy in development of State
standards in Higher Education, and will carry the order of the Minister of Health. The
order defines the committee members and endorses the commission existence and regulates
its functioning (responsibilities, rights, place in MLE system, etc). The general
structure of the commission will fit the structure of MLE. The commission will co-ordinate
the efforts of development of State standards in Higher Education, including MLE,
description of specialists and curriculum development. The TB staff will manage the system
and provide technical and intellectual support. The commission will report to the Board of
Education and Medical Science of the Ministry of Health and its functioning will be
subject to current legislation. Discipline represented will be Medicine, Dentistry and
Pharmacy. The speciality members will rotate.
2. Mission of the Education Commission
? To provide professional expertise of State Standards documents in Medicine and Dentistry
(requirements to professional descriptions, curricula and diagnostic tools).
? To provide expert assessment of introduction of State Standards in Medicine and
Dentistry.
? To control the objectivity of assessment. To defend rights of examinees. (As you put –
Control of the MLE procedure and implementation to ensure equal opportunities)
3. Responsibilities & rights of the Educational Commission
Responsibilities
? Commission will ensure quality professional expertise of ways of introduction of state
standards in Medicine and Dentistry.
? Commission will ensure strict fulfilment of item-review procedures.
? Committee will ensure fulfilment of plans according to the deadlines.
? Committee will oversee timely submission of expert reports.
? Committee will set the marking grade.
? Commission will provide expertise licensing examination results for improvement of
medical curriculum.
Rights
? Commission may provide expertise of licensing examination results for improvement of
medical curriculum
? Commission may promote adaptation of foreign practices and of modern technologies in
education and measurement
? Commission may administer additional item-review if requirements were not met
? Commission may annual the results of licensing examinations if the technology is broken
and to prescribe repetition of test examinations…
4. Organization of work
? Commission will meet once a year
? Review of test materials is done in two stages: 1- review for meeting state standards
– is executed by specialty research laboratories; and 2 – item review.
? Committee will prepare review reports
? Faculty members’ work on item review is part of their academic work
? Logistics of Commission meetings (collection of items, printing, copying, computer
support, etc) is maintained by Testing Board
Following information from the report represents different comments from presenter and
participants.
? Committee will define the schedule of MLE administration
? Standards will be set low at the beginning
? MLE as an independent external examination should not exclude the school’s internal
examinations. On the contrary, school’s should be encouraged to develop high quality
internal examinations.
? MLE introduction will lead the way to improvement in education and curricula reform
? MLE may lay the foundation for candidates ranking for the purpose of selection into
competitive specialities
? A certificate of licensure provides the right to practice rather than a diploma
? Testing Board maintains the item bank. The committee experts are responsible for
modifying the items.
Official representatives and Medical School Faculty as members of TB committee, will
facilitate cross checking of each other’s work.
Additional submission points
? The system is automated to create tests, statistical reports and score reports. There is
a possibility to have on-line testing – via the internet and develop distant sites for
testing.
? Culture of medical education in Ukraine does not yet allow TB to introduce more than one
version of MLE in one given administration.
? It was debated that item writing should be viewed as an educational activity for
faculty, considering the time devoted to this activity.
? Security is still a problem. Protection of MLE booklets is the main issue.
? MLE allows standardisation of core knowledge. Basic and clinical skills should be taught
as a vocational skill. Each school’s objective should be to assess the ability to use
the knowledge. Professional and individual behaviour should be part of the general outcome
scheme in medical schools. There is a new Journal in Ukraine “Medical Education”.
Faculty should publish their ideas about educational reform. It is important that medical
schools will upgrade the quality of their internal examination to match MLE. MLE can give
international weight to the diploma in Ukraine.
Report on the Sectorial Studies # 1 & 2
Sectorial Study #1 – Build a computer model of introduction and functioning of licensing
examination
Miriam Friedman met with Marina Mrouga and Yury Bogachkov to discuss the computer model
for licensure testing. The model should incorporate a broad paradigm of testing aspects
and should include quantifiable parameters as evidence of test validity. The overall
question is: why do we use the testing and how it impacts the health system?
The model stated 3 main parameters
1. state situation
2. public health situation
3. parameters of public system we attempt to influence
The following system of testing has to be incorporated in the model:
1. Technology
2. Parameters of Technology
3. Define the way we use technology
4. Local and outside sites which use the technology
The model should also include obstacles to change:
Licensing Test technology List of parameters to be changed
Define the obstacles to change
Mechanism for quantifying the parameters
List of parameters that should be changed and quantified:
1. Legislation
2. Doctors quality
3. Quality of health services
4. National health statistics
5. Manpower distribution
6. Career choice
7. Attrition rate from medical school
8. Educational reform
9. Quality of education
10. Influence on health related fields
11. Quality of teachers
12. Methods of learning
13. Structure and ratio within medical schools
14. Reform in methods of admission
15. Changes within the private and state sector in medicine
16. Increase in motivation and better preparation for the test
17. Better mechanism to protect patients
The computer model will also incorporate a mapping component which will document standards
of the profession, elements of the programme and the testing components.
Sectorial Study #2 – Development of a Business Plan of the Testing Board
M Friedman met with TB staff members to discuss the TB Business plan (Appendix L)
A business plan template was presented to TB staff. Start-up costs, operating costs and
test development costs were considered for generating the overall cost of MLE. The
division by number of examinees produces the fee per candidate. Projected costs for item
bank expansions, numbers of examinees and the cost of the development of new programmes
were incorporated in the 5 year projected business plan. TB staff will finalise the plan.
Plan a Five Day Workshop at the Centre for Medical Education, Dundee
An outline of the 5 day workshop in Dundee was discussed with TB staff. The workshops will
include topics on trends in Medical Education, assessment technologies, distance learning,
evidence based medicine. Appendix M outlines the workshop agenda from March 18 to March 23
for 6 TB staff.
A meeting with delegates of the Commission of European Committees in Kyiv
Irina Bulakh, Marina Mrouga and Miriam Friedman met with the programme officers of the
Delegation of European Communities in Ukraine: Andriy Sidorenko and Sergiy Ladny, on
Thursday morning February 10th.
The secotorial Studies 1 & 2 were presented and plans for the April seminar were
reported. A questionnaire was prepared by TB staff (Appendix N) and was sent to Medical
School faculty. The replies will be used as discussion points in the April seminar. Self
regulations of professional association and the impact on other professions was also
discussed. Budgetary aspects of the grant were clarified.
Strength of the MLE programme
I found my visit to Kyiv a fascinating experience from a variety of aspects:
1. The success of TB staff to introduce changes in licensure procedures within a culture
which is not entirely ready for such a change, was quite impressive.
2 The positive communication links with the MOH provide good support and backing to the TB
staff.
2. The high level expertise of TB staff in spite the limited budget could only be
explained by high motivation and long hours of work.
3. The liaison with medical schools via faculty involvement in item writing, is an
excellent model for educational reform.
4. The impact on other health professions such as Pharmacy and Nursing is another
indication of the success of the MLE programme.
5. The broad scope of the TB plan to develop innovative testings in all phases of Medical
Licensure, demonstrated the seriousness of this organisation.
6. The comprehensive understanding of TB staff of the impact of licensure on the quality
of care in Ukraine, is unique and has far reaching implications.
Recommendation
1. TB needs to re-organise its structure, define responsibilities, establish committees
which will support its testing and educational activity. An International Committee could
be instrumental in obtaining international recognition.
2. TB staff need to be exposed to the latest in test technologies through conferences and
international experts.
3. Alternative forms of MLE should be eventually introduced to minimise the threat of
security breach.
4. The standard for MLE should be revisited annually to find ways to elevate the standards
and yet increase the pass rate. Currently the passing score is low and may convey the
wrong messages with respect to minimal competence. 5. TB staff should work closely with
medical schools to identify causes for low performance of their students. 6. TB staff
should opt for pass/fail policies for local and foreign candidates. 7.The broader the
educational scope of TB, the more influence MLE will have on the health system in Ukraine.
8.It is recommended to expand MLE by introducing a clinical skills assessment employing
standardised patients. List of Appendix’s Appendix A Tacis Bistro/SSTA Seminar Programme
Appendix B Schedule of Activities – Prof. II Hart Appendix C Testing Board Structure
Appendix D Running the Testing Board-Prof. M. Friedman Ben-David Appendix E Running the
Testing Board- Prof. M. Friedman Ben-David Appendix F List of Participants Appendix G
Medical Licensing – Professor Y. Voronenko Appendix H1 Prospective Plan of TB – Dr I.
Bulakh Appendix H 2 Bistro project – Dr Marina Mrouga Appendix I Route of Medical
Licensure -Prof. M. Friedman Ben-David Appendix J Innovations in Assessment Prof. M.
Friedman Ben-David Appendix K Computer Support – Yury Bugachkov Appendix L Business Plan
for Test Centre – Prof. M. Friedman Ben-David Appendix M Proposed Programme – Dundee
visit March 18-23, 2000 Appendix N Testing Board Questionnaire.
Звіт професора Єна Харта
Загальні коментарі та пропозиції
Неможливо не подякувати доктору Булах та всім
співробітникам з Центру Тестування за той обсяг
роботи, який вони виконали за три короткі роки та
за якість їхньої продукції.
Вони створили національний центр
стандартизованого тестування, який є унікальним
у Східній Європі. Центр тестування був
підтриманий Урядом і підпорядковується
Міністерству охорони здоров’я. Він працює в
умовах нестабільного фінансування і
започатковувався в умовах деякої антипатії з
боку головних споживачів їх продукту - медичних
закладів та їх випускників.
Завдяки послідовній стратегії включення
медичних закладів до процесу в якості партнерів,
антипатія почала зменшуватися. Ця стратегія
також справила бажаний вплив на навчальний план
цих медичних закладів. Як завжди, найбільш
важливий двигун, який спонукає студентів
навчатися та викладацький склад викладати, - це
система оцінювання.
Таким чином, Центр тестування пройшов через
кризові роки національного становлення,
антипатії та напруження і зараз вступає в період
свого зрілого віку, оскільки він набирає досвіду,
його більш приймають та поважають.
Все що робиться, робиться добре, але Центр
безперечно зможе працювати ще краще за
стабільного фінансування та ефективної
інфраструктури. Це без сумніву важливо для
зростання та розширення підприємства та його
діяльності.
Консультативна група Центру медичної освіти
Університету Данді дасть поради та рекомендації
протягом наступних місяців для того, щоб
допомогти Центру нарощувати та розвивати нові
тестові ініціативи.
Після короткого ознайомчого візиту ще
передчасно давати чіткі та детальні
рекомендації - це буде зроблено протягом
наступних декількох місяців після нашої оцінки
ситуації та можливостей майбутнього розвитку
Центру тестування. Не дивлячись на це, навіть на
цій ранній стадії можна порекомендувати
наступне:
- Оскільки поточний 5 річний стратегічний план
підходить до свого завершення, важливо щоб Центр
тестування продовжував розвиватися в
майбутньому
- Мета існування
- Викладення місії
- Стратегічний план на наступні 5 років
- Заходи мають бути вжиті зараз, з тим щоб сприяти
розвитку банку тестових завдань та апробації
інших форматів тестових завдань вибіркового
типу, наприклад розширеного вибору. Без сумніву
повинно пройти декілька років, щоб можна було
дійсно використовувати їх в обов’язкових
іспитах, але цей процес необхідно розпочати
зараз. Ці завдання можна робити на більш високому
рівні ніж завдання типу А з точки зору
розв’язання поставленої задачі.
- Більш нова стратегія для комп’ютеризації
одночасного (on line) тестування повинна бути
прийнята. Починайте працювати над цим зараз.
- Продовжувати працювати над пошуком
інноваційних джерел фінансування, включаючи -
- Продаж послуг для інших дисциплін та напрямів в
межах країни
- Продаж послуг та ваших професійних знань
тестування на великих масштабах іншим країнам
- Фонди, які мають відношення до міжнародної
освіти та охорони здоров’я
- Міжнародні урядові агентства по сприянню
охороні здоров’я та освіти
Протягом наступних місяців консультанти з
Данді будуть працювати з вами, щоб внести свої
пропозиції та розглянути деякі з цих
рекомендацій.
На заключення, я хочу висловити подяку та
вдячність доктору Булах та всім співробітникам
Центру Тестування за їх невтомну допомогу,
відповіді на всі мої нескінчені запитання та за
їх надзвичайну гостинність.
Я також хочу подякувати співробітникам EU TACIS
BISTRO/SSTA за їх допомогу та підтримку.
Експерт Європейської Комісії
Проект TASIC BISTRO/SSTA
BIS/98/025/006
Ян Харт
Заслужений професор
Identification Phase
Consultant’s Report
Visit to Kyiv, November 9-14, 1999
This report was prepared with the financial assistance from the Commission of European
Communities. The views expressed herein are those of the Consultant and do not represent
any official view of the Commission.
Introduction
This, the first report in the TACIS BISTRO/SSTA supported Project on the “Development
of an Independent Test Board to Introduce a Licensure System into the Health Sector in
Ukraine”, is based on the preliminary visit of the EU Consultant (Professor Ian R Hart)
from the Centre for Medical Education in Dundee, to the Testing Board in Kyiv, Ukraine.
The visit took place from Tuesday 9th to Sunday the14th of November and consisted of three
formal project working days and two days of informal meetings and preliminary report
writing.
An outline of Professor Hart’s schedule during the visit is attached as Appendix A.
This report addresses the first 5 activities included in the Schedule of Activities
(Appendix B) as modified following discussions during the current visit.
These activities were:
ACTIVITY |
DESCRIPTION |
IP-1 Identification Phase |
Overview of licensure and certification system in
Ukraine in regard to legislative and professional aspects |
IP-2 Identification Phase |
Clarification and working out the details of
the timetable of the project |
IP-3 Identification Phase |
Re-organization of the Testing Board staff
structure and employment of new staff with identification of particular roles in the
project |
IP-4 Identification Phase |
Identification of EU and Ukrainian project staff |
IP-5 Identification Phase |
Working out of optimal regime of joint work with
foreign experts |
The Report follows, organized under the following headings:
- Overview of Licensure and Certification System in Ukraine (IP-1)
- Progress in Developing Standardised Testing for Health Trainees in Ukraine (IP-1)
- Current Status and Staffing of the Testing Board (IP-3 & 4)
- Revised Timetable and Programme of Collaboration (IP-2 & 5)
- General Comments and Recommendations
1. Overview of Licensure and Certification System in Ukraine (IP-1)
The following comments are based on discussion with Dr Bulakh and the staff of the
Testing Board and some very helpful documents prepared for me by the staff. These
documents are reproduced in Appendices C1-3.
Ukraine has a long history of education in the health sciences and its oldest medical
schools date back to the late 1700’s. There are now 22 schools graduating either
doctors, dentists, pharmacists or all three. Eighteen of these institutions graduate
medical doctors.
The medical schools have had traditional curricula which had changed very little until
the past decade. The assessment process has also been very traditional and based almost
totally on written (essay) and oral examinations. There has been no tradition of
standardized testing in medical schools or for graduation from medical schools, and
indeed, no real experience with standardized testing in higher education in general.
Nineteen ninety one (1991) saw the introduction of major reforms in the Health Care
system in Ukraine with the promotion of a merit-based personnel selection in a free labour
market. This shift in the system now requires medical graduates to meet quality standards
– a difficult goal to achieve when the standards to be met have not been explicitly
defined and the assessment tools incapable of meeting standards of reliability and
validity.
Over the intervening years attempts to change the evaluation system have occurred –
as have some changes in the education of medical students.
- 1992-93 – the Ministry of Health and some medical educators looked at the medical
education systems in several Western countries including USA, UK, Germany and Canada.
- 1992 – a Multiple Choice Question examination was established at the end of the 2 year
“internship” that all graduation students must take. This is actually produced in
several different forms depending in the special area each graduate has taken internship
training in. The bank has not grown and is probably public knowledge by now. No meaningful
psychometrics are produced. This examination is not administered by the Testing Board, but
by another government agency.
- 1993 – the International Biomedical Agency acted as a site for the US ECFMG Clinical
Skills pilot examination.
- 1993 – the Ministry of Education worked with the US agency ETS (Educational Testing
Service) to implement something like the ETS Scholastic Aptitude Test (SAT) for high
school students in the Ukraine. It had major problems and was abandoned. The experience
left standardized testing and psychometrics with a bad name in the country
- In 1992, 1994 and 1996 major restructuring took place in the curricula of the medical
schools, and of the internship. These were structural changes rather than functional, and
there was no major changes in what was taught and the teaching/learning methods used. Such
constant change in the system has led to resistance to further changes, especially to the
use of standardized testing.
In 1996 the Government introduced a national program for medical education assessment
reform in line with state policy for reform of the higher education system.
This led to the introduction of a project to establish a Medical Licensing Examination
(MLE) with two basic principles –
- Medical educators must be trained in the need and the skills for creating standardized
tests, and
- The examination must be piloted satisfactorily before being made mandatory for
licensure.
The Testing Board was established in 1996 with the mandate to develop standardized
tests for the licensing of doctors, dentists and pharmacists.
The Route to Medical Practice in Ukraine
Undergraduate
All 18 medical schools have a 6 year curriculum. The first three years are in the
sciences basic to medicine and the last three years are clinical.
Until 1991 the curriculum undertaken by an individual differed depending on whether
they were place in a Public Health stream or other possible specialty streams. In the past
several years the students have a choice between a medicine stream or a surgical stream.
Testing is by written and oral examinations. To graduate, a student has to pass all the
appropriate in-school examinations.
Postgraduate – Internship
Having passed all the local school examinations most students enter either a medical or
surgical internship with specific subspecialty training. These internships are for two
years.
Postgraduate – Magister
Students who have shown special aptitudes are chosen for further academic training and
enter a 3 year program (including the Internship 2 years) with special academic training
in their chosen specialty/subspecialty. In addition to having to pass the Internship
Examination at the end of two years, these doctors have to complete a project and thesis
before their training is considered complete.
Internship Examination
This is taken by every graduate at the end of two years of postgraduate training.
Passing this gives them the right to practice medicine.
2. Progress in Developing Standardized Testing for Health Trainees in Ukraine (IP-1)
The long-term goal of the Medical Licensing Examination Implementation Project is to
improve the basis of the training of medical specialists in Ukraine through the
incorporation of modern and effective educational methods and strategies.
The Objectives of the Project are to:
- define the level of minimal competence expected of medical graduates
- develop and pilot standardized assessment of medical graduates
- create an infrastructure for standardized assessment at a national level
- develop effective feedback mechanisms for the improvement of medical teaching.
The Ministry of Health independent Testing Board has been responsible for meeting these
objectives and has been in operation since 1996. The Board is responsible for the
production, administration, scoring and psychometrics and banking of quality multiple
choice items and examinations in the medical, dental and pharmacy disciplines at the
national level. It is a bilingual organization – Ukrainian and Russian. All of its tests
and test items must be available in both official languages.
Since piloting its first examination in 1996, it has run integrated, standardized
examinations at both the basic science Krok 1 (Step 1) level at the end of third year and
final year Krok 2(Step 2) level in both Medicine and Dentistry. The total number of
graduates from the 18 Medical schools is approximately 7,000 each year. From the 12 Dental
schools, approximately 1500 graduate each year.
As can be seen from the following Table (extracted from Document 3 in Appendix C3), in
1999, pilot examinations were run at both Krok 1 and 2 levels for over 14,000 medical
students and a mandatory high stakes pass/fail final (Krok 2) examination was run for 1516
dental students.
|
Pilot Examinations |
Mandatory
Examination |
1996 |
1997 |
1998 |
1999 |
1999 |
Schools
that took part in testing |
1 |
3 |
12 |
19 |
12 |
Students
(tested) |
470 |
1222 |
4551 |
14532 |
1516 |
Exam
type |
Course
exam: IM, AN+HIS |
Sum of
course exams: IM DEN |
Integrated
exam: Krok 2M
Krok 2D, IM |
Integrated
exam: Krok 2M
Krok 1 M |
Integrated
exam: Krok 2D |
The Krock 2 Dental examination will continue to be mandatory for graduation from dental
school. The examination is of high reliability and stable in performance.
It is planned to run the Medicine examinations in 2000 again as a pilot with more (300)
questions in each in order to get reliability data on the blueprint sub-scores, i.e. to
have enough questions for each discipline within the overall examinations to give
reproducibility in the results. The plan is then to run the Medicine Krock 1 & 2
examinations in the year 2001 as mandatory examinations.
Dr Miriam Friedman, during her visit and workshop in January or February, 2000 will be
looking more into the details of the item development process, databanking infrastructure
and process, examination administration and the psychometrics of the items and the
examinations.
In the limited time available to look at these issues during this visit, it was my
impression that, in general all of these components of a testing centre appear to
functioning at an acceptable level. Later in this consultancy we hope to be able to give
specific recommendations regarding increasing the efficiency (and cost-effectiveness) of
the overall process. In addition, we anticipate in future reports broadening the scope of
types of test items to be used in the examination.
Some comments on the process and infrastructure follow:
All of the questions currently in the databank (about 4,000 in total) and used in the
examinations are Type A multiple choice questions – 95% or so being one correct answer
from 5 choices, the remainder one from four.
The items are written by approximately 20 to 30 faculty members at each school. These
have all undertaken training on item writing organized by the Testing Board.
About 200 questions are commissioned each year from each school. Each group are asked
for items in specific areas – driven by the needs of the databank.
Questions written by each content expert are supposed to be vetted by the other experts
in that department.
New questions are vetted at the Testing Board for format, translated and entered into
the bank tagged with all other appropriate information regarding them.
New questions are then vetted by appropriate content experts from the central
examination committee who allocate them to the various blueprint subject areas.
Each examination is set according to a predetermined blueprint.
Examinations are administered by paper and pencil and marked by OMR scanning with
random hand checking.
After the examination, each item’s difficulty index, point biserial and the
distribution of distractors chosen are entered into the bank. Occasional questions are
removed from the scoring process because of faulty DI or PB.
The reliability index used is Crohnbach’s. The Medicine pilots and the mandatory
Dental Krock 2 examinations taken in 1999 had reliability indices in the range of 0.95.
Pass/fail decisions are based on an Absolute Standard using the well-recognized
Nedelsky and Hofstee methods.
Feedback to students is given in the form of comparison performance in each subject
area in the blueprint.
Scoring and feedback turnaround time to students and faculty is 10 days.
The pilot examination questions are made available after the examination to both
students and faculty.
Questions in the databank are culled regularly by the content experts on the central
examination committee.
3. Current Status and Staffing of the Testing Board (IP-3 & 4)
Testing Board Staff Structure
Position |
Name |
Sector |
Responsibility (in MLE technological cycle) |
Director |
Bulakh I. Y. |
|
Running of the Testing Board |
Deputy director |
Mrouga M. R. |
|
Examination materials, reports, relations with medical
schools, international relations |
Deputy director for financial management |
Bardachova I. P. |
Finance + personnel |
Financial affairs, personnel management issues |
Head of the department #1 |
Bogachkov Y. M. |
IT |
Development and implementation of technological support means
for licensing examinations |
Head of the department #2 |
Artemchuk L. N. |
Items pool |
Items pool maintenance for "Krok 1 Medicine",
"Krok 2 Medicine" and "Krok 1 Dentistry" examinations |
Head of the department #3 |
Shershneva M. B. |
Items pool |
Items pool maintenance for "Krok 2 Dentistry" and
"Krok 2 Pharmacy" |
Head of the department #4 |
Vilgota M. M. |
Liaison |
Logistics for Expert Committees sessions and MLE
administration, technical processing of items pool, scoring data processing and routing |
Director assistant, translator |
Alexandrovych N. V. |
Structural + translation and editing |
Director assistant work, items pool translation and editing |
IT expert |
Rusina L. I. |
IT |
Software development |
Expert/ consultant |
Khomchenkova N. I. |
Expertise |
Items pool expertise (medical) for "Krok 2
Medicine" |
PC operator |
Beresa Y. O. |
Technical |
Data processing and technical management |
Office manager |
Udra V. P. |
Secretarial |
Office management |
The current staffing structure and responsibilities of the Testing Board is as shown
above. As constituted, it appears to be filling the current functions of the Board
adequately, though there is no doubt that the staff are working very hard to keep up.
At this early stage in the consultancy, which is mainly an information gathering phase
of the work, it is too early to make recommendations regarding the reassignment of current
staff or the recruitment of new staff.
This issue will be addressed and recommendations made at the time of the Workshop at
the CME in Dundee and after the visit of Dr Friedman early next year.
4. Revised Timetable and Programme of Collaboration (IP-2 & 5)
BISTRO/98/025 Project:
Schedule of Activities |
ACTIVITY |
DESCRIPTION |
TIMETABLE |
IP-1
Identification Phase |
Overview of licensure and certification system in
Ukraine in regard to legislative and professional aspects |
November/ December 1999 |
IP-2
Identification Phase |
Clarification and working out the details of
the timetable of the project |
November/ December 1999 |
IP-3
Identification Phase |
Re-organization of the Testing Board staff
structure and employment of new staff with identification of particular roles in the
project |
November/ December 1999 |
IP-4
Identification Phase |
Identification of EU and Ukrainian project staff |
November/ December 1999 |
IP-5
Identification Phase |
Working out of optimal regime of joint work with
foreign experts |
November/
December 1999 |
AP-1
Action Phase |
Four day workshop in Kyiv with Miriam Friedman -
to train staff in their roles in running an assessment centre |
January/February 2000 |
AP-2
Action Phase |
Five day training on the management of assessment
technology at the CME in Dundee for 6 Ukrainian staff |
March (the middle)
2000 |
AP-3
Action Phase |
Facilitation of the development of computer-based
models of the MLE testing technology in the regional centres |
December 1999-
May 2000 |
AP-4
Action Phase |
Development of a business plan of the Testing
Board and definition of an efficient and customer-appropriate model of MLE System funding
under existing Ukrainian financial and social conditions |
December 1999-
May 2000 |
AP-5
Action Phase |
Application of the experience obtained in the
course of the project at the MLE testing cycle and evaluation of the results |
December 1999-
May 2000 |
AP-6
Action Phase |
Presentation at the 9th Ottawa Conference |
March 1-3, 2000 |
AP-7
Action Phase |
Round table seminar involving Testing Board, EU
partner organization, IBA-Kyiv, MOH and other involved institutions to discuss and decide
on:
-Operational Budget of the Testing Board (MLE System)
-Examination price
-Structure of the MLE System (Testing Board) funding |
April 8-14, 2000 |
AP-8
Action Phase |
Draft final Report circulated, feedback Final
report prepared
Printing and circulation of the final Report
All financial accounting to EU
Project finalized |
May 2000 June 2000
July 2000 |
The above timetable and schedule of activities is also found as Appendix B
There will be two Workshops and a final round table Seminar involving CME/Dundee
personnel.
Workshop in Kyiv – January or February
EU Consultant: Dr Miriam Friedman
Testing Board staff
Licensing Examination stakeholders
Potential funding agencies
The Goals:
- To Testing Board staff how to effectively run the subdivisions of the Testing Board with
stress on human resources management, including principles of distribution of duties,
responsibilities and teamwork using Western assessment centers as an example
- To strengthen Testing Board co-operation with medical schools directed towards use of
licensing examination results to facilitate positive changes in curriculum, improvement of
teaching methodology and increase of students' motivation to learning
- To promote the development of licensure system in health sector and application of
standardized testing for measurement of professional competence level of specialists in
Ukraine through collaboration with other stakeholders
Day 1. Participants: Dr. Miriam Friedman, 10 Testing Board
staff
Dr. Miriam Friedman makes brief presentation of each topic followed by Testing Board
feedback with description of local existing system/approaches in regard to the topic and
further discussion |
Time |
Presentation/ activity |
The presenter |
|
Running the Testing Board:
- Stake holders for medical licensing examinations, influence of licensure system to
examinees, educational institutions and system of education
- Staff structure
- Policy issues including examination regulations and public & internal policies
(USMLE for example)
|
|
|
Assessment center budget planning for reproducing of
licensing examinations :principal budget structure of foreign assessment center; operating
cost, start up cost/ developmental costs; projections for numbers of examinees to the
future and others. |
|
|
Materials:
- Orientation manuals to examinees
- Training manuals
- Test scoring manuals
- Test materials
- Scoring sheets
- Security and handling of test materials
|
|
|
Transition from pencil and paper examination to
computer-based examination: major problems (security, logistics and others) and solutions,
principles of on-line testing. |
|
Day 2. Participants: Dr. Miriam Friedman, 10 Testing Board
staff, 10 representatives from medical schools and Ministry of Health Care of Ukraine, 4-5
representatives from International Renaissance Foundation (local Soros foundation),
International Researches and Exchanges Board (USA), British Council, mass media (?) |
Time |
Presentation/ activity |
The presenter |
|
Presentation of BISTRO Project "Development of
Independent Testing Board to Introduce Licensure System in Health Sector in Ukraine" |
Dr. Miriam Friedman, Dr. Irina Bulakh |
|
Reform of medical education in Ukraine |
Dr. Yuri Voronenko (Ministry of Health Care of Ukraine) |
|
Licensure / Certification systems for professional competence |
Dr. Irina Bulakh |
|
Stake holders for medical licensing examinations, influence
of licensure system to examinees, educational institutions and system of education |
Dr. Miriam Friedman |
|
Policy issues including public and internal policies e.g.
USMLE |
Dr. Miriam Friedman |
|
Licensing examinations application in Ukraine in medical
education system:
- the first results of MLE implementation
- presentation of CD-ROM "Medical Education in Ukraine. Integrated Licensing
Examinations"
|
Yuri Bogachkov, PhD |
Day 3. Participants: Dr. Miriam Friedman, 10 Testing Board
staff, 10 representatives from medical schools and Ministry of Health Care of Ukraine.
Round table discussion with Dr. Miriam Friedman giving expert input |
|
Testing Board co-operation with medical schools directed
towards use of licensing examination results to facilitate positive changes in curriculum,
improvement of teaching methodology and increase of students' motivation to learning:
- Requests and suggestions coming from medical schools
- Visualization of problems with teaching and learning due to standardized measurement of
students' knowledge
- Ways for co-operation
- Examination results report forms
|
|
Day 4. Participants: Dr. Miriam Friedman, Ukrainian
Project team |
|
Conclusions for the workshop |
|
|
Discussions of project issues: sectorial studies, Ottawa
conference, publications, coming training in CME, University of Dundee, next steps |
|
A Workshop at the CME, Dundee – March 2000
This will involve the Centre staff and consultants with 6 or so key people from the
Testing Board in Kyiv and address the optimization of the functional roles of these staff
in increasing the efficiency of the Board’s activities.
The detailed program for that Workshop will be worked out at the conclusion of Dr
Friedman’s January/February workshop.
Final Seminar in Kyiv – April 8-14th 2000
EU Consultant
Testing Board
MOH
And other involved institutions
To work out Testing Board’s operational budget, examination costing and pricing and
the sources of ongoing funding for Testing Board activities
5. General Comments and Suggestions
Dr. Bulahk and all the staff at the Testing Board are to be congratulated on how far
they have come in 3 short years and on the quality of their product.
They have built a centre for standardized testing that is unique in Eastern Europe and
one of the few national standardized testing centers anywhere in the world. This they have
done with the support of the Government through the Ministry of Health and in an
environment of fragile funding and some degree of antipathy from the major consumers of
their product – the medical schools and their graduates.
Due to the carefully pursued strategy of getting buy-in from the medical schools by
involving them as partners in the process that antipathy appears to be on the wane. Their
strategy is also starting to have the desired effect on the curricula of the medical
schools. As always, the most important driver of student learning and faculty teaching is
the assessment system.
That said, having gone through the pioneering years of crises, antipathy and struggle,
the Testing Board is now entering its early adulthood period as it matures and becomes
more accepted and respected.
It is doing what it does well, but it could almost certainly do it better with a more
efficient infrastructure and stable funding. And these things are doubly important to the
growth and expansion of the enterprise and its activities.
The University of Dundee Centre for Medical Education Consulting group looks forward to
giving advice and making recommendations over the next few months to assist the Board with
that growth and the development of new testing initiatives.
Following this brief and very preliminary visit it is premature to make many firm and
detailed recommendations – these will come as our appreciation of the situation and
future of the Testing Board develops over the next few months. However, even at this early
stage some suggestions are in order:
- As the current 5 year strategic plan comes to a close it is important that the maturing
Testing Board develops in the near future –
- A vision
- A Mission Statement
- A strategic plan for the next five years
- Steps should be taken now to look at developing a bank and piloting some other types of
multiple choice items e.g. extended matching questions. Although it may be a few years
before they are actually used in mandatory examinations the process should start now. Such
items can get to higher level thinking and problem solving better than A type questions.
- An earlier rather than later strategy for on-line computerized testing should be
adopted. Start working on it now.
- Continue to build on the strategy to seek innovative sources of funding including –
- Selling services to other disciplines and groups within the country
- Selling services and your expertise in large scale testing to other countries
- International education and health related Foundations
- International education and health related government agencies
Over the next few months the Dundee CME consultants will collaborate with you in moving
forward with some of these suggestions.
Finally, I would like to extend my thanks and appreciation to Dr Bulahk and all the
staff at the Testing Board for their untiring assistance to me in answering the unending
questions that I posed to them; and for their wonderful hospitality.
I would also like to thank the EU TACIS BISTRO/SSTA staff members for their assistance
and support.
Professor Ian R Hart
EU expert Project
November 22, 1999
|