Експертнi висновки

У відповідних підрозділах міститься аналітична інформація у вигляді експертних звітів професорів Міріам Фрідман Бен-Девід та Єна Харта.

The report of the professor Miriam Friedman Ben-David

DEVELOPMENT OF AN INDEPENDENT TEST BOARD TO INTRODUCE A LICENSURE SYSTEM INTO THE HEALTH SECTOR IN UKRAINE
Tacis BISTRO/SSTA project:
Development of an Independent Test Board to introduce a Licensure System into the Health Sector in Ukraine.

Progress Report

Consultant’s Report

Professor Miriam Friedman Ben-David

Visit to Kyiv, February 5-10, 2000

This report was prepared with the financial assistance from the Commission of European Communities. The views expressed herein are those of the Consultant and do not represent any official view of the Commission.

Introduction

This progress report of the TACIS Bistro/SSTA supported project on the “Development of an Independent Test Board to Introduce a Licensure System in the Health Sector in Ukraine", is based on the visit of the EU Consultant (Professor Miriam Friedman Ben-David) from the Centre for Medical Education in Dundee, to the Testing Board in Kyiv, Ukraine. The visit took place from Saturday February 5th to Thursday February 10th and consisted of four formal working days.

An outline of Professor Friedman’s Schedule during the visit is attached as Appendix A.

Background
In June 1999, The Testing Board (TB) an independent not for profit professional testing organisation, was officially registered with the mandate to develop standardised tests for the licensing of doctors, dentists and pharmacists.
The first mandatory licensing examination in dentistry was implemented for all 1,516 dentistry students who graduated in 1999. The “Krok 2 Dentistry” assesses the dentistry graduates’ level of professional competence in clinical dentistry sciences.

Further mandatory implementation of Medical Licensing Examinations (MLE) are planned for medical school graduates in the year 2001. It is projected that approximately 19,000 medical students will take the Basic Science examination “Krok 1” at the end of third year and the Clinical science examination “Krok 2” at the end of the final year of their undergraduate medical education.

The mandatory implementation of MLE in Medicine and Dentistry followed a 4 year development and piloting of the Krok 1 & 2 Medicine and Krok 2 Dentistry.

Although the Testing Board is employing the most recent advanced testing technology currently employed in the western world, absolute adaptation of western testing practice might not work in Ukrainian conditions due to different social environment and legislation. Therefore, the TB is exploring and investigating new testing technologies while considering adaptation and modification of procedures which will match the Ukrainian social and educational culture.

In the process of test development and its mandatory implementation the Testing Board “Discovers” their “Consumers”. The medical school graduates as well as the medical schools are the current recipients of the examination results An individual certificate form for “Krok 2 Dentistry” was developed which documents the passing status of examinees and contains general evaluation of the candidates’ professional competence as well as useful diagnostic information.

The schools obtain school reports which include the mean score for the school, school performance for each of the exam sub tests and national mean scores.

TB staff have developed a spider-web Macro analysis report which indicates in a graphic form performance on the examination sub tests relative to a standard. The report identifies Macro level knowledge gaps which are further investigated using a micro-level analysis employing a psychometric analysis as well as flagging difficult topics. The coding system applied to the examination items in the bank provide the general classification (topics) as well as specific classification (sub tests) of the knowledge gaps analysis.
The TB activities point in the direction of high quality test technology with the incorporation of educational principles which will eventually institute educational reform and will enhance teaching and learning practices.

Since the Krok 1 & 2 Medicine and Krok 2 Dentistry employ the most advanced concepts of licensing examinations, it is hoped that eventually the MLE examinations will drive the educational system.

A great deal of time is invested by TB staff to market the examinations, (CD-Rom, workshops, publications and others). Further studies are planned to identify educational and social components which could be influenced by the licensing procedures. Thus the TB views itself as an organisation that introduces advanced testing technologies as well as instigating reform in education.

My consultation visit focused mainly on the structure and communication lines in the Testing Board as well as on issues of marketing the examinations to the medical schools, and government officials. The cost involved in running the TB for the wide scope of its activities was a major concern for the TB staff. At this junction, TB staff have an urgent need to liase with top International testing experts to further validate their operations. They are also interested in developing knowledge and understanding of educational innovations in order to better function in their role of instigating educational reform in Ukraine.

The report addresses the following activities as outlined in the Schedule of Activities (Appendix B), developed and reported by Professor Ian Hart during his Consultant visit November 9-14, 1999.

Activity AP-1- Four day Workshop in Kyiv with
Miriam Friedman to train staff in their roles in running an assessment centre.

Activity AP-2- Planning a Five day training on the management of assessment technology at the Centre for Medical Education in Dundee for 6 Ukrainian staff.

Activity AP-3 To build a computer model of
(Sectorial Study #1) introduction and functioning of licensing examination (note: Slight modification of the original statement in Appendix B).
Activity AP-4 Development of a business plan of the
(Sectorial Study #2) Testing Board and definition of an efficient and customer appropriate model of MLE System funding under existing Ukrainian financial and social conditions

The progress Report is organised under the following headings:

1. Report on the Four day Workshops in Kyiv
February 5-10,2000.

2. Planning of the Five day Workshops at the Centre for Medical Education Dundee for Testing Board staff.

3. Report on Sectorial studies # 1 & 2.

4. Report on a meeting with delegates of the Commission of European Communities in Kyiv.
1. Report on the Four day Workshops in Kyiv February 5-10, 2000

On Sunday February 6, 2000, Testing Board staff (Appendix C), met with Professor Friedman to discuss the following topics:

1. Running the Testing Board: Public and Internal Policy
(Appendix D).

2. Running the Testing Board: TB divisions, their interactions, divisional products and responsibilities
(Appendix E).

Policy and structure of Testing Boards in the USA were presented for discussions. The main topics of concern were:

1. How to develop high quality examinations with low cost.

2. Who should pay for MLE – The Medical Schools? The medical students?

3. Seventy percent of medical students are “contract students” – i.e. they pay for their medical education. For these students money is the only criteria for entering medical school. MLE should serve as a gate keeper for preventing incompetent medical students to obtain a license.

4. Pass/fail policies are needed to prevent the failing students from re-entering undergraduate education.

5. Lack of guidelines and policies for Foreign Graduates who would like to be treated differently than the Ukraine medical students.

6. What are the effective ways to approach medical schools and convince them to buy into the MLE system?
How should resistance in schools be overcome?

7. Innovative methods of evaluating the medical school teachers are needed.

8. TB staff are interested in methods for identifying causes of poor and high performance on MLE by medical school.

9. TB provides practice tests to the medical schools. TB may score the practice test for fee and provide diagnosis reports to students.

10. How should the TB expand their testing services to provide for additional revenue, which will allow development of new testing technology?

11. The advantages of establishing an International Committee for MLE, were explored.

On Sunday afternoon, the discussions were devoted to planning the Monday and Tuesday (February 7 & 8) workshops. A number of representatives were invited from the Ministry of Health, Medical Schools, Pharmaceutical, Dental and Nursing education, and programme officers at the Delegation of European Communities in Ukraine (Appendix F).

The two day workshops were aimed at marketing MLE and obtain a wide support from the participants for the mandatory implementation of MLE.

Monday February 7, 2000

A number of topics were presented by the Head of Board of Education and Medical Science of MOH, Professor Yury Voronenko, Director of TB, Dr Irina Bulakh, Deputy Director, Dr Marina Mrouga, visiting consultant, Professor Miriam Friedman and other representatives (Appendix A).

The main presentations include:

- Medical Licensing examination in Ukraine: Current achievements.
Y.V. Voronenko (Appendix G)

- Prospective plan of Testing Board development.
I. Bulakh (Appendix H.1)

- Bistro project: Development of Independent Testing Board to introduce Licensure System in Health Sector in Ukraine.
Marina Mrouga (Appendix H.2)

- Routes for Medical Licensure.
Miriam Friedman Ben-David (Appendix I)

- Innovation in Assessment.
Miriam Friedman Ben-David (Appendix J)
Main discussion topics:

1. Testing in the Nursing profession.
Ten thousand nurses are currently tested in the Ukraine employing a 600 item testing booklets. The testing activity is endorsed by the Ministry of Health.

2. Expected outcomes of the Bistro project are:
a. Development of an efficient MLE system
b. Development of a strategic management plan
c. Structure of the Testing Board

3. Issues of accredetain were discussed. Currently four levels of higher education organisations exist in Ukraine: Schools, Colleges, Institutes, University/Academy. All levels are expected to maintain high standards of professionalism among faculty. Admission policies as well as the number and the scope of subject matters should be evaluated. If universities can not demonstrate expected standards they should not be qualified as a university.

4. The implementation of Step 3 examination was discussed. It is impossible to prepare quality standardised tests for each speciality. One approach may include the preparation of tests for blocks of specialities.

5. Assessment of interns.
Plans are under way to establish online testing for interns. TB works closely with Soros foundation to develop a distance learning course in methods of education and measurement which is computer based. In this project TB will focus on general educational tendencies not specifically related to the medical profession.

6. It is essential for TB members to participate in International Conferences and interact with international experts such as the National Board of Medical Examiners (NBME), USA, Centre for Medical Education(CME), Dundee, Scotland.

7. Standardised patients programmes were presented for the assessment of clinical skills. Participants were asked to comment on the possibility of administering such an exam as part of the MLE procedure. This examination, will add a more comprehensive assessment of medical schools students in which knowledge as well as clinical skills and attitudes will be tested.
Tuesday February 8, 2000

In the second workshop day a number of topics were further presented (Appendix A).
Computer support of Medical Licensing examination.
Yuri Bogachkov (Appendix K)

Modern tendencies in Medical Schools Assessment.
Miriam Friedman Ben-David

Central discussion issues included role of Educational Commission in the System of Licensing Examinations

1. General information about Educational Commission

The Educational Commission will define the overall policy in development of State standards in Higher Education, and will carry the order of the Minister of Health. The order defines the committee members and endorses the commission existence and regulates its functioning (responsibilities, rights, place in MLE system, etc). The general structure of the commission will fit the structure of MLE. The commission will co-ordinate the efforts of development of State standards in Higher Education, including MLE, description of specialists and curriculum development. The TB staff will manage the system and provide technical and intellectual support. The commission will report to the Board of Education and Medical Science of the Ministry of Health and its functioning will be subject to current legislation. Discipline represented will be Medicine, Dentistry and Pharmacy. The speciality members will rotate.

2. Mission of the Education Commission

? To provide professional expertise of State Standards documents in Medicine and Dentistry (requirements to professional descriptions, curricula and diagnostic tools).

? To provide expert assessment of introduction of State Standards in Medicine and Dentistry.

? To control the objectivity of assessment. To defend rights of examinees. (As you put – Control of the MLE procedure and implementation to ensure equal opportunities)
3. Responsibilities & rights of the Educational Commission

Responsibilities
? Commission will ensure quality professional expertise of ways of introduction of state standards in Medicine and Dentistry.

? Commission will ensure strict fulfilment of item-review procedures.

? Committee will ensure fulfilment of plans according to the deadlines.

? Committee will oversee timely submission of expert reports.

? Committee will set the marking grade.

? Commission will provide expertise licensing examination results for improvement of medical curriculum.

Rights

? Commission may provide expertise of licensing examination results for improvement of medical curriculum

? Commission may promote adaptation of foreign practices and of modern technologies in education and measurement

? Commission may administer additional item-review if requirements were not met

? Commission may annual the results of licensing examinations if the technology is broken and to prescribe repetition of test examinations…

4. Organization of work

? Commission will meet once a year

? Review of test materials is done in two stages: 1- review for meeting state standards – is executed by specialty research laboratories; and 2 – item review.

? Committee will prepare review reports

? Faculty members’ work on item review is part of their academic work

? Logistics of Commission meetings (collection of items, printing, copying, computer support, etc) is maintained by Testing Board

Following information from the report represents different comments from presenter and participants.

? Committee will define the schedule of MLE administration

? Standards will be set low at the beginning

? MLE as an independent external examination should not exclude the school’s internal examinations. On the contrary, school’s should be encouraged to develop high quality internal examinations.

? MLE introduction will lead the way to improvement in education and curricula reform

? MLE may lay the foundation for candidates ranking for the purpose of selection into competitive specialities

? A certificate of licensure provides the right to practice rather than a diploma

? Testing Board maintains the item bank. The committee experts are responsible for modifying the items.

Official representatives and Medical School Faculty as members of TB committee, will facilitate cross checking of each other’s work.

Additional submission points

? The system is automated to create tests, statistical reports and score reports. There is a possibility to have on-line testing – via the internet and develop distant sites for testing.

? Culture of medical education in Ukraine does not yet allow TB to introduce more than one version of MLE in one given administration.

? It was debated that item writing should be viewed as an educational activity for faculty, considering the time devoted to this activity.

? Security is still a problem. Protection of MLE booklets is the main issue.

? MLE allows standardisation of core knowledge. Basic and clinical skills should be taught as a vocational skill. Each school’s objective should be to assess the ability to use the knowledge. Professional and individual behaviour should be part of the general outcome scheme in medical schools. There is a new Journal in Ukraine “Medical Education”. Faculty should publish their ideas about educational reform. It is important that medical schools will upgrade the quality of their internal examination to match MLE. MLE can give international weight to the diploma in Ukraine.

Report on the Sectorial Studies # 1 & 2

Sectorial Study #1 – Build a computer model of introduction and functioning of licensing examination

Miriam Friedman met with Marina Mrouga and Yury Bogachkov to discuss the computer model for licensure testing. The model should incorporate a broad paradigm of testing aspects and should include quantifiable parameters as evidence of test validity. The overall question is: why do we use the testing and how it impacts the health system?

The model stated 3 main parameters

1. state situation
2. public health situation
3. parameters of public system we attempt to influence

The following system of testing has to be incorporated in the model:

1. Technology
2. Parameters of Technology
3. Define the way we use technology
4. Local and outside sites which use the technology

The model should also include obstacles to change:

Licensing Test technology List of parameters to be changed





Define the obstacles to change




Mechanism for quantifying the parameters


List of parameters that should be changed and quantified:

1. Legislation
2. Doctors quality
3. Quality of health services
4. National health statistics
5. Manpower distribution
6. Career choice
7. Attrition rate from medical school
8. Educational reform
9. Quality of education
10. Influence on health related fields
11. Quality of teachers
12. Methods of learning
13. Structure and ratio within medical schools
14. Reform in methods of admission
15. Changes within the private and state sector in medicine
16. Increase in motivation and better preparation for the test
17. Better mechanism to protect patients

The computer model will also incorporate a mapping component which will document standards of the profession, elements of the programme and the testing components.

Sectorial Study #2 – Development of a Business Plan of the Testing Board

M Friedman met with TB staff members to discuss the TB Business plan (Appendix L)

A business plan template was presented to TB staff. Start-up costs, operating costs and test development costs were considered for generating the overall cost of MLE. The division by number of examinees produces the fee per candidate. Projected costs for item bank expansions, numbers of examinees and the cost of the development of new programmes were incorporated in the 5 year projected business plan. TB staff will finalise the plan.

Plan a Five Day Workshop at the Centre for Medical Education, Dundee

An outline of the 5 day workshop in Dundee was discussed with TB staff. The workshops will include topics on trends in Medical Education, assessment technologies, distance learning, evidence based medicine. Appendix M outlines the workshop agenda from March 18 to March 23 for 6 TB staff.

A meeting with delegates of the Commission of European Committees in Kyiv

Irina Bulakh, Marina Mrouga and Miriam Friedman met with the programme officers of the Delegation of European Communities in Ukraine: Andriy Sidorenko and Sergiy Ladny, on Thursday morning February 10th.

The secotorial Studies 1 & 2 were presented and plans for the April seminar were reported. A questionnaire was prepared by TB staff (Appendix N) and was sent to Medical School faculty. The replies will be used as discussion points in the April seminar. Self regulations of professional association and the impact on other professions was also discussed. Budgetary aspects of the grant were clarified.
Strength of the MLE programme

I found my visit to Kyiv a fascinating experience from a variety of aspects:

1. The success of TB staff to introduce changes in licensure procedures within a culture which is not entirely ready for such a change, was quite impressive.

2 The positive communication links with the MOH provide good support and backing to the TB staff.

2. The high level expertise of TB staff in spite the limited budget could only be explained by high motivation and long hours of work.

3. The liaison with medical schools via faculty involvement in item writing, is an excellent model for educational reform.

4. The impact on other health professions such as Pharmacy and Nursing is another indication of the success of the MLE programme.

5. The broad scope of the TB plan to develop innovative testings in all phases of Medical Licensure, demonstrated the seriousness of this organisation.

6. The comprehensive understanding of TB staff of the impact of licensure on the quality of care in Ukraine, is unique and has far reaching implications.
Recommendation

1. TB needs to re-organise its structure, define responsibilities, establish committees which will support its testing and educational activity. An International Committee could be instrumental in obtaining international recognition.

2. TB staff need to be exposed to the latest in test technologies through conferences and international experts.

3. Alternative forms of MLE should be eventually introduced to minimise the threat of security breach.

4. The standard for MLE should be revisited annually to find ways to elevate the standards and yet increase the pass rate. Currently the passing score is low and may convey the wrong messages with respect to minimal competence. 5. TB staff should work closely with medical schools to identify causes for low performance of their students. 6. TB staff should opt for pass/fail policies for local and foreign candidates. 7.The broader the educational scope of TB, the more influence MLE will have on the health system in Ukraine. 8.It is recommended to expand MLE by introducing a clinical skills assessment employing standardised patients. List of Appendix’s Appendix A Tacis Bistro/SSTA Seminar Programme Appendix B Schedule of Activities – Prof. II Hart Appendix C Testing Board Structure Appendix D Running the Testing Board-Prof. M. Friedman Ben-David Appendix E Running the Testing Board- Prof. M. Friedman Ben-David Appendix F List of Participants Appendix G Medical Licensing – Professor Y. Voronenko Appendix H1 Prospective Plan of TB – Dr I. Bulakh Appendix H 2 Bistro project – Dr Marina Mrouga Appendix I Route of Medical Licensure -Prof. M. Friedman Ben-David Appendix J Innovations in Assessment Prof. M. Friedman Ben-David Appendix K Computer Support – Yury Bugachkov Appendix L Business Plan for Test Centre – Prof. M. Friedman Ben-David Appendix M Proposed Programme – Dundee visit March 18-23, 2000 Appendix N Testing Board Questionnaire.

Звіт професора Єна Харта

Загальні коментарі та пропозиції

Неможливо не подякувати доктору Булах та всім співробітникам з Центру Тестування за той обсяг роботи, який вони виконали за три короткі роки та за якість їхньої продукції.

Вони створили національний центр стандартизованого тестування, який є унікальним у Східній Європі. Центр тестування був підтриманий Урядом і підпорядковується Міністерству охорони здоров’я. Він працює в умовах нестабільного фінансування і започатковувався в умовах деякої антипатії з боку головних споживачів їх продукту - медичних закладів та їх випускників.

Завдяки послідовній стратегії включення медичних закладів до процесу в якості партнерів, антипатія почала зменшуватися. Ця стратегія також справила бажаний вплив на навчальний план цих медичних закладів. Як завжди, найбільш важливий двигун, який спонукає студентів навчатися та викладацький склад викладати, - це система оцінювання.

Таким чином, Центр тестування пройшов через кризові роки національного становлення, антипатії та напруження і зараз вступає в період свого зрілого віку, оскільки він набирає досвіду, його більш приймають та поважають.

Все що робиться, робиться добре, але Центр безперечно зможе працювати ще краще за стабільного фінансування та ефективної інфраструктури. Це без сумніву важливо для зростання та розширення підприємства та його діяльності.

Консультативна група Центру медичної освіти Університету Данді дасть поради та рекомендації протягом наступних місяців для того, щоб допомогти Центру нарощувати та розвивати нові тестові ініціативи.

Після короткого ознайомчого візиту ще передчасно давати чіткі та детальні рекомендації - це буде зроблено протягом наступних декількох місяців після нашої оцінки ситуації та можливостей майбутнього розвитку Центру тестування. Не дивлячись на це, навіть на цій ранній стадії можна порекомендувати наступне:

• Оскільки поточний 5 річний стратегічний план підходить до свого завершення, важливо щоб Центр тестування продовжував розвиватися в майбутньому
•  Мета існування
  • Викладення місії
  • Стратегічний план на наступні 5 років
  • Заходи мають бути вжиті зараз, з тим щоб сприяти розвитку банку тестових завдань та апробації інших форматів тестових завдань вибіркового типу, наприклад розширеного вибору. Без сумніву повинно пройти декілька років, щоб можна було дійсно використовувати їх в обов’язкових іспитах, але цей процес необхідно розпочати зараз. Ці завдання можна робити на більш високому рівні ніж завдання типу А з точки зору розв’язання поставленої задачі.
• Більш нова стратегія для комп’ютеризації одночасного (on line) тестування повинна бути прийнята. Починайте працювати над цим зараз.
• Продовжувати працювати над пошуком інноваційних джерел фінансування, включаючи -
  • Продаж послуг для інших дисциплін та напрямів в межах країни
  • Продаж послуг та ваших професійних знань тестування на великих масштабах іншим країнам
  • Фонди, які мають відношення до міжнародної освіти та охорони здоров’я
  • Міжнародні урядові агентства по сприянню охороні здоров’я та освіти

Протягом наступних місяців консультанти з Данді будуть працювати з вами, щоб внести свої пропозиції та розглянути деякі з цих рекомендацій.

На заключення, я хочу висловити подяку та вдячність доктору Булах та всім співробітникам Центру Тестування за їх невтомну допомогу, відповіді на всі мої нескінчені запитання та за їх надзвичайну гостинність.

Я також хочу подякувати співробітникам EU TACIS BISTRO/SSTA за їх допомогу та підтримку.

Експерт Європейської Комісії

Проект TASIC BISTRO/SSTA

BIS/98/025/006

Ян Харт

Заслужений професор

Identification Phase

Consultant’s Report

Visit to Kyiv, November 9-14, 1999

This report was prepared with the financial assistance from the Commission of European Communities. The views expressed herein are those of the Consultant and do not represent any official view of the Commission.

Introduction

This, the first report in the TACIS BISTRO/SSTA supported Project on the “Development of an Independent Test Board to Introduce a Licensure System into the Health Sector in Ukraine”, is based on the preliminary visit of the EU Consultant (Professor Ian R Hart) from the Centre for Medical Education in Dundee, to the Testing Board in Kyiv, Ukraine. The visit took place from Tuesday 9th to Sunday the14th of November and consisted of three formal project working days and two days of informal meetings and preliminary report writing.

An outline of Professor Hart’s schedule during the visit is attached as Appendix A.

This report addresses the first 5 activities included in the Schedule of Activities (Appendix B) as modified following discussions during the current visit.

These activities were:

The Report follows, organized under the following headings:

1. Overview of Licensure and Certification System in Ukraine (IP-1)
2. Progress in Developing Standardised Testing for Health Trainees in Ukraine (IP-1)
3. Current Status and Staffing of the Testing Board (IP-3 & 4)
4. Revised Timetable and Programme of Collaboration (IP-2 & 5)
5. General Comments and Recommendations

1. Overview of Licensure and Certification System in Ukraine (IP-1)

The following comments are based on discussion with Dr Bulakh and the staff of the Testing Board and some very helpful documents prepared for me by the staff. These documents are reproduced in Appendices C1-3.

Ukraine has a long history of education in the health sciences and its oldest medical schools date back to the late 1700’s. There are now 22 schools graduating either doctors, dentists, pharmacists or all three. Eighteen of these institutions graduate medical doctors.

The medical schools have had traditional curricula which had changed very little until the past decade. The assessment process has also been very traditional and based almost totally on written (essay) and oral examinations. There has been no tradition of standardized testing in medical schools or for graduation from medical schools, and indeed, no real experience with standardized testing in higher education in general.

Nineteen ninety one (1991) saw the introduction of major reforms in the Health Care system in Ukraine with the promotion of a merit-based personnel selection in a free labour market. This shift in the system now requires medical graduates to meet quality standards – a difficult goal to achieve when the standards to be met have not been explicitly defined and the assessment tools incapable of meeting standards of reliability and validity.

Over the intervening years attempts to change the evaluation system have occurred – as have some changes in the education of medical students.

• 1992-93 – the Ministry of Health and some medical educators looked at the medical education systems in several Western countries including USA, UK, Germany and Canada.

• 1992 – a Multiple Choice Question examination was established at the end of the 2 year “internship” that all graduation students must take. This is actually produced in several different forms depending in the special area each graduate has taken internship training in. The bank has not grown and is probably public knowledge by now. No meaningful psychometrics are produced. This examination is not administered by the Testing Board, but by another government agency.

• 1993 – the International Biomedical Agency acted as a site for the US ECFMG Clinical Skills pilot examination.

• 1993 – the Ministry of Education worked with the US agency ETS (Educational Testing Service) to implement something like the ETS Scholastic Aptitude Test (SAT) for high school students in the Ukraine. It had major problems and was abandoned. The experience left standardized testing and psychometrics with a bad name in the country

• In 1992, 1994 and 1996 major restructuring took place in the curricula of the medical schools, and of the internship. These were structural changes rather than functional, and there was no major changes in what was taught and the teaching/learning methods used. Such constant change in the system has led to resistance to further changes, especially to the use of standardized testing.

In 1996 the Government introduced a national program for medical education assessment reform in line with state policy for reform of the higher education system.

This led to the introduction of a project to establish a Medical Licensing Examination (MLE) with two basic principles –

1. Medical educators must be trained in the need and the skills for creating standardized tests, and
2. The examination must be piloted satisfactorily before being made mandatory for licensure.

The Testing Board was established in 1996 with the mandate to develop standardized tests for the licensing of doctors, dentists and pharmacists.

The Route to Medical Practice in Ukraine

Undergraduate

All 18 medical schools have a 6 year curriculum. The first three years are in the sciences basic to medicine and the last three years are clinical.

Until 1991 the curriculum undertaken by an individual differed depending on whether they were place in a Public Health stream or other possible specialty streams. In the past several years the students have a choice between a medicine stream or a surgical stream.

Testing is by written and oral examinations. To graduate, a student has to pass all the appropriate in-school examinations.

Postgraduate – Internship

Having passed all the local school examinations most students enter either a medical or surgical internship with specific subspecialty training. These internships are for two years.

Postgraduate – Magister

Students who have shown special aptitudes are chosen for further academic training and enter a 3 year program (including the Internship 2 years) with special academic training in their chosen specialty/subspecialty. In addition to having to pass the Internship Examination at the end of two years, these doctors have to complete a project and thesis before their training is considered complete.

Internship Examination

This is taken by every graduate at the end of two years of postgraduate training. Passing this gives them the right to practice medicine.

The long-term goal of the Medical Licensing Examination Implementation Project is to improve the basis of the training of medical specialists in Ukraine through the incorporation of modern and effective educational methods and strategies.

The Objectives of the Project are to:

• define the level of minimal competence expected of medical graduates
• develop and pilot standardized assessment of medical graduates
• create an infrastructure for standardized assessment at a national level
• develop effective feedback mechanisms for the improvement of medical teaching.

The Ministry of Health independent Testing Board has been responsible for meeting these objectives and has been in operation since 1996. The Board is responsible for the production, administration, scoring and psychometrics and banking of quality multiple choice items and examinations in the medical, dental and pharmacy disciplines at the national level. It is a bilingual organization – Ukrainian and Russian. All of its tests and test items must be available in both official languages.

Since piloting its first examination in 1996, it has run integrated, standardized examinations at both the basic science Krok 1 (Step 1) level at the end of third year and final year Krok 2(Step 2) level in both Medicine and Dentistry. The total number of graduates from the 18 Medical schools is approximately 7,000 each year. From the 12 Dental schools, approximately 1500 graduate each year.

As can be seen from the following Table (extracted from Document 3 in Appendix C3), in 1999, pilot examinations were run at both Krok 1 and 2 levels for over 14,000 medical students and a mandatory high stakes pass/fail final (Krok 2) examination was run for 1516 dental students.

The Krock 2 Dental examination will continue to be mandatory for graduation from dental school. The examination is of high reliability and stable in performance.

It is planned to run the Medicine examinations in 2000 again as a pilot with more (300) questions in each in order to get reliability data on the blueprint sub-scores, i.e. to have enough questions for each discipline within the overall examinations to give reproducibility in the results. The plan is then to run the Medicine Krock 1 & 2 examinations in the year 2001 as mandatory examinations.

Dr Miriam Friedman, during her visit and workshop in January or February, 2000 will be looking more into the details of the item development process, databanking infrastructure and process, examination administration and the psychometrics of the items and the examinations.

In the limited time available to look at these issues during this visit, it was my impression that, in general all of these components of a testing centre appear to functioning at an acceptable level. Later in this consultancy we hope to be able to give specific recommendations regarding increasing the efficiency (and cost-effectiveness) of the overall process. In addition, we anticipate in future reports broadening the scope of types of test items to be used in the examination.

Some comments on the process and infrastructure follow:

All of the questions currently in the databank (about 4,000 in total) and used in the examinations are Type A multiple choice questions – 95% or so being one correct answer from 5 choices, the remainder one from four.

The items are written by approximately 20 to 30 faculty members at each school. These have all undertaken training on item writing organized by the Testing Board.

About 200 questions are commissioned each year from each school. Each group are asked for items in specific areas – driven by the needs of the databank.

Questions written by each content expert are supposed to be vetted by the other experts in that department.

New questions are vetted at the Testing Board for format, translated and entered into the bank tagged with all other appropriate information regarding them.

New questions are then vetted by appropriate content experts from the central examination committee who allocate them to the various blueprint subject areas.

Each examination is set according to a predetermined blueprint.

Examinations are administered by paper and pencil and marked by OMR scanning with random hand checking.

After the examination, each item’s difficulty index, point biserial and the distribution of distractors chosen are entered into the bank. Occasional questions are removed from the scoring process because of faulty DI or PB.

The reliability index used is Crohnbach’s. The Medicine pilots and the mandatory Dental Krock 2 examinations taken in 1999 had reliability indices in the range of 0.95.

Pass/fail decisions are based on an Absolute Standard using the well-recognized Nedelsky and Hofstee methods.

Feedback to students is given in the form of comparison performance in each subject area in the blueprint.

Scoring and feedback turnaround time to students and faculty is 10 days.

The pilot examination questions are made available after the examination to both students and faculty.

Questions in the databank are culled regularly by the content experts on the central examination committee.

3. Current Status and Staffing of the Testing Board (IP-3 & 4)

Testing Board Staff Structure

The current staffing structure and responsibilities of the Testing Board is as shown above. As constituted, it appears to be filling the current functions of the Board adequately, though there is no doubt that the staff are working very hard to keep up.

At this early stage in the consultancy, which is mainly an information gathering phase of the work, it is too early to make recommendations regarding the reassignment of current staff or the recruitment of new staff.

This issue will be addressed and recommendations made at the time of the Workshop at the CME in Dundee and after the visit of Dr Friedman early next year.

4. Revised Timetable and Programme of Collaboration (IP-2 & 5)

The above timetable and schedule of activities is also found as Appendix B

There will be two Workshops and a final round table Seminar involving CME/Dundee personnel.

Workshop in Kyiv – January or February

The Goals:

• To Testing Board staff how to effectively run the subdivisions of the Testing Board with stress on human resources management, including principles of distribution of duties, responsibilities and teamwork using Western assessment centers as an example
• To strengthen Testing Board co-operation with medical schools directed towards use of licensing examination results to facilitate positive changes in curriculum, improvement of teaching methodology and increase of students' motivation to learning
• To promote the development of licensure system in health sector and application of standardized testing for measurement of professional competence level of specialists in Ukraine through collaboration with other stakeholders

5. General Comments and Suggestions

Dr. Bulahk and all the staff at the Testing Board are to be congratulated on how far they have come in 3 short years and on the quality of their product.

They have built a centre for standardized testing that is unique in Eastern Europe and one of the few national standardized testing centers anywhere in the world. This they have done with the support of the Government through the Ministry of Health and in an environment of fragile funding and some degree of antipathy from the major consumers of their product – the medical schools and their graduates.

Due to the carefully pursued strategy of getting buy-in from the medical schools by involving them as partners in the process that antipathy appears to be on the wane. Their strategy is also starting to have the desired effect on the curricula of the medical schools. As always, the most important driver of student learning and faculty teaching is the assessment system.

That said, having gone through the pioneering years of crises, antipathy and struggle, the Testing Board is now entering its early adulthood period as it matures and becomes more accepted and respected.

It is doing what it does well, but it could almost certainly do it better with a more efficient infrastructure and stable funding. And these things are doubly important to the growth and expansion of the enterprise and its activities.

The University of Dundee Centre for Medical Education Consulting group looks forward to giving advice and making recommendations over the next few months to assist the Board with that growth and the development of new testing initiatives.

Following this brief and very preliminary visit it is premature to make many firm and detailed recommendations – these will come as our appreciation of the situation and future of the Testing Board develops over the next few months. However, even at this early stage some suggestions are in order:

• As the current 5 year strategic plan comes to a close it is important that the maturing Testing Board develops in the near future –
  • A vision
  • A Mission Statement
  • A strategic plan for the next five years

• Steps should be taken now to look at developing a bank and piloting some other types of multiple choice items e.g. extended matching questions. Although it may be a few years before they are actually used in mandatory examinations the process should start now. Such items can get to higher level thinking and problem solving better than A type questions.

• An earlier rather than later strategy for on-line computerized testing should be adopted. Start working on it now.

• Continue to build on the strategy to seek innovative sources of funding including –
  • Selling services to other disciplines and groups within the country
  • Selling services and your expertise in large scale testing to other countries
  • International education and health related Foundations
  • International education and health related government agencies

Over the next few months the Dundee CME consultants will collaborate with you in moving forward with some of these suggestions.

Finally, I would like to extend my thanks and appreciation to Dr Bulahk and all the staff at the Testing Board for their untiring assistance to me in answering the unending questions that I posed to them; and for their wonderful hospitality.

I would also like to thank the EU TACIS BISTRO/SSTA staff members for their assistance and support.

Professor Ian R Hart

EU expert Project

November 22, 1999