Assessment Modalities

The assessment program requirements are documented in the Training Program Requirements policy and the Fellowship Training Program Handbook. The specific requirements and information for each of the training program assessment modalities are detailed in this handbook.

The assessment modalities are:

• Mini-Clinical Evaluation Exercise (miniCEX)
• Multi-Source Feedback (MSF)
• Multiple-Choice Question assessment (MCQ)
• Case Based Discussion (CBD)
• Structured Assessment using Multiple Patient Scenarios (StAMPS)
• Logbooks
• Projects

Mini Clinical Evaluation Exercise

Introduction

The Mini-Clinical Evaluation Exercise (miniCEX) is a workplace based assessment used to evaluate a candidate’s clinical performance in real life clinical settings.

All candidates training on an ACRRM training program must complete formative MiniCEX assessments for CGT and when undertaking AST in a clinical discipline. Candidates must be in posts accredited by ACRRM for the corresponding stage of training or specific AST discipline. Formative miniCEXs are conducted progressively throughout training.

In exceptional circumstances a candidate working towards Fellowship may be required to complete a summative miniCEX. This is determined by the College Censor-in-Chief (CIC).

The miniCEX training course for Clinicians - online course is designed for clinicians who are asked to conduct formative miniCEX assessments with ACRRM registrars.

Requirements

Core Generalist Training (CGT)

Nine (9) miniCEX reviews must be submitted to obtain ‘satisfactory completion’ for formative assessment. However, there is no barrier to completing more than 9 MiniCEX if additional learning opportunities are available.

MiniCEX reviews are completed progressively during CGT, a plan of 3 per year of training would be appropriate to allow time for reflection on feedback and further clinical skill development. MiniCEX may include a combination of face-to-face and telehealth consults, but requirements cannot be met through telehealth consults only.

Consults must include a:

• Range of types of consults, age groups and a mix of genders.

• Cover a minimum of five system focuses for the sonsultations:

  • cardiovascular
  • respiratory
  • abdominal
  • neurological
  • endocrine
  • musculoskeletal region
  • mental health
  • neonatal/paediatric
  • antenatal (first visit)

• Detailed history taking of at least one (1) new patient or detailed updating patient database information on a returning patient (of at least medium complexity).

The physical examinations are required to be undertaken only in the context of a face-to-face patient consultation. A ‘Physical Exam Reference’ document is provided for guidance on undertaking a systematic physical assessment.

A formative MiniCEX can be conducted within the context of the candidate’s medical educator visit or at any time at the instigation of the candidate or supervisor. The MiniCEX reviews are to be conducted by a doctor, meeting one of the following criteria:

• FACRRM
• an ACRRM accredited supervisor

The assessor may be onsite viewing the consults face-to-face or offsite viewing the consults virtually.

At a minimum three (3) different assessors are required to complete the nine (9) MiniCEX assessments. Three (3) of the MiniCEX assessments must be conducted by a Medical Educator from an ACRRM accredited training organisation (who does not work in the same workplace as the candidate).

Advanced Specialised Training (AST)

Candidates undertaking AST in Aboriginal and Torres Strait Islander Health, Adult Internal Medicine, Emergency Medicine, Mental Health, Paediatrics, Palliative Care, Remote Medicine and Surgery are required to have a formative MiniCEX conducted on a minimum of five (5) patient interactions (consults) during the AST component of their training.

The MiniCEX consults should be undertaken progressively during training and include a range of types of consults, age groups and mix of genders. The same principles apply around face-to-face and telehealth consults as described for CGT above.

A formative MiniCEX can be conducted at any time at the instigation of the candidate or supervisor. The MiniCEX assessments must be conducted by a doctor who is an ACRRM accredited supervisor or mentor for the post or holds a Fellowship of the Specialist discipline relevant to the AST.

MiniCEX format

MiniCEX consists of the following:

• A short encounter between a candidate and patient which is observed by a supervisor. This encounter generally consists of the following components, a focused history taking focused clinical examination and assessment and takes approximately 15-20 minutes.
• Discussion of patient management and provision of oral and written feedback to the candidate by the supervisor to assist the candidate in planning for future patient encounters. This takes approximately 5-10 minutes.

The process is the same for summative MiniCEX except the assessment is undertaken by a College appointed assessor and candidates are not provided with feedback prior to ratification of results.

There are five categories:

1. Communication skills
2. History taking
3. Physical examination
4. Clinical management
5. Professionalism
6. Overall clinical competence

For each consultation the proportion of each of these will vary but as many as possible should be assessed for each MiniCEX.

For each consultation, each category is assessed as:

• Beginning
• Progressing
• Achieved
• Exceeded

Using the following domans:

1. Communication

   • Patient centered communication evident. Built trust and rapport with patient.
   • Showed empathy and respect. Asked patient for their story.
   • Explored patient issue using a range of relevant question types.
   • Considered and discussed the impact of presentation on patient function.
   • Flexible in approach. Considered cultural values, attitudes and beliefs.
   • Explained aspects of care clearly.
   • Involved patient in decision making and provided appropriate advice.

2. History taking

   • Obtained a clinical history including presenting problems, epidemiology and cultural context.
   • Questions focused and appropriate.

3. Physical examination

   • Sound assessment conducted and several key differentials considered.
   • Relevant signs and symptoms covered.
   • Assessment organised, logical and efficient.
   • Patient comfort and safety considered.

4. Clinical management

   • An appropriate range of evidence gathered, and most plausible diagnosis provided to the patient.
   • All required appropriate tests arranged.
   • Short-term management and possible long-term management plan appropriate and discussed with patient.
   • Follow-up arranged.
   • Clearly addressed ethical / potential legal / work cover issues.

5. Professionalism

   • Ensured patient privacy and confidentiality.
   • Demonstrated a commitment to teamwork, collaboration, coordination and continuity of care.
   • Critically appraised own performance.
   • Clinical documentation is in accordance with professional standards.
   • Provided accurate and ethical certification for sickness, employment, social benefits and other purposes.

6. Overall

   • Overall approach systematic and consistently competent across marking categories.
   • Made clear efforts to ensure patient comfort and safety and to reduce risks where appropriate.
   • Communication skills effective. Patient involved in decision making.
   • Diagnosis sound and based on information gathered.
   • Appropriate history and assessment undertaken.
   • Relevant further tests arranged to confirm diagnosis as required.
   • Management appropriate and includes short and some long-term recommendations based on information gathered.

More detailed descriptors for the competency standards for Fellowship- beginning/progressing/achieved are provided in the Rural Generalist Curriculum.

For the purpose of fulfilling the mandatory requirements of the miniCEX, it is expected that the physical examination will be a thorough and complete assessment of the relevant system. See Physical Exam Reference for guidance on the standard expected for physical assessment.

The MiniCEX form, patient consent proforma and Physical Exam Reference are available on the Assessment Resources webpage.

Roles and responsibilities

Candidate

The candidate is responsible for ensuring that they meet the mandatory requirements for MiniCEX. A candidate must ensure that completed MiniCEX forms are provided to the College.

Assessor

The assessor observes and scores consultation using ACRRM MiniCEX form. The assessor provides oral and written feedback to the candidate. The completed form is given to the candidate and a copy submitted to the College Training team via the supervisor.

Multi-Source Feedback

Introduction

The Multi-Source Feedback (MSF) is used widely in a range of professions. The College includes MSF in the assessment program as a valid and reliable method of assessing interpersonal and professional behaviour, development and clinical skills.

All candidates must complete the MSF assessment during core generalist training. The assessment must be undertaken in a post accredited by the College for CGT. MSF cannot be undertaken in a post that may be recognised for CGT but not accredited by the College, these include posts accredited by a Postgraduate Medical Council or other Specialist Medical Colleges.

Requirements

Candidates are required to demonstrate ‘satisfactory completion’ of at least one (1) MSF.

Satisfactory completion requires submission of:

• a completed MSF report covering the two (2) components
• a completed reflective exercise and
• evidence of discussion about the report results with a Medical Educator and
• remediation if required.

The first three (3) components of the MSF process must have fulfilled within four (4) months from the date of enrolment in MSF.

The MSF assessment is not awarded a pass/fail standard, however if concerns are raised in any component of the MSF, the training organisation or College may require the candidate to repeat the MSF or undertake another type of assessment to gain further information or to determine if remediation has been effective.

Once all the components of the MSF, including remediation if required, have been completed, the MSF outcome is presented to the BoE for ratification. The BoE determines ‘Satisfactory Completion’.

The MSF must be conducted through Client Focused Evaluations Program (CFEP). The ACRRM version of the MSF is the preferred tool. Alternate versions of MSF offered by CFEP are also acceptable. Both components of the tool must be completed; however, it is acceptable to undertake the Colleague Feedback Evaluation Tool (CFET) components and Doctors’ Interpersonal Skills Questionnaire DISQ separately. If completed separately a completed reflective exercise is required for each component. See the CFEP website for further information.

MSF format

The MSF tool consists of two (2) components:

1. A colleague assessment tool and a self-assessment tool; (collectively known as Colleague Feedback Evaluation Tool — CFET) and

2. A patient assessment tool (Doctors Interpersonal Skills Questionnaire — DISQ).

   Colleague tool\ The colleague tool involves a response from a minimum of 12 nominated colleagues, in order to obtain more complete data, 15 or more colleagues are preferred to participate. Colleagues are required to rate the candidate in 20 different areas. There is also a provision for qualitative comments.

   Self-assessment tool\ Completion of the self-assessment is a mandatory requirement for the MSF.

   Patient tool\ The patient tool involves a minimum of 30 patients participating in an anonymous questionnaire. Patients are required to rate the candidate in 12 different areas. There is also a provision for qualitative comments.

MSF report

A candidate ‘mean’ score for each question is provided. National means and performance bands have been calculated from data generated from previous ACRRM candidates.

See the How to guide for reading ACRRM MSF reports.

Roles & Responsibilities

It is the candidate’s responsibility to:

• Enrol and inform CFEP that they are undertaking MSF for as part of ACRRM training requirement.

• Advise CFEP of the practice environment they will be working in whilst undertaking the MSF assessment.

  Candidates undertaking their MSF assessment in an environment where a significant proportion of the patients may have trouble in completing a questionnaire are able to ask an appropriate person. Candidates can also request CFEP to send an alternative DISQ patient survey for an Aboriginal Medical Service (AMS).

Self-assessment tool

• Candidates must submit a completed self-reflective exercise and evidence of discussion with Medical Educator to the College within two (2) months from the date the MSF report was received.
• In extenuating circumstances, an extension of time may be considered. The candidate must submit an Application for Special Consideration to the College, containing a written statement for the reasons for the requested extension. This must be submitted prior to the expiration of the deadline.
• If the deadline is exceeded without prior approval, the College reserves the right to report an ‘incomplete’ grade. In this instance the candidate will be required to re-enrol, pay the MSF assessment fee and recommence the process.

Colleague tool

• Candidates must complete and return the colleague list providing names and email addresses of at least 15 colleagues.
• A personal email address must be provided for each nominated colleague. Email addresses must be independently verifiable by CFEP. Generic email addresses such as practicemanager@ or reception@ will not be accepted due to security reasons.

CFEP suggests nominating:

• Five doctors

  • three GP colleagues who are close to the candidate, e.g. neighbouring GPs, partners
  • two doctors from outside of the candidate’s immediate practice, e.g. consultant, candidates

• Five Non-Medical Clinical Colleagues

  • this should include a mix of people within a candidate’s practice and elsewhere, e.g. practice nurses, pharmacists, physiotherapists, midwives

• Five managerial or administrative staff

  • this should include a mix of people within the candidate’s practice and from elsewhere, e.g. practice manager, reception staff, managerial staff of the local Primary Health Organisation.

Patient tool The candidate is responsible for contracting a member of staff, e.g. a receptionist or an administrative officer to collect the completed patient questionnaires. This must be a person who has an opportunity to see the candidate’s patients after consultations.

The candidate is responsible for ensuring that the contracted person is provided with instructions to ensure this process is undertaken anonymously and in an ethical and professional manner, as follows:

• the candidate must hand the patient questionnaires with enough envelopes and staff guidelines to the contracted to the staff member for collection,
• the patient must not be advised of the questionnaire or invited to participate until after a consultation has been conducted,
• a confidential process must be adopted (a sealed box) for collecting completed questionnaires,
• patients should preferably complete the questionnaire whilst in the waiting room before they leave the premises,
• an envelope must be provided to each patient in which they must place their completed questionnaire,
• completed questionnaires must be handed back in a sealed envelope,
• if a patient insists on taking questionnaires away to complete, these must be returned the following day,
• under no circumstances should the candidate be given access to individual questionnaires,
• sealed questionnaires are not to be opened by anyone, and
• when a minimum of 30 questionnaires has been completed, these should be posted to CFEP in the large envelope provided.

CFEP will also provide instructions for this process when they send patient questionnaires to candidates.

Summary of MSF process

Multiple Choice Question Assessment

Introduction

The Multiple-Choice Question (MCQ) assessment assesses clinical reasoning and application of clinical knowledge.

reasoning and applied clinical knowledge.

The MCQ assessment focuses on assessing the ability to manage medical care in a rural or remote environment. The assessment aims to cover all domains of rural and remote practise and is one of the mandatory summative assessments for CGT.

Requirements

Candidates are required to achieve a pass in the MCQ assessment. The standard expected is that of a doctor practising at Fellowship level.

Standard setting for the MCQ assessment is based on the modified Angoff method. This involves setting a standard score for test items prior to the test, using judgements by experts based on the projected performance of ‘borderline candidates.’

There is a detailed extensive post assessment statistical analysis of the performance of each question and the overall exam. This process may result in some questions being removed from the assessment that don't meet statistical performance criteria.

MCQ format

The assessment is conducted over three hours and consists of 125 multiple-choice questions. The MCQ assessment is currently conducted using a pencil-and-paper format. All candidates must complete the assessment by completing an Answer Sheet. A completed Answer Sheet is returned via email to the College at the completion of the assessment by the invigilator. The Answer Sheet is marked automatically using the College assessment management system.

Venue and Invigilators

For information relevant to venue and invigilator requirements refer to Venues and Invigilators section of this handbook.

It is the candidate's responsibility to ensure that they can access a College approved venue (refer to page 3 of the Assessment Venue Requirements) on the day of the assessment.

All candidates undertake the MCQ on the same day and at the same time, regardless of their location. Candidates and invigilators will be notified of the assessment start time in Australian Eastern Standard Time (AEST). Each candidate and invigilator are advised to check their local time zone and adjust the start time if required.

Questions

The majority of questions mostly consist of a clinical case presentation, a brief targeted lead-in question and four options from which candidates are required to choose the single best option. The stem of the clinical case may include text and images. There are no negative marks for incorrect answers.

Questions are designed to address specific topics within the Rural Generalist Curriculum Questions focus on topics and concepts that are either common or important to the everyday experience of independent and safe rural and remote doctors practising in Australia. Questions are researched and written by practising rural doctors using up-to-date Australian references. The selected questions undergo quality assurance checks by the Lead Assessor.

Content

The assessment covers a range of primary care, acute care, community and hospital presentations. The patients represented include all genders, indigenous and non-indigenous patients and all age groups. The assessment samples content from the curriculum domains and learning areas.

The approximate percentage of questions for learning areas appearing in an assessment is outlined in the table below.

*The total of percentages is greater than 100% due to multiple domains being assessed multiple times within one assessment.

Preparation tools

Introduction to MCQ Assessment

The Introduction to MCQ Assessment online course is available to candidates who enrol in the MCQ assessment to give insight into the MCQ assessment.

MCQ familiarisation activity (MCQFA)

The MCQFA is available for candidates enrolled in a MCQ assessment. Candidates are advised by the Assessment team once the MCQFA is ready to be accessed.

The MCQFA provides candidates the opportunity to become familiar with the format of questions used in an actual MCQ assessment.

Candidates can complete the activity over a number of hours; however, it is suggested that candidates try and complete the MCQFA under 'simulate assessment conditions' using the same time limit and without accessing additional resources or support.

The MCQFA requires the single best answer for each question. One mark is awarded for each correct answer and there are no marks deducted for an incorrect answer.

Summary of MCQ process

Case Based Discussion

Introduction

Case Based Discussion (CBD) is an assessment of clinical reasoning and application of knowledge based in the registrar's real life clinical context. The candidate is required to demonstrate evidence of their clinical knowledge and how they apply that knowledge by demonstrating appropriate patient assessment, formulating differential diagnoses, ordering relevant investigations and applying appropriate management plans with partnership with the patient.

CBD is one of the summative assessment requirements for CGT (Core Generalist Training) and for AST in Palliative Care.

The case notes submitted for CBD assessment must be copies of the unedited original notes and contain clear evidence that the registrar is the doctor primarily responsible for patient care during the time period covered by the notes.

Registrars may also complete CBD as a formative assessment. The format of this assessment remains consistent for both formative and summative assessment.

Requirements and responsibilities

CBD assesses a total of six (6) cases. To pass the CBD assessment overall, candidates will need to achieve ‘at expected standard’ in five (5) of the six (6) cases.

All three (3) sessions are conducted irrespective of the outcome of each individual session. Results of individual session are not advised. The overall results are ratified by the Board of Examiners once all three (3) CBD sessions are completed. Candidates who do not obtain a ‘pass’ grade will be required to re-enrol, submit 12 new cases and complete the assessment again.

Core Generalist Training

Candidates must submit twelve (12) cases at CBD enrolment covering a minimum of six (6) of the areas listed below.

No more than three (3) cases can be from a single area.

Two (2) mental health cases must be included in the submission to ensure that at least one (1) case is suitable for the assessment.

• Aboriginal and Torres Strait Islander Health
• Addictive Behaviours
• Adult Internal Medicine
• Aged Care
• Dermatology
• Emergency
• Mental Health (Mandatory)
• Musculoskeletal
• Obstetrics and Gynaecology
• Occupational Health
• Ophthalmology
• Oral Health
• Palliative Care
• Paediatrics
• Rehabilitation
• Sexual Health
• Surgery

The latest consultation date must be no earlier than three (3) calendar months prior to the date of submission. The final consultation must have a direct link to the preceding consultations, for example it cannot be clinician-initiated telehealth consult to 'monitor progress' or discuss an old investigation result unless this is clearly noted in the preceding consultation notes.

All three (3) CBD sessions must be completed within a six (6) month period from the date of enrolment.

CBD clinical setting

This assessment is ideally suited to candidates who are working in a primary care context, however candidates working in an emergency or RFDS setting may still also complete this assessment.

Case notes may all be submitted from one post or submitted from a combination of different posts and/or settings. All case notes must meet the eligibility criteria for the setting of the patient encounters (see below). The case notes may be from consults conducted face to face or by teleconsultation providing the case requirements are met.

In all settings the case notes must contain evidence that the candidate is the doctor primarily responsible for the patients care.

Primary care setting

Cases should:

• Include clinical notes with a minimum of two (2) consultations for each case.
• Be at least medium level of complexity - not URTIs or medical certificates.
• Be sufficient to demonstrate continuity of care by the registrar.
• Include investigations.
• Include referrals.
• Include Health Care Plans.
• Include any other documents relevant to the case.

Candidates should be able to explain clinical reasoning on all aspects of the case.

Examples of relevant investigations include but are not limited to:

• results for review
• referral to specialists or other health care providers

Emergency or RFDS Setting

Cases should be described as high acuity and be from one of the following areas:

• One or more chronic illness, with severe exacerbation or progression
• Acute or chronic illness or injury which poses a threat to life or bodily function e.g. multiple trauma, acute MI, pulmonary embolism, severe respiratory distress, progressive severe rheumatoid arthritis, psychiatric illness, with potential threat to self or others, peritonitis, ARF
• An abrupt change in neurological status, e.g., seizure, TIA, weakness, sensory loss.

AST - Palliative care

Candidates must submit a total of 12 cases within 12 months from date of enrolment. Candidates have the option to submit all 12 cases at the time of enrolment, OR they can submit a minimum of four (4) cases at enrolment followed by four (4) cases at a time, within a 12-month period from the date of enrolment.

The latest consultation date must be no earlier than three (3) calendar months prior to the date of submission.

Candidates are required to submit cases where they have played a central role in at least four of the following areas:

1. Malignancy
2. Neurodegenerative disease
3. Organ failure
4. Frailty
5. Dementia
6. HIV/AIDs

One case may cover more than one of the areas listed above.

Candidates should expect assessors to explore any area of management including how they have developed / changed their understanding of dying through the course of the term, and how they plan to manage elements of self-care and personal well-being in their future practice.

Candidates will be assessed on their management of physical, psychological, intellectual and spiritual distress across all cases. To achieve competency candidates must demonstrate evidence of caring and empathy for a patient with a terminal illness and their family. This could also include staff who are caring for the dying. They must demonstrate an evidence-based approach to palliative care, pain management and treatment of associated behavioural problems.

Refer to the Palliative Care Learning Area in the Rural Generalist Curriculum for further information on required competencies, knowledge and skills.

CBD format

The cases submitted for assessment are those of actual patients that presented themselves to and were managed by the candidate with the last consult date being within 3 months of the submission date. The candidate is to provide evidence in the form of de-identified patient records for this assessment. Case notes may be print outs or PDF from the patient electronic clinical records and these original records cannot be altered apart from redaction of identifying information, including not copying and pasting investigations to the consult notes. Copies of handwritten clinical notes written on health service letterhead are permissible. Candidate generated case summaries without supporting documentation are not permissible.

Candidates must be the primary doctor in the cases for discussion. It is possible that additional consultations conducted by other doctors may be included where these contribute to the overall approach / case discussion.

When first submitting the cases for assessment, the cases uploaded are considered the ‘submission cases’ and cannot be swapped for other cases unless deemed to be out of date. If any cases are returned to the candidate to be de-identified, the same cases must then be de-identified and uploaded without removing / adding any new pages. No new cases will be permitted in this instance.

Submissions should contain enough clinical information for assessors to be able to undertake a comprehensive evaluation of the candidate's clinical knowledge and clinical reasoning skills. The last consultation listed in the case notes should be no older than three months at the time of submission.

The cases submitted are reviewed by an assessor. If the assessor is not satisfied that all cases are of suitable complexity, cover a satisfactory range of curriculum areas and have relevant supporting documents attached, all cases will be returned with ineligible cases flagged. The candidate will be required to resubmit 12 cases meeting all requirements and may include cases already submitted and not flagged as ineligible.

The CBD Lead Assessor, or delegate, will select six (6) cases for the assessment from the twelve (12) cases submitted. The assessment takes place over three (3) one-hour sessions. A different assessor conducts each session, and two (2) cases are discussed in each one-hour session. Candidates are notified of the cases to be discussed at the beginning of each session.

There are five (5) categories scored by the Assessor for each case that determines the global rating for that case:

• communication skills,
• history taking,
• physical assessment,
• clinical management, and
• professionalism.

Each case will be given a global CBD rating of either:

• ‘At expected standard for FACRRM’, or
• ‘Below expected standard for FACRRM’.

A candidate practicing ‘at the expected standard’ would be expected to:

• demonstrate an overall systematic approach and be consistently competent across grading categories
• make clear efforts to ensure patient comfort and safety and to reduce risks where appropriate,
• have effective communication skills,
• take an appropriate history and assessment,
• consider appropriate diagnoses based on information gathered,
• arrange for relevant further tests to clarify the diagnosis,
• provide appropriate management and include short and some long-term recommendations based on information gathered, and
• involve the patient in decision making.

Candidates are encouraged to undertake formative CBD. Candidates may initiate this with their supervisor or medical educator at any time. Formative CBD forms are available.

CBD delivery

The CBD assessment is conducted using an online assessment platform. It is the candidate’s responsibility to ensure that they can access a venue/space and the computer to be used meet the requirements as detailed in the IT Requirements Form. If using a private residence, the room must be free of interruptions. For the CBD assessment an invigilator is not required.

A phone is also required should either the candidate and/or Assessor lose internet connectivity. In this case, candidates may use a personal laptop or computer at their chosen venue.

IT Requirements

For information relevant to the IT requirements refer to the Assessment IT Requirements Form.

Cancellation

Candidates and Assessors are to advise the College as soon as possible if they are unable to attend a scheduled session. Failure to attend a session without provision of a minimum of one week’s notice will result in an additional administration charge of $395.

Where a candidate or Assessor is more than 10 minutes late, the session will be cancelled and rescheduled at no cost to the candidate.

Prior to and during the assessment

A candidate is required to:

• Be present in the room at least five (5) minutes prior to scheduled session start time.
• Prior to the session, check the room that will be used has access to a computer and camera, and ensure that they are in working order including able to access internet and online assessment platform.
• Have access to a phone, in the event of internet connection loss. The session can be continued by using hotspot on your phone to join the Zoom meeting with details provided.
• Login into the virtual room at least fie (5) minutes to the start of the session.
• Be alone in the room with access to the submitted printed clinical case notes only and no other clinical material, either printed or electronic.
• Provide proof of identification for each session.
• Listen to the pre-assessment briefing and respond to assessor questions.
• Advise the Assessor if unable to hear or understand the questions.
• Advise the College of any incident by submitting an incident report to assessment@acrrm.org.au within 48 hours of the session affected.

Quality assurance

CBD is a complex assessment requiring significant behind the scenes quality assurance work prior to scheduling of the assessment sessions. This quality assurance process ensures that candidates have the highest chance of success in this assessment.

The following quality assurance processes occur:

• administrative review of all case notes and attachments for compliance and privacy,
• clinical review of case notes to ensure complexity and domain coverage,
• Assessors are experience Fellows and trained CBD Assessors,
• all assessment sessions recorded and randomly moderated/reviewed,
• review of recording where a candidate fails two (2) cases whether by the same assessor or not,
• review and endorsement of all session outcomes by the Lead Assessor prior to presentation to BoE, and
• ratification of results by Board of Examiners.

Assessors

The primary responsibility of the Assessor is to ensure that the candidate is provided with the opportunity to demonstrate their clinical abilities in a fair and uniform testing conditions, and to ensure the integrity, consistency and fairness of the assessment process.

An Assessors’ responsibilities include:

• Advise the College of a conflict of interest with the candidate to be assessed.

• Advise as early as possible if the session needs to be rescheduled.

• Login into the virtual assessment room at least five minutes to the start of the session.

• Conduct a pre-assessment briefing session (maximum five minutes) before the first case is conducted. This briefing is to ensure that the:

  • candidate is informed of the assessment process,
  • mandatory requirements and the standard required for a positive outcome are explained and understood,
  • candidate is aware of the criteria they will be assessed against, and
  • candidate and assessor’s perceptions of the assessment match.

• Advise the candidate which two cases have been selected for discussion.

• Allow up to 30 minutes to discuss each case.

• Ask questions relating to the cases selected for discussion.

• Score the candidate according to marking criteria and return the completed forms to the College within two working days of the session.

• Inform the College of any incidents relating to the assessment as soon as possible and submit an incident form within two working days.

• Relevant policies and procedures for the conduct of the CBD are adhered to and that the integrity of the assessment session is not compromised.

The assessor must not be on call during the assessment and will not interrupt the assessment process by responding to any electronic communication device.

Preparation tools

An Introduction to CBD Assessment online course aims to support candidates in preparing for their CBD assessment.

Summary of CBD process

Structured Assessment using Multiple Patient Scenarios (StAMPS)

Introduction

The Structured Assessment using Multiple Patient Scenarios (StAMPS) is a blend of the Objective Structured Clinical Assessment (OSCE) and the traditional viva vocé (Viva) assessment.

StAMPS aims to assess higher order functions in a highly contextualised framework, where candidates have opportunity to explain what they do and demonstrate their clinical reasoning, rather than of simply providing evidence of knowledge, listing facts or recalling protocols. The assessors also ask the candidates how they would deal with system or patient factors that prevent the ‘standardised' approach being applied.

StAMPS is used as an assessment modality for Core Generalist Training and several Advanced Specialised Training (AST) programs.

Requirements

Candidates are required to achieve a pass grade in StAMPS.

Candidates are rated across 8 scenarios which are comprised of 3 questions in each scenario.

Grading is based on a numerical score which is the sum of 48 individual scores within the following domains:

• Management in the Rural-Remote context (applied to each of the three questions).
• Problem Definition and Structured Approach (applied to each scenario).
• Communication and Professionalism (applied to each scenario).
• Flexibility to changing context (applied to each scenario).

StAMPS format

The StAMPS scenarios are designed to measure a Registrar’s understanding of core and general principles, rather than only applying them to the specific nominated patient. The scenarios reflect real life where often clinical management is required prior to a definitive diagnosis being known. The scenarios and questions are sometimes unfolding in nature, with information being progressively revealed.

Candidates are provided with an opportunity to explain the rationale behind their thinking, as well as an opportunity for the assessor to explore issues in greater depth than is possible in a written paper.

The scenarios are in the viva vocé format where the candidate discusses the scenario directly with the Examiner. The candidate may be asked to clarify their answers when these are unclear and to expand on answers if there is insufficient detail.

Each scenario takes the form of introductory case information and then three questions relating to that case, sometimes with the inclusion of additional information provided.

StAMPS scenarios are written and researched using current Australian references. Care is taken to ensure that the scenarios reflect realistic patient presentations or issues that a FACRRM might reasonably encounter. The StAMPS exam is conducted online over three (3) hours and delivered across a series of rotations over one (1) or two (2) days, dependent upon the number of candidates enrolled. Candidates have a designated virtual exam (Zoom) room and Examiners rotate between candidates. Candidates will typically start and finish at the same time which is in Australian Eastern Standard Time. All candidates and invigilators are advised to check their local time zone and adjust the start time to their local time, if required.

The StAMPS consists of eight (8) scenarios, each of ten minutes duration. Candidates have 10 minutes at the commencement of the exam to read and familiarise themselves with all eight (8) scenarios during the pre-exam reading time. Candidates are expected to have read and be prepared for their first scenario by the start of the commencement of the first rotation.

Scenarios are shown to the candidates using the “share screen” function. A copy of the assessment material is also sent to the invigilator which is used only in the event of a technical issue on the exam day.

An interval of 10 minutes is placed between scenarios consisting of 5 minutes for candidates to read the exam material for the following scenario and 5 minutes to allow for any technical issues that may arise.

Candidates must not discuss any part of the assessment with other candidates undertaking the StAMPS assessment until all scheduled rotations have concluded (this could be over a two-day period). Any such conduct will be considered a breach of the Academic Code of Conduct and be dealt with accordingly.

Venue and Invigilators

For information relevant to venue and invigilator requirements refer to Venues and Invigilators.

It is the candidate’s responsibility to ensure that they can access a College approved venue and computer that meets the IT minimum requirements (refer to page 3 of the Assessment Venue Requirements) on the day of the assessment.

Candidates may take use their own laptop or computer at the venue. The College will not be liable for any technical issues / difficulties caused by using alternative configurations.

Content

The ‘Community Profile’ details key logistical issues about the location where the assessment is set and provides information regarding other relevant community factors.

The Community Profile and the content differs between CGT and various AST.

Core Generalist Training StAMPS

The context of the assessment is described in the Core Generalist Training StAMPS Community Profile. The setting is a rural town in Australia. The candidate is the most senior doctor in the town and works across the general practice and local hospital. The candidate is on a one in four on call roster and does outreach clinics in Aboriginal communities. Telephone specialist back up is always available including a video telehealth facility at the hospital.

The profiles are published on the College website. Candidates are permitted to retain the ‘Community Profile’ for reference during the reading time and throughout the assessment.

The content is mapped to the Core Generalist component of the Rural Generalist Curriculum 5th edition, 2022. The approximate percentage of cover for each domain is outlined below.

The total of this column is greater than 100% due to multiple domains being assessed multiple times within one assessment.

The assessment aims to sample across the Core Generalist content of the Rural Generalist Curriculum Learning Areas. The likelihood of a curriculum topic appearing in an assessment is outlined below.

Key: always covered ***** / occasionally covered *

Advanced Specialised Training

Emergency Medicine, Paediatrics, Mental Health, Adult Internal Medicine and Surgery are assessed by StAMPS using content mapped to the specific component of the curriculum.

Paediatrics, Mental Health, Adult Internal Medicine and Surgery share a common Community Profile, the setting is a rural hospital that services a vast district.

Emergency Medicine has a specific StAMPS Community Profile, the setting is a regional hospital that services a vast district.

Scoring Explanation

Following on from its successful application to the Core Generalist Training (CGT) StAMPS exam in 2021, Emergency Medicine and Adult Internal Medicine in 2022, the Behavioural Anchored Rubric System (BARS) will be fully implemented for all StAMPS modalities in 2023.

Each scenario is scored in six (6) areas (three Management scores [one for each question and one score for each of the remaining three domains). Each question/domain is scored independently on a linear scale where 0 is the lowest possible score and 7 is the highest possible score. The scores are summed across all eight scenarios and the total is used to determine the overall Pass or Fail outcome. A maximum score of 336 is possible.

StAMPS standard setting is achieved using a Delphi panel during scenario writing, scenario road-testing, moderation/calibration meetings, assessor training, scenario-specific scoring rubrics and post-exam statistical analysis.. The overall pass mark in any given exam sitting may vary due to relative difficulty of scenarios and characteristics of the candidates and this is statistically verified during the post assessment analysis.

The highest scoring candidates who do not meet the pass mark are automatically reviewed prior to the release of results. As a result of this review process, a candidate may be elevated to a “Pass”.

Preparation Tools

An Introduction to StAMPS assessment online course is available to enhance understanding of the context and structure of the StAMPS assessment. It includes information regarding what the StAMPS assessment is, how the scenarios are developed and provides the opportunity to practice the process.

Formal StAMPS preparation activity

Candidates must provide evidence of completion of at least one formal StAMPS preparation activity with the College, to be eligible to enrol in Core Generalist StAMPS.

A formal StAMPS preparation activity must be conducted by a medical educator or supervisor who has knowledge of the StAMPS format. The activity must Include information on the StAMPS format and StAMPS scenario practice and feedback.

The following activities meet these requirements:

• A StAMPS Mock Exam is offered by the College, dates and enrolments are on the website. The content is based on the Core Generalist component of the Curriculum, but the process is relevant to both Core Generalist and Advanced Specialised StAMPS.
• The StAMPS Study Groups are offered by the College leading up to the assessment. Study groups are routinely held each semester for Core Generalist and AST EM StAMPS. Study groups may also be held for other AST StAMPS. Dates and enrolments are on the website.

Other activities will be considered on a case-by-case basis against defined criteria.

Summary of the StAMPS process

Logbooks

Introduction

Logbooks are used as an assessment modality for Core Generalist Training, and for Advanced Specialised Training in Emergency Medicine.

Logbooks are also used during training to demonstrate experience; in this situation the logbook provides a record of procedures or cases. A Case Log proforma is available.

Requirements

Procedures in the Core Generalist Training and AST EM logbooks require certification. The ‘certifier’ refers to the person immediately responsible for the actions of the candidate to ensure patient safety.

The minimum qualification for performing the role of a certifier in the logbook is a registered medical practitioner at the level of senior registrar or equivalent. Where possible, the certifier should hold a Fellowship or other appropriate postgraduate qualification in the relevant discipline. The certifier of a procedure is not necessarily the candidate’s day to day supervisor or principal supervisor.

The certifier must have personally observed the candidate perform the procedure or personally observed the outcome of the procedure performed. An example of the latter would include the receiving Emergency Department consultant examining a patient who has undergone an emergency retrieval and who has had a chest tube inserted by the candidate at another location. Even though the consultant was not present when the tube was inserted, he/she would be able to ascertain whether the procedure had been correctly performed.

A procedure will be accepted as certified if either:

• enough information is recorded about the location and the certifier to allow the College to verify that the procedure was certified; or
• the procedure is signed off by a certifier

Procedural Skills logbooks submitted for completion of training will be audited if there are concerns about the accuracy of the logbook.

Core Generalist Training

The Core Generalist Training (CGT) Procedural Skills Logbook (Logbook) contains those procedural items from the Core Generalist component of the Rural Generalist Curriculum. Procedures are classified as essential or important.

Satisfactory completion of the CGT logbook is a mandatory requirement for award of FACRRM for all candidates, unless exempted through Recognition of Prior Learning.

Logbook entries may begin at any point in the candidate’s training cycle or during the two years prior to commencing training.

Medical students in their final two years can commence having basic procedures in the CGT logbook certified during a rural clinical school placement, for example :

• oropharyngeal airway
• intravenous access
• spirometry and peak flow measurement
• nasogastric tube insertion
• perform Glasgow coma scale
• local anaesthesia
• application cast/back slab
• use ophthalmoscope
• urethral catheterisation
• foetal heart sound detection.

The CGT Procedural Skills logbook is available through the “My Training Portfolio” section on a candidate’s “My College” dashboard, accessible from the College website. The logbook must be submitted online.

Candidates are required to present their logbook to either their principal supervisor or medical educator for inspection and discussion at least every six months. The candidate is wholly responsible for maintaining their logbook including ensuring each entry is accurate.

If procedures cannot be certified at the specified level but can be certified at a lower level, this should still be recorded. If all logbook requirements cannot be met, a letter of explanation must be emailed to training@acrrm.org.au following submission of the logbook.

There are two versions of the logbook:

• 2019 Logbook applies to registrars who commenced training in 2021 or earlier
• 2022 Logbook applies to registrars commencing training in 2022 or later.

The requirements for each version of the logbook is described below.

2019 Logbook

There are four levels of minimum competency, candidate performed the procedure to the standard of a:

A. Practitioner operating independently – demonstrated on a real patient\ B. Pass in an accredited course or certified satisfactory by a supervisor – demonstrated on a simulated patient\ C. Practitioner under supervision – demonstrated on a real patient\ D. Practitioner assisting an independent practitioner – demonstrated on a real patient

Each procedure has a defined minimum level of competency that must be met before the certifier can assign competency. A higher level of competency is also acceptable e.g. a candidate appropriately performs a specified task to the standard of an independent practitioner on a real patient when only simulation is required, is eligible for the certifier to sign that competency has been achieved.

Candidates are required to achieve certification at specified competency levels or higher for:

• 100% of the essential procedures and
• 75% of the important procedures.

2022 Logbook

There are four logbook levels:

A. Performed on a patient under observation.

B. Demonstrated in a simulated setting under observation.

C. Performed on a patient without observation.

D. Observed or assisted someone else doing the procedure on a patient or observed the procedure on video (e.g. YouTube) link to video required.

Candidates are required to complete:

• 100% of the essential procedures must be completed at level A or B and
• 100% of the important procedures may be completed at any level A, B, C or D.

Advanced Specialised Training

Emergency Medicine

Completion of the AST EM procedural skills logbook is required for the candidate to pass their AST in Emergency Medicine.

The AST EM procedural logbook requires certification of the candidate reaching the required competency level (A-D) and performing the procedure a specified number of times.

Competency levels:

A. Performed to the standard of an independent practitioner on a real patient and not just a simulated environment

B. Performed to a pass standard in a certified course in a simulated environment

C. Performed under supervision to the standard of a practitioner working under supervision

D. Assisted with the supervisor performing the task

The AST Emergency Medicine Logbook may be downloaded or a hardcopy obtained by emailing the training@acrrm.org.au.

Surgery

Candidates undertaking AST in Surgery are required to maintain a log of all surgical procedures undertaken during training.

This is a practice that needs to be continued throughout a surgical career.

The candidate may use any appropriate surgical logbook, for example the RACS Morbidity Audit and Logbook.

An appropriate logbook would:

• use standardised terminology, for example SNOMED clinical descriptors

• be easily sorted by procedure, to enable a supervisor to see how often a procedure has been performed

• be able to be shared electronically and in printed form

• contain the following data set for each entry:

  • date of procedure
  • name of hospital where procedure performed
  • patient name, age, gender, and hospital ID
  • name of primary surgeon
  • level procedure performed: 1st assistant, 2nd assistant, observed
  • level of supervision: independent, supervised
  • complications

Projects

Introduction

Projects are a substantial piece of original work done by a Registrar. Options for projects depend on the discipline but may include:

• research and development of a practical resource
• research and development of a local disease prevention or health promotion project
• a research project that contributes to current knowledge in a particular discipline and relating to key learning objectives in the specific curriculum.

Projects are the main summative assessment for the following ASTs:

• Academic Practice
• Aboriginal and Torres Strait Islander Health
• Population Health
• Remote Medicine

Registrars are strongly encouraged to share their project through:

• publication in a peer-reviewed journal
• presentation in the workplace or training organisation as appropriate or
• oral presentation or poster at a conference

Requirements

Registrars should aim to complete the project during the 12 months AST training time.

Registrars are required to achieve a pass grade in their project.

The project is expected to be presented at a level comparable to a project completed for Master’s Degree.

The project is expected to be undertaken independently by the candidate. Candidates seeking to submit a group project for their AST must have this approved prospectively. The candidate will need to demonstrate a lead role in the project and detail their responsibilities in the project proposal.

All projects will require ethics approval. A candidate’s project proposal, including details of ethics approval must be submitted to the College for review and approval by the Censor in Chief before commencement of the actual project.

Completed projects must include submission of a piece of assessable written work of approximately 4000–5000 words in length.

Content

The project must relate to the knowledge, skills and attributes in the relevant Learning Area of the Rural Generalist Curriculum.

The completed project proposal must include:

• Academic supervisor name and qualifications
• Type of project
• Background to the project including how the project will benefit the community
• Project aims and benefits
• Project value or importance to rural medicine
• Relevance to the chosen AST curriculum
• Methodology that will be used to collect and evaluate information/ data
• Ethical considerations and progress with the ethics approval process
• Engagement with the community especially if the project relates to Aboriginal and Torres Strait Islander health
• Project timeline

The completed written submission must include:

• projects’ aim/question
• projects’ value or importance to rural medicine
• that appropriate permissions were gained including ethics approval
• a critique of the relevant literature (literature review)
• methodology used to collect and evaluate information/ data in the project
• interpretation of results
• discussion of major findings
• evaluation of success
• a sound conclusion and
• recommendations for further work
• an Academic supervisor report

College support

The College offers the following support for Registrars undertaking projects:

• Providing feedback on the project proposal.
• Assistance with finding an Academic Supervisor if the Registrar has been unable to find a suitable supervisor.
• Facilitating peer support through linking Registrars completing a project.

Academic supervisor Registrars are required to have an Academic Supervisor to provide support and guidance in completing the project. Local rural clinical school or training organisations may be able to assist in identifying a suitable supervisor.

The Academic Supervisor must be nominated and provide input into the project proposal. An Academic Supervisor report is required to be submitted mid-project and with the final project. The supervisor report is initiated by the Registrar. The Registrar completes their section first and then the supervisor.

Process

Enrolment

1. Registrars must enrol in the project at the beginning of their AST. Enrolment is completed online.
2. A project proposal must be uploaded as part of enrolment process. Registrars must document a project timeline demonstrating completion of the project within 12 months.

Project proposal approval

The College will review the proposal against the criteria set for the project. The proposal will either be given approval or feedback will be provided on what is required to gain approval.

Project submission

A written report on the project must be submitted within 12 months. The final report must be accompanied by an Academic Supervisor Report.

Project grading

1. The AST Project is assessed against the project making criteria.
2. Projects that meet the standard are awarded a pass grade.
3. Projects that do not meet the standard are awarded a fail grade and information provided on what aspects require improvement to meet the standard. A second attempt is provided for the Registrar to revise the project and resubmit for grading. A re-grading fee applies, and this is considered a second attempt.
4. Project grading is ratified by the Board of Examiners and Registrars are provided with an outcome soon after.