Jacqui Molkenthin, JEML Consulting

Across my work, with a wide range of EpAOs, I have seen a variety of approaches to the design of questions for knowledge tests. Knowledge tests requirements across the assessment plans vary, for example, they may be multiple choice test (MCQs), short answer, scenario based, and some contain a mixture. But there is one thing in common, the assessment plan does not write the questions or answers, that is the role of the EpAO. This means that EpAOs must design question and answer banks that are fit for purpose, which essentially means that they must be valid, reliable, comparable, manageable and minimise Bias (Ofqual Condition D1.2). In the Ofqual technical evaluation of end-point assessments in June 2020, they identified “more than one correct answer in multiple choice questions” in over a quarter of EPAs they reviewed (page 7). The report then went onto highlight effective practice (page 8).

The Institute for Apprenticeships and Technical Education (IfATE) provides guidance on the use of knowledge tests in end-point assessment, but does not publish guidance on writing knowledge test questions. Ofqual provide guidance within their criteria (Criteria D) detailing how they expect organisations to explain how they develop their multiple choice tests, but they do not provide guidance on how to write them.

Given how critical it is for EpAOs to make sure they get the question, and answer, design right, I thought it may help if I share a few hints and tips on writing multiple choice questions. Readers may find it helpful to read this article in conjunction with another article of mine about ensuring the correct level in assessment design. But before I get started, readers may want to get used to some terms that are often used for multiple choice question design, and are seen across internet searches and academic papers/reports. The multiple-choice question or scenario may be referred to as the “stem”, the correct answer may be referred to as the “key”, and the remaining incorrect answers may be referred to as the “distractors”.

As with all my articles, I am not a representative of Ofqual, IfATE or the ESFA; what I am sharing are hints and tips based on practical experience, which I hope are useful to EpAOs. Within this article, I have chosen to use the, well known, highway code to help provide examples so that readers can understand the principles I am trying to explain:

1. Make sure the question is clear and that all answers are plausible:

Do not use words that could be misleading and do not include unnecessary information. Make sure the apprentice can understand the question without having to read the answer options, and make sure that all answers are plausible but that only one is correct. For example:

QUESTION - When should a lorry driver stop at the stop line at traffic lights?

ANSWER – When the traffic lights are: a) red; b) amber; c) red and amber; d) green

This question is misleading as 3 of the answers could be correct: a) Red: Stop on red; b) Red and amber: Stop and wait; c) Amber: Stop unless you have already crossed the line.

This question contains some unnecessary / irrelevant / distracting information by adding in the word lorry driver. It would perhaps have been better to say ‘when should a driver’.

This question has also automatically enabled the apprentice to rule out option d, an obvious incorrect answer, meaning that they only have to choose from 3 possible answers.

Perhaps a more suitable question would be “at traffic lights, what does it mean when the red and amber light is showing?”.

2. Avoid open or multiple interpretation questions:

Avoid questions that lead the reader to think that the correct answer is based on their opinion rather than fact, for example, words such as could, should, might, or think (just like the example question in bullet point 1). Instead use words such as must, compulsory or required. Outside of MCQs, if you look at Ofqual conditions, you start to understand the importance of such wording. The video series “

” details when and how they use specific words, such as ‘must’, ‘where’ and ‘any’, and their importance, demonstrating just how critical the use of a words it to its meaning and interpretation.

3. Avoid Negatives:

Make sure the question enables the apprentice to demonstrate that they know the correct answer and have the required knowledge. To do this, you need to avoid negative questions, and avoid answer options such as ‘none of the above’. For example:

QUESTION - When do you not need to stop at the stop line at traffic lights?

ANSWER – When the traffic lights are: a) red; b) amber; c) red and amber; d) none of the above

Not only does this question have a high risk of being misread, it fails to enable to apprentice to demonstrate that they know the right answer. It also fails to take into account a whole range of other factors that could influence why, or when, they do or do not need to stop at traffic lights.

4. Avoid using pictures and charts where possible:

For example, if the traffic light question had the answers displayed as pictures of traffic lights, rather than text answers of red, amber, red and amber, green; this could have a negative impact on apprentices if, for example, they are colour blind, or of the assessment was paper based and the papers had been printed in black and white. I am not saying, do not use pictures and charts, but think carefully about their accessibility to all users and all types of assessment delivery options before choosing to use them.

5. Avoid lengthy questions, but be aware of the level:

Lengthy questions take up valuable test time, and can have an impact on equality. For example, a long question may penalise people with English as second language or those with reading difficulties. Longer questions may also cause apprentices to panic because of it taking up valuable test time to read. However, remember to take into account the level of the assessment, a level 2 multiple choice question, focussed around recall, will look different to a level 6 multiple choice question, focussed around critical thinking. It is less common to see multiple choice questions used at higher levels, but it is still worthwhile taking this into consideration.

6. Keep answer lengths similar:

One answer longer than the others tends to give away the answer, as longer answers tend to be the correct answer. You may not believe me on this, but I have found that when I have written questions and answers, I have unintentionally made the correct answer longer because I was making sure that the answer was correct beyond doubt (this tends to be more of a challenge at higher levels).

7. Use stand-alone questions:

Make sure the answer to a question can’t be found in a question elsewhere in the test, and make sure that an apprentice doesn’t have to get an answer correct in one question in order to be able to understand and answer another question. When writing questions, you are often writing them based on a theme, or a particular Knowledge, skill, or behaviour (KSB) of the standard, and when writing on this basis it is easy to unintentionally come up with questions that link together.

8. Use neutral language where possible:

Ensure questions and answers are free of gender, ethnic, age, political, cultural stereotyping or discrimination. For example: avoid things such as TV show names as they may not be recognised by some apprentices because of age, culture or personal preferences; and avoid gender stereotypes such as fireman, air stewardess, and barmaid, and replace with gender neutral language such as firefighter, flight attendant and bar tender. In fact, you may recall recent press coverage about the change of cricket terminology from ‘batsman’ to ‘batter’.

9. Avoid Homonyms:

Avoid using words that sound the same but are written differently and have different meaning and words that are written the same but have different meaning. For example: ‘pain’ and ‘pane’; ‘loan’ and ‘lone’; ‘break’ and ‘brake’; lead (dog lead, or metal, or to be in front); plane (woodworking tool or aeroplane). If they must be used, ensure that the meaning is clear to the apprentice.

10. Mapping:

Map each question to the KSBs of the standard, in accordance with the assessment plan. In other words, when writing a multiple-choice question, do not write questions that do not map to the KSBs, and do not write questions for KSBs that are not assessed via the multiple-choice knowledge test.

Jacqui Molkenthin, JEML Consulting

I hope my ideas and suggestions are helpful. You can find a whole range of other articles of mine supporting the EpAO sector on FE News and my LinkedIn updates.

You may also be interested in these articles:

Sponsored Video

Register, Login or Login with your Social Media account:


Advertisers

Upcoming FE Events

Advertiser Skyscrapers

Latest Education News

Further Education News

The FE News Channel gives you the latest education news and updates on emerging education strategies and the #FutureofEducation and the #FutureofWork.

Providing trustworthy and positive Further Education news and views since 2003, we are a digital news channel with a mixture of written word articles, podcasts and videos. Our specialisation is providing you with a mixture of the latest education news, our stance is always positive, sector building and sharing different perspectives and views from thought leaders, to provide you with a think tank of new ideas and solutions to bring the education sector together and come up with new innovative solutions and ideas.

FE News publish exclusive peer to peer thought leadership articles from our feature writers, as well as user generated content across our network of over 3000 Newsrooms, offering multiple sources of the latest education news across the Education and Employability sectors.

FE News also broadcast live events, podcasts with leading experts and thought leaders, webinars, video interviews and Further Education news bulletins so you receive the latest developments in Skills News and across the Apprenticeship, Further Education and Employability sectors.

Every week FE News has over 200 articles and new pieces of content per week. We are a news channel providing the latest Further Education News, giving insight from multiple sources on the latest education policy developments, latest strategies, through to our thought leaders who provide blue sky thinking strategy, best practice and innovation to help look into the future developments for education and the future of work.

In Jan 2021, FE News had over 173,000 unique visitors according to Google Analytics and over 200 new pieces of news content every week, from thought leadership articles, to the latest education news via written word, podcasts, video to press releases from across the sector, putting us in the top 2,000 websites in the UK.

We thought it would be helpful to explain how we tier our latest education news content and how you can get involved and understand how you can read the latest daily Further Education news and how we structure our FE Week of content:

Main Features

Our main features are exclusive and are thought leadership articles and blue sky thinking with experts writing peer to peer news articles about the future of education and the future of work. The focus is solution led thought leadership, sharing best practice, innovation and emerging strategy. These are often articles about the future of education and the future of work, they often then create future education news articles. We limit our main features to a maximum of 20 per week, as they are often about new concepts and new thought processes. Our main features are also exclusive articles responding to the latest education news, maybe an insight from an expert into a policy announcement or response to an education think tank report or a white paper.

FE Voices

FE Voices was originally set up as a section on FE News to give a voice back to the sector. As we now have over 3,000 newsrooms and contributors, FE Voices are usually thought leadership articles, they don’t necessarily have to be exclusive, but usually are, they are slightly shorter than Main Features. FE Voices can include more mixed media with the Further Education News articles, such as embedded podcasts and videos. Our sector response articles asking for different comments and opinions to education policy announcements or responding to a report of white paper are usually held in the FE Voices section. If we have a live podcast in an evening or a radio show such as SkillsWorldLive radio show, the next morning we place the FE podcast recording in the FE Voices section.

Sector News

In sector news we have a blend of content from Press Releases, education resources, reports, education research, white papers from a range of contributors. We have a lot of positive education news articles from colleges, awarding organisations and Apprenticeship Training Providers, press releases from DfE to Think Tanks giving the overview of a report, through to helpful resources to help you with delivering education strategies to your learners and students.

Podcasts

We have a range of education podcasts on FE News, from hour long full production FE podcasts such as SkillsWorldLive in conjunction with the Federation of Awarding Bodies, to weekly podcasts from experts and thought leaders, providing advice and guidance to leaders. FE News also record podcasts at conferences and events, giving you one on one podcasts with education and skills experts on the latest strategies and developments.

We have over 150 education podcasts on FE News, ranging from EdTech podcasts with experts discussing Education 4.0 and how technology is complimenting and transforming education, to podcasts with experts discussing education research, the future of work, how to develop skills systems for jobs of the future to interviews with the Apprenticeship and Skills Minister.

We record our own exclusive FE News podcasts, work in conjunction with sector partners such as FAB to create weekly podcasts and daily education podcasts, through to working with sector leaders creating exclusive education news podcasts.

Education Video Interviews

FE News have over 700 FE Video interviews and have been recording education video interviews with experts for over 12 years. These are usually vox pop video interviews with experts across education and work, discussing blue sky thinking ideas and views about the future of education and work.

Events

FE News has a free events calendar to check out the latest conferences, webinars and events to keep up to date with the latest education news and strategies.

FE Newsrooms

The FE Newsroom is home to your content if you are a FE News contributor. It also help the audience develop relationship with either you as an individual or your organisation as they can click through and ‘box set’ consume all of your previous thought leadership articles, latest education news press releases, videos and education podcasts.

Do you want to contribute, share your ideas or vision or share a press release?

If you want to write a thought leadership article, share your ideas and vision for the future of education or the future of work, write a press release sharing the latest education news or contribute to a podcast, first of all you need to set up a FE Newsroom login (which is free): once the team have approved your newsroom (all content, newsrooms are all approved by a member of the FE News team- no robots are used in this process!), you can then start adding content (again all articles, videos and podcasts are all approved by the FE News editorial team before they go live on FE News). As all newsrooms and content are approved by the FE News team, there will be a slight delay on the team being able to review and approve content.

 RSS IconRSS Feed Selection Page