Using the Rasch Model to Analyse Multiple Choice Questions

This is the final post in my week long series on assessment and reporting at Paradigm Trust schools.

Monday: Exploring Comparative Judgement Data With Charts

Tuesday: Communicating Assessment Scores with Parents and Carers

Wednesday: How Good Are Our Multiple Choice Questions?

Thursday: What Do We Do With the Multiple Choice Data?

Today’s post (Friday) is a jump. My colleagues at Paradigm Trust were already busy evaluating our assessments and we don’t really need a new statistical tool, but Rasch is too useful to ignore.

Back in November, a group of us went to ResearchEd Ipswich. Deep Ghataura gave a talk on using the Rasch model to measure question difficulty. It’s taken six months, but now we are using Rasch to help us evaluate our multiple choice questions.

Rasch allows us to:

  1. measure the performance of a learner (useful)
  2. measure the difficulty of a question (it allows us to predict the probability of a certain child answering a specific question correctly. We may find a use for that!)
  3. measure the infit (quirkiness) of a question (useful when we rewrite the questions)
  4. measure the infit of pupils’ performances in a test (do they have odd gaps?)
  5. measure the standard error of a pupil’s score (really useful for an honest understanding of assessment scores).

In other words, it’s almost miraculous. We’ve found these measures in our NoMoreMarking data (which also uses Rasch) so helpful, that we’ve developed a multiple choice analysis spreadsheet to analyse and present data in a comparable way.

 

Screenshot 2019-05-17 at 18.44.16
The vertical axis shows you how difficult the question is. The horizontal axis shows how ‘quirky’ the response was (i.e. did the high scoring learners mostly get it right, and the low scoring learners got it mostly wrong? Or was something odd going on?)

In the chart above, question 10 is challenging and not too ‘quirky’ – the high scoring learners mostly got it right, and the low scoring pupils mostly got it wrong. Question 8 got an unusual response. When we went back to the question, we found it was phrased badly. We have improved it for next time.

The assessment above was also unbalanced – we have several easy questions (6, 13, 12) and not enough challenging questions. We can address that next time.

Our Rasch spreadsheet (here) is easy to use, but probably less robust than commercial packages (here): you are welcome to have a go. I’ve locked all of the tabs to view only, except the raw data tab, where you should add your dichotomous MCQ data (1 for correct, 0 for incorrect).

Screenshot 2019-05-16 at 16.25.06
dichotomous data

The other tabs have the data listed above and some pretty data visualisation for the questions. You should be able to ‘look-under-the-bonnet’ to see the formulas if you like that sort of thing. I can’t recommend this Rasch textbook highly enough.

I’d recommend you make a copy for your own data. Please let me know if you have success (or spot errors) – it’s still early days with this for us.

Although Deep Singh Ghataura (@DSGhataura) ‘s ideas and influence runs through this post – the mistakes are all mine 🙂

Ben

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s