Last month, Dan Milavitz, the project lead for Exam Study Guides, emailed me with an idea. He suggested that we should try to work toward rebuilding the level-based practice tests that existed in the Judge Program era.
What really caught my eye from Dan’s email was this:
“I had an L2 candidate passed off to me a couple weeks ago who got a [non-passing]% on his rules exam and that would likely have been avoided had his expectations been calibrated by a practice exam.”
This was a reasonable problem to consider – the exam team wants to set folks up to pass their exams, and that’s tough if they know the topics on which they might be tested but not the difficulty level of the questions they’ll be expected to answer. We don’t want to make candidates fail an exam to find out that they weren’t ready to take it, and needed to study more.
I thought about it for a week or so, and I ran some numbers. Doing full practice tests would be tough because practice tests would require an extensive pool of questions, and those questions would have to mirror the expected difficulty of the actual exam. But only a handful of questions make their way from the update quizzes to the exams each season – it’s much more likely for questions to make their way to the practice quizzes, to which every judge has access. And it’s not easy to create a new exam that mirrors the difficulty of another exam, especially because it’ll be taken under different circumstances, all with a smaller sample size than the Judge Program had, and entirely designed by volunteers.
But then I had an idea: we already know the difficulty of the questions in the pool, so why don’t we just pull a small slice from there?
What’s in a Preparation Test?
Preparation Tests (or “Prep Tests” when I don’t want spellcheck reminding me that there’s only one e in “preparation”) are, essentially, a sample of the exam a judge is about to take.
Each Prep Test has exactly five questions, and is associated with an advancement exam. All five of these questions were selected from the exam itself, and are about as difficult as the exam. Almost every question I selected has a success rate of the exam’s passing score, ±10%, which is the same success rate we target for each question on the exam. Almost all these questions have had at least fifteen candidates see them on an exam, so they’ve been refined and revised by feedback. In short, they’re five of the best questions on that exam.
Or, at least, they used to be. Questions moved to the Prep Test will no longer appear on their respective exams. That’s why each Prep Test is only five questions, and there’s only one attempt per person, for now.
Prep Tests are unscored – while you’ll learn what you got right and wrong, the “passing score” is 0%. That’s because Prep Tests are designed as a tool to help candidates evaluate how difficult the exam will be, not to act as a gate telling the proctor that they’re ready to test.
Using Preparation Tests
Your mentor who’s helping you prepare for the exam can generate you a Prep Test. Because it’s a self-diagnosis tool, the test is unproctored, and can be taken online. My recommendation is to treat the Prep Test just like you would the exam, though, and to take the test closed-book… all these questions come from our real exams, which means that if you’re having to look up the answers, you’re probably not ready for the exam.
After you take a Prep Test, you can go over the contents with your mentor, though, as with all our exams, please don’t share it online. I tried to include a variety of topics on each prep test, but five questions can’t cover everything an exam might throw at you, so check the Study Guide for your exam to see on what else you might be tested.
Prep tests have a different role than practice quizzes. Practice quizzes are a great way to quickly test your knowledge (or learn something new!). The pool of practice quizzes grows with every update quiz, and they often include set-specific mechanics. They’re also a great way for judges who miss an update quiz to learn something. Prep tests are a tool to evaluate whether you’re ready to take your next exam to level up. You can generate yourself a practice quiz right now, but you’ll need your mentor to generate a Prep Test for you.
Prep Tests aren’t currently required. I say “currently” because it’s probable that we’ll include these in our next levels tune-up, coming very soon. Joe and I have talked a bit about what we’ll require, and it will likely be a requirement to take a Prep Test, but not a requirement to pass one.
Go Forth and Test
Preparation Tests are available right now for the L1 Rules and Policy, L2 Rules, L2 Policy, and L3 Policy exams, so if you’re working with a candidate, you can generate a test right now. If you’re working with a mentor, you can ask them to generate a Prep Test for you when you and they agree you’re about ready for the exam.
And if you have suggestions about how to make our exams better, or any part of Judge Foundry better, say something, like Dan did! Email the appropriate project lead or email the board, or post to the Feedback Forum on JudgeApps, or just talk to us at events. And even if we can’t implement the specific solution you’re suggesting, it’s helpful to hear our members articulate a problem so we can look at what we can do about it.
