Judge Foundry tests have been live for a couple weeks now, and as I’ve seen some results come in and received some feedback, I thought it might be helpful to share some tips, both for test-takers and test-givers.
I’m managing exams for Judge Foundry, which means I’ve touched every question currently in the pool (though I didn’t write most of them). As part of that, I have access to general stats on exams – while I can’t see how any individual performed on an exam, I can see how each question and each exam are performing overall. As a bit of background, in addition to managing testing for Judge Foundry, I’m also on the exams team for the Flesh and Blood Judge Program, and I previously managed the Level Three Exam under the Magic Judge Program.
Exams Available
As background, the exams that Judge Foundry currently offers are:
Level One Rules and Policy – Basic rules and Regular REL policy. To study for this test, you should understand evergreen keywords, the steps to casting a spell, layer seven, the parts of the turn, and other basic Magic building blocks that you’d need to teach a new player the game or to answer questions at FNM. You’ll want to review the JAR, and learn your Two-Headed Giant, since we expect Level One Judges to encounter 2HG at prereleases. You’ll also want to understand the basics of Commander – not the weird rules that you only see there, but things like commander tax and color identity.
Level Two Rules – The whole CR is fair game for this one (with the exception of less common multiplayer formats like Planechase, Archenemy, and the other stuff found in CR 801-809). You’ll want to learn your layers, get more in depth with how spells and abilities work, understand copy and replacement effects, and know how double-faced cards work (both modal and transforming).
Level Two Policy – This is mostly about the IPG and the MTR (though the JAR is still fair game). You’ll want to know both documents pretty well – while you won’t be tested on trivia like “How many seconds do players get for the seventh card of a pack in a called draft?”, you are expected to be able to resolve most common calls at an RCQ. That also means learning the tournament shortcuts and other parts of the communication policy. You can take this test at the same time as you take your L2 Rules exam, or on a different day, but you’ll need to pass both within 60 days to advance to Level Two.
Level Three Policy – L3s are expected to be able to handle anything a large competitive event throws at them, so you’ll want to be more fluent in the IPG and MTR for this one (and, again, know your JAR). To pass this test, you’ll need to know the penalties and remedies for every infraction, including various upgrades, downgrades and partial fixes. This test is pretty difficult, because we want Level Three Judges to be able to handle most common policy problems quickly and to have other judges consult with them – if you don’t know your IPG, you probably won’t pass.
The certification exams named above are all closed-book and untimed. They can be proctored in person by any judge who can certify judges of the exam’s level (for example, any Level Two Judge can administer an L1 exam). Most often, that will be the judge who conducts the advancement interview, but it doesn’t have to be.
If you need remote certification (for example, because you live more than two hours from a judge who can test you and won’t be at a large event for a while) contact joe.klopchic@judgefoundry.org.
If you need alternatives to these testing processes, we’d like to accommodate you. You can work with your mentor (or exam proctor) to devise reasonable accommodations that allow you demonstrate your knowledge and comprehension of the material. As one example, you and your proctor could decide to have the proctor read the exam questions to you. In the future, we hope to provide more guidance about accommodations for both candidates and proctors, in consultation with the community. Please feel free to reach out to us if you’d like to share your thoughts directly!
Update Quizzes and Advanced Update Quizzes – These quizzes are designed to keep you up to date on your rules and policy. Each will ask you five questions (four rules, one policy) and you can take each quiz twice. Level Two and Three Judges take the regular update, Level Four and Five Judges take the advanced update. These quizzes are open book – your best source for information will be the release notes, update bulletin and Toby’s blog post, but don’t neglect the primary documents themselves. While they’re timed (at two hours), that’s mostly just to keep you from starting one and forgetting about it for days.
Soon, we’ll be rolling out the Advanced Rules and Policy Exam for promotion to Level Four and Level Five and the exam for Event Admin Certification, along with the Murders at Karlov Manor Update Quiz.
The Philosophy of Judge Foundry Exams
I’ve received feedback from a number of judges that these exams are more difficult than the Judge Academy exams, or at least more difficult than they were expecting. This was useful feedback, because it motivated me to share a few tips on how to study for these exams and to talk about our philosophy in writing them – the article you’re reading now!
First, we expect judges to know things. This might not sound radical – if you asked a player or TO if they expected judges to know rules and policy, of course they would say “Yes!” And, indeed, everyone who sits for an exam does know a great many things. Unfortunately, there’s a big difference between knowing your rules and policy, and knowing how to look up answers about rules and policy. At Judge Foundry, we’re testing the former.
There are a few reasons for this. First and foremost, we think it creates better judges. Players will feel better playing in an event adjudicated by a Judge Foundry judge who can give answers to level-appropriate questions without having to look every answer up. We expect that you’ll have to look up some rules and policy during an event – even the best judges do, and looking things up when you’re unsure is a great way to make sure you’re answering a question correctly. But a Level One Judge should be able to explain how trample works without looking it up, and the same is true of a Level Two Judge and cascade, or a Level Three Judge and the fix for Looking at Extra Cards.
Second, we don’t want to test trivia. Again, this isn’t a radical idea! The goal of the exam is to test stuff that we expect judges to know, and to let them look up the obscure stuff. But an open-book test requires us to change the questions to more more trivia, because it’s pretty easy to just look up the answer to straightforward, common questions. And at that point, we’re not actually distinguishing what judges actually know, or how they’ll behave on the floor when they might assume they know the answers and don’t look them up.
It’s our goal to test the core things we expect judges to encounter at events. We don’t need you to know the rules for Grand Melee off the top of your head, at any level. We don’t need Level One Judges to know the IPG, because they’re not expected to work Competitive REL events. We don’t need Level Two Judges to answer questions about stickers, because sticker decks are pretty rare (unless you’re working an Unfinity Limited event, in which case we expect you’ll probably read the release notes ahead of time). And we don’t need anyone to know how Sylvan Library works (as Toby notes, it isn’t a real card).
Closed-book tests let us test what we think are reasonable things to know, without getting into an arms race with candidates based on their ability to quickly access information. We think that creates better results overall.
Third, update quizzes are about learning, not about testing. Update quizzes are not designed to be difficult. They’re designed to expose every L2+ to the questions they might encounter when judging a Limited event of the new set, and to make sure they’re aware of broader rules and policy changes that accompany a set release.
If you see a question on an update quiz and aren’t sure of the answer, so you go check the release notes, then answer it correctly… the update quiz has done its job of helping you to do better at your next event. While our certification exams are designed to evaluate judges for their fitness for promotion, update quizzes are designed to help judges maintain their current knowledge level.
Lastly, we continually refresh test content. We want tests to remain relevant for judges. For example, as keywords move in and out of evergreen or deciduous status, we’ll move questions in and out of the pools so they stay relevant. I’m also pretty aggressively gardening the exams right now, as our results roll in, to remove or change questions that are too easy or too hard. This is in response to both exam stats, and to user feedback – members of the testing team and I review every comment and while we might not always act on them right away, we do consider them when evaluating whether a question should stay in the pool or get some changes.
After you complete an exam, please do go in to review it and provide feedback to us about your experiences so we can improve questions for others – that’s especially the case for update quizzes where some of the questions will make their way onto the certification exams.
Tips for Taking Tests
First, take your time. These exams are untimed and we mean that. You can go through the whole thing a couple times – it may be that a later question jogs your memory on a question you already answered. You can write as much as you want on the test itself – your proctor will only grade the answers on the answer sheet, so feel free to make notes for yourself, work out math, and whatever else you need to do.
Second, try to think of the answer to the question before you read the multiple choice responses. If the answer you came up with is on the list… great! If not, you’re probably missing something. Thinking of the answer first means you’re not going shopping for a response, you’re applying your knowledge, then picking out the response that matches it.
Third, understand the answer splits to eliminate incorrect answers. Consider this question:
At an FNM, a player accidentally draws an extra card. How do you fix this?
A. Issue a formal Warning, then put a random card back on top of the library. Record the penalty with the scorekeeper.
B. Issue a formal Warning, then reveal the player’s hand and let the opponent choose a card to shuffle into the library. Record the penalty with the scorekeeper.
C. Put a random card back on top of the library, then ask the player to be more careful.
D. Reveal the player’s hand and let the opponent choose a card to shuffle into the library, then ask the player to be more careful.
There are four responses, and it’s testing two things: what’s the penalty and what’s the remedy? Once you remember that we don’t generally issue formal penalties at Regular REL events, you can eliminate half the answers! You could also eliminate half the answers by remembering that the remedy prescribed by the JAR is to put a random card back on top. If you know both these things, you can confidently answer C, but if you only know one, you can at least narrow your choices down by 50%, then try to deduce the answer to the half of the question.
Fourth, understand exam conventions. If there are players named in a question, names starting with A always signify the active player, and names starting with N always signify the inactive player. This might not matter for every question, but if it does matter, that’s always how it’ll work (except in rare questions where a turn is passed and the new turn starts – players aren’t changing names as part of cleanup!).
Policy questions occasionally bring in judges whose names start with J and spectators whose names start with S. You’ll also occasionally see a second AP and second NAP in team or 2HG games.
Beyond this, you can assume everything that the question tells you is absolutely true. Because you can’t investigate a test, no question will assume you need to test the veracity of a statement.
Fifth, have a plan for multi-answer questions. There are only a few questions that ask users to “select all that apply” on the L1 exam, but these questions get more common as you level up. These questions are tougher in a few ways – you can’t use answers you know to be incorrect to narrow down the correct one, and you need to evaluate every answer against the question.
For getting this far into the article, I’ll share with you three secrets about the testing rules to multi-answer questions:
- There’s always at least one correct answer
- All answers might be correct
- Only one answer might be correct
When you hit one of these questions, circle or highlight that it’s a multi-answer question – you don’t want to get it wrong because you just answered A after seeing that it was correct and then don’t bother reading B. Then consider each answer against the question, without trying to game out the question – now that you know that the only rule is that at least one answer is correct, you shouldn’t get alarmed if you believe that the most correct choice is all of them.
Finally, if it’s unclear, ask. Our volunteers did an outstanding job of writing our tests, and I think I did an okay job editing them. But we miss things – occasionally a card tag is bad, or we mixed up AP and NAP, or there was a rules change that should have been incorporated, but we missed it. While proctors can’t provide more information than what the test provides or let you look things up, you can go to them if you feel that there’s a problem with the test itself. When in doubt, select the most correct answer, make a note that you feel a question was unclear, and after you get your test graded, see if the explanation clears it up.
Pointers for Proctors
Just as many judges aren’t used to taking closed-book judge tests, you might not be used to administering them. Here are few hints to help you help your candidates:
First, it’s okay for candidates to fail. If the test was perfunctory, we wouldn’t bother with it. Some people need to study more before they’re ready to certify or level up. Be sure to support your candidates as they work toward their next attempt. Cooldowns are listed on the exam page – we want judges to take the time to study and fill in the gaps rather than trying to jam attempts until they crash across the finish line, but we also don’t want to have an oppressively long wait between attempts.
Second, give the candidate some space, but keep an eye on them. I tested for Level One in my kitchen while my proctor read a book. I tested for Level Two at an SCG IQ while my proctor was running the Top 8. I tested for Level Three at Ryan Stapleton’s house while he made guacamole (it was good). I’ve administered a bunch of tests, and the most common case is that I’ll have someone come to the LGS where I’m playing FNM and set up in a corner where I can be available but also give them all the time they need.
Keeping your candidate where you can check on them or see them helps you know if they’re done or if they need help or if people keep coming over to bother them. It also means you can vouch for the candidate later if someone questions whether they really took the test without looking things up.
Third, consider taking the exam yourself while waiting. This isn’t required (and you may see questions that you encountered when you were leveling up, if you’re reading this article in a few months), but it’s a good way to pass the time and review your own knowledge, and to become familiar with the questions so you can go over them with the candidate after they finish. It also gives you a chance to jot down feedback to submit along with the answers.
If you encounter a question where you think the answer is wrong, make a note of it and email me at johnbrian@judgefoundry.org. Please don’t change the result for the candidate on the fly – if the answer is wrong, we’ll adjust the candidate’s score later along with fixing the question.
Lastly, submit the candidate’s answers right away. Tests aren’t official until they’re submitted on JudgeApps. You really, really don’t want to lose the candidate’s answer sheet before you submit the results (if this happens, email TxJoe at joe.klopchic@judgefoundry.org). And entering in the results immediately means the candidate will be able to view their exam later to read the explanations and to provide feedback.
Pencils down
I hope that this article helped explain better our exam philosophy and provided you with some useful advice, whether you’re taking or giving exams. As always, I’m interested in your feedback – you can email me at johnbrian@judgefoundry.org or the whole board at board@judgefoundry.org, or comment on JudgeApps ou Reddit. You can also follow us on Facebook and Twitter.
