Murders at Karlov Manor Update Quiz Wrapup

The Murders at Karlov Manor update quizzes just closed, so just as I did with the The Lost Caverns of Ixalan quizzes, I’m going to take a moment to walk through some questions from the quizzes and talk about a few of the questions. And tomorrow, we’ll be opening the quizzes for Outlaws of Thunder Junction.

Before I jump in, I want to thank everyone who worked on this exam. The project was led by Steven Zwanger, with questions written and edited by Nathan Long, Tobias Vyseri, Travis Lauro and me. Paul Baranay and Dan Collins worked on the backend to make update quizzes happen and to keep exams running smoothly. Please join me in thanking these judges for their work.

With the completion of this exam, I believe we also have enough questions relegated from the update quizzes to launch the practice exam pools. These pools are currently very LCI/MKM focused, but will diversify over time as questions rotate out of the evaluation exams and as we have more quizzes. Look for the launch of practice exams later this month.

Investigating the scene

The quiz was five questions long (four rules, one policy), and judges needed a 60% or better score to pass. The advanced question pool had fourteen questions (enough for two attempts), while the standard pool had a whopping 21 questions (enough for four attempts)! I wouldn’t expect this many questions on every update quiz, but we’ll always try to ensure everyone can attempt the quiz twice.

The standard quiz was attempted 160 times by 137 judges. That’s 53% of eligible test-takers – if you’re a Level Two or Level Three Judge who wasn’t in the 53%, remember that update quizzes are the easy way to fulfill a maintenance requirement. Here’s a chart on how folks did:

As before, almost everyone passed on the first attempt. Of those who didn’t, most passed on the second, and a few took three or four attempts. 

The advanced quiz was taken 24 times by 19 judges, 68% of Judge Foundry’s L4s and L5s. However, note that there was an expedited promotion wave in there, and that some L4s and L5s took their standard quiz before being promoted, so this number is a little bit of an undercount. Here’s the chart for the advanced quiz:

Remember that, with the advanced quiz, we’re dealing with pretty small numbers. While it looks like there are big differences in the percentage of L5s who got every question right, versus missing one versus missing two… the actual difference is only one judge at each step.

Where were the bodies buried?

Part of the purpose of update quizzes is to waterfall questions down into Judge Foundry’s evaluation exams and practice pools. Most questions from our update quiz will find another home on one of our other tests, though a few will be discarded.

In general, when determining relegation, I’m looking at pass rates and suitability. We want our questions on our evaluation exams to have a pass rate of ±10% of the exam’s passing score. While the score a questions receives on a practice quiz isn’t a precise match to how it’ll perform on an evaluation exam (for a number of reasons, including that practice quizzes are open-book, judges tend to take their evaluation exams a lot more seriously, practice quizzes test things you may have just learned by listening to JudgeCast or reading the release notes, and other factors), it is a good indicator that a question might be a good fit.

For example, there was a question on the standard quiz with a 33% pass rate overall – that questions is going to be way too hard for the L2 exam. However, it had a 67% pass rate on just the set of L3s – if a question on the L3 exam had this pass rate, it would be a candidate for relegation to a practice quiz or maybe some tweaks based on feedback, because it’s a little too hard. But it’s close enough that we might try it on the L3 Exam for a bit to see how it performs with judges who are studying before taking it and to show where we need to do more policy education.

On this exam, here’s how relegation broke down:

Just… one more question

Judges enjoyed this section last time, so I’m once again going to pull out a small sample of questions with interesting results, to dig in deeper. None of these questions are being added to our evaluation exams, for reasons that will become apparent as I discuss them. I’m going to link these questions, so if you saw the question, you can see how you answered, but if you haven’t seen this question before, the link will probably not resolve.

Don’t try this at home

Our first question is Q2374, a policy question for the advanced quiz. This one had a 46% pass rate. Yikes!

During a game at a Pro Tour, Alfred plays Agency Coroner face-down, believing it to have the disguise ability. Nadia responds by casting No More Lies. When Alfred puts his card into exile, he notices that it doesn’t have disguise and calls a judge. The judge notes that Alfred has a Coveted Falcon and a Bubble Smuggler in hand. What best describes what infraction, if any, has occurred, and what penalty and remedy apply?

1. Alfred has committed Game Play Error–Hidden Card Error. The penalty is a Warning. The remedy is that Alfred reveals his hand, and Nadia selects either the Coveted Falcon or the Bubble Smuggler to swap with Agency Coroner.
2. Alfred has committed Game Play Error–Hidden Card Error. The penalty is a Game Loss. No further remedy applies.
3. Alfred has committed Game Play Error–Game Rule Violation. The penalty is a Game Loss. No further remedy applies.
4. Alfred has committed Game Play Error–Game Rule Violation. The penalty is a Warning The remedy is to back up the casting of No More Lies and back up the casting of Agency Coroner.
5. Alfred has committed Game Play Error–Hidden Card Error. The penalty is a Warning. The remedy is that Alfred must swap the Agency Coroner with either the Coveted Falcon or the Bubble Smuggler.

When you read this question, you likely got to “I know what policy says to do… but I really don’t like it!” And that’s fine – indeed, questions on the L3 Policy and Advanced Rules and Policy often test judges’ understanding of policy by positing cases where your natural instinct is to do one thing, but policy is to do something different, because we want to see if you know policy, not whether policy is well-crafted enough that you’ll do the correct thing accidentally. One judge told me that if this was a printed exam, he would have taken out a sharpie and added “F. Deviate.” However, another commenter nailed it on the head:

In a case like this, we don’t have anything truly crazy happening, but the results, where AP gets to choose which spell will be countered, is clearly not an ideal situation. Is it worth adding more sentences to the IPG to cover scenarios like this? Probably not right now, but if R&D starts printing face-down cards in every set, maybe it’s worth reconsidering.

As for this question, though, it’s either getting discarded or moved to Policy Practice. This pass rate is way too low for it to be considered for the L3 Policy or Advanced Rules and Policy exams. 

Suspension of the rules

Next, we’ve got Q2320, a rules question from the advanced quiz. 82% of respondents got this one right.

Ada has a Detritivore in exile with one time counter on it. At the beginning of her upkeep, what best describes what happens?

1. Ada removes the last time counter from Detrivore. Ada must put the trigger to cast Detrivore on the stack first, followed by the trigger to destroy a nonbasic land. She picks a target nonbasic land. When the trigger to cast Detritivore resolves, she must cast it.
2. Ada removes the last time counter from Detrivore. She puts the trigger to cast Detritivore and the trigger to destroy a nonbasic land on the stack in any order. She picks a target nonbasic land. When the trigger to cast Detritivore resolves, she may choose not to cast it. If she does, it remains exiled and is no longer suspended.
3. Ada removes the last time counter from Detrivore. She puts the trigger to cast Detritivore and the trigger to destroy a nonbasic land on the stack in any order. She picks a target nonbasic land. When the trigger to cast Detritivore resolves, she must cast it.
4. Ada removes the last time counter from Detrivore. Ada must put the trigger to cast Detrivore on the stack first, followed by the trigger to destroy a nonbasic land. She picks a target nonbasic land. When the trigger to cast Detritivore resolves, she may choose not to cast it. If she does, it remains exiled and is no longer suspended.

Update quizzes aren’t just about teaching judges the new rules, they’re also about saying “Hey, the old rules changed!” When Wizards makes a change to the rules, we want to make sure judges are aware of it, so both exams had a couple questions about suspend on them, even though suspend wasn’t in the set. 

Note that this question did undergo a change during the process after we received some feedback (from “loses suspend” to “is no longer suspended”) – the exams team reads all feedback submitted on questions, and we’ll regularly make changes if there’s something misleading or confusing about a question. In this case, we felt the change wasn’t enough to merit forking it into a new question ID, so the stats here reflect both before and after the wording change.

88% is in the range of target pass rates we’re looking for in a question, so this might normally be a candidate for the L2 Rules or Advanced Rules and Policy exam, but this question is bound for the Hard Rules Practice Exam instead. The change to suspend is very technical, and mostly won’t matter – it’s not something we feel like judges need to have memorized, as long as they remember “Oh, I remember this changed – let me see if that changes affects this call I’m taking.” So it doesn’t make a good evaluation question, at any level.

The case of the missing card

Our last questions today is Q2387, a policy question from the standard quiz. It has a 57% pass rate, and what’s interesting is that the wrong answers were spread out among all four options. 

After a Murders at Karlov Manor draft, how many cards should Alison have in her pool which are legal to play in her deck?

1. 42.
2. 45.
3. Up to 42, depending on how many cards from The List Alison drafted.
4. Up to 45, depending on how many cards from The List Alison drafted. 

I generally discourage judges from trying to game a test – just pick the answer that seems most correct, and don’t worry that this is the fifteenth A in a row (this should naturally happen to one out of every 268,435,456 exams, if there are no multi-selects and there are always four answers). However, if you read this question and think “This seems really obvious – a booster pack has fifteen cards, so it’s just 15×3!”, consider why we’d be testing you on basic multiplication.

We put this question on the test because the shift to play boosters is something judges should realize changes a few things. First, when you tell players to make sure they have the right number of cards in their packs… you should know the right number! Second, every card in that pack is playable, The List cards included (and if you haven’t drafted Syr Konrad in MKM, you’re missing out).

This is also relevant for judges who are working large events where we have called drafts and judges may need to step in to call a pod that fell off the clock – make sure you start your app at 14 and not 15 or you’ll have issues (and don’t do called drafts when you have only one pod).

This question is going to be discarded. It’s relevant in exactly this moment in time, when the switch is being made to play boosters, but in a few sets, this change will just be normalized and you’ll feel like a dinosaur (wait, that was last set…), starting conversations with “Remember when draft boosters had fifteen cards, and they were all from this set…?”

Case Closed

That’s it for the Murders at Karlov Manor quiz update. Join us very soon for the Outlaws at Thunder Junction quiz, and stop by later this month if you’d like to take a practice quiz to improve your rules and policy knowledge.