When there is smoke... there are bug reports.

Today we are here with Rose and Emma who handled quality assurance for Cipher Zero. Both of you do a lot more than just QA. Can you tell us about your other roles?

Rose: My main role here at Zapdot is community manager: taking care of the Discord, Steam forums, social media, and the like. I also help out marketing and outreach and I also occasionally bully our creative director when it comes to making sure we hit deadlines.

Emma: I hop around a bit, although I’m mostly QA at the moment. But I’ve done a little production, a little customer support, a little writing, a little research…basically, I’m happy to give anything my best shot.

For those of us not super familiar with QA, there’s the internal tests side and the external tests side. Emma, you worked on the internal side, making sure that we have a solid enough product to offer to playtesters. What does your process look like?

Emma: A big part of getting started with QA is making a test plan - this is essentially a big document or spreadsheet where you make a line for every action the player takes in the game, and you use it to track where there are bugs and issues over time. As the game gets worked on and added to, you continue to add tests to cover the new functionality. It helps ensure you aren’t overlooking any part of the player experience. The test plan for Cipher isn’t that big compared to some of the other projects I’ve worked on, but it still can take a while to do a run through.

In addition to the full test plan, it’s important to have a smoke test too. This is a really short version that identifies the most critical features - like, does the game start? Can the player submit solutions? That way you can quickly find super critical issues and make sure the team is aware.

What are some non-QA related skills that led you to potentially approaching the process differently compared to someone else in the same position?

Emma: I’ve also gotten some training in accessible player experiences, so accessibility was a big frame I tried to look through while testing the game from the early stages of development. That’s when you need to start thinking about accessibility in a game - from the start. It’s way easier than trying to cram in changes later in development, and it ultimately just makes your game better too.

There are a lot of useful resources out there that can help developers with this. We pulled a lot from the Game Accessibility Guidelines, for example. I think the team overall did a great job considering accessibility and identifying what could be done to allow more players to experience the game

On the external side, this is a question for Rose. What does play-testing on Cipher usually look like? Can you give us a bit of a run-down?

Rose: For us, communication before the actual playtest starts with the playtester picking a timeslot that works for them. We help get them sorted out with a build that we want them to test, make sure they’ve got any additional instructions for that, such as any instructions on how to swap builds or access beta builds via passwords.

Once the playtest actually starts, I talk them through what we’ll be doing for that day, what we expect from them. I’m there on the call with them as they go through whatever we’re testing, and I’m there as help if they get stuck but not in a way that gives the answer, more like helps redirect their thoughts towards what could help lead to the answer.

Playtest is over when they finished what we were testing, we run out of time, or they hit a point where they feel like they’d walk away from the game naturally if they were playing on their own. There’s a follow up survey to ask them about the mechanics they encountered, things they liked, things they didn’t, and an invite to come back for more testing.

What are some of the key things you watch for when you observe a playtester? How do you usually interact with the playtester?

Rose: Usually a playtest is quiet, I don’t talk much except when we’re talking through strategy or ideas on how to solve the puzzle. I’ve had playtests where I only talked at the very beginning to get the person set up, and then two, three hours later at the end when they’re done and we talk about the follow up stuff. Otherwise, it’s just them working through the puzzles. Some testers loved to chat, and it helps them work through the puzzles while not actually talking out loud about the puzzle itself.

Each playtester is different of course, so I’ve got a short time to learn habits and quirks, especially with new testers. I have to watch out for when they’re getting frustrated or stuck and step in at the right moment to not let them bang their head against the wall too long but also not potentially break concentration when they’re almost there. Getting them to vocalize when they’re starting to get to that point is really helpful but not all playtesters are going to speak up, even when encouraged, so I’ve got to keep an eye out on that.

QA aside, how do you deal with the non-QA related challenges that come with running QA? For example, no-shows for playtesters, broken builds, organizing playtest schedules, and other logistics?

Rose: No-shows are always a toss up. Sometimes it’s just life gets in the way, gotta reschedule, which is no problem. Sometimes, it’s just pure ghosting, like they don’t want to hurt our feelings by backing out, which I can get! But it’s also okay to say, “Hey, I change my mind.”

Broken builds usually also lead to reschedule, unless we can squeeze a quick fix and get the new build up and going. Scheduling is always fun. I’m thankful for a flexible schedule myself, which lets me meet the needs of playtesters’ schedules. They’re people with lives, they’re already doing us a massive favor by playtesting, no reason not to accommodate the best we can.

During Cipher Zero’s playtest, what was the biggest consistent problem you encountered during playtesting and how we ultimately resolve it?

Rose: I would say trying to help guide a tester through a puzzle without giving them the answer or giving too much direction that it felt like I did too much. That just takes time to figure out, and really understand the puzzle, the mechanics, the solution, but also the tester. There’s no real solving this sort of thing as a whole, you just come as prepared as you can.’

Emma: I didn’t do a lot of running playtests, but I’m with Rose here from what I did do. My instinct when I get stuck on a puzzle (or honestly any challenge in any game) is to stop and come back to it the next day, but you can’t really do that in a playtest. The time allotted is the time allotted, and you just have to figure out how to roll with that.

Here’s a broader question. In your opinion, what are some of the biggest QA botches you’ve seen in the industry?

Rose: In recent memory? There’s a few, though one stands out. It should have been a hit but the launch was absolutely riddled with bugs, crashes, performance issues that just made the game unplayable. The issues had been brought up by QA but crunch time led to many of the bugs to remain and leadership pushed to launch with all the issues still there. The game didn’t feel like the game the devs wanted to make until a few years down the line and who knows how many patches.

Personally seen: Two months before a game’s launch, I told the devs, ‘Hey, if you pick this character with these specific units, when you load into the map, the game crashes.” It was 100% reproducible but they didn’t fix it for launch. This because the combination wasn’t something the meta players were using — and in all honesty, it was a silly build that worked if you only knew how it worked — so those in charge didn’t see it as a reason to fix it.

The problem came is where the main story quests had that exact combination of character and specific units, so the moment you got into the mission, the game crashed. It took them multiple days to fix it before anyone could actually progress through the story, the Discord and Steam forums were a nightmare during those days.

Emma: As Rose pointed out, I feel like most QA botches aren’t completely on QA themselves but on harsh schedules or communication/prioritization issues at higher levels. Not to mention, as games get bigger and more complex, there simply might be bugs that don’t make themselves apparent until after launch, no matter how much QA you do beforehand. None of that is to excuse rough launches, but buggy games aren’t necessarily buggy due to bad QA but a whole host of other possible compounding issues.

I think QA is one of those departments where they can go unacknowledged for a job well done, because it’s sort of invisible, you know? But then once there’s a problem, that’s where the blame goes first, justified or not.

This is your chance to convince the world QA has cool moments! Tell us, what was the best thing you experienced during Cipher Zero QA?

Rose: Some of the best moments came from playtesting. Some would complete the level and then immediately go back and look at all the details of the world. I had one playtester who was like, “Can I just sit here for like a minute and listen to the music? This is cool.” It’s those little things that add up and makes all of this totally worth it.

Other than that, just seeing things come together piece by piece and getting to make sure those pieces fit together is really satisfying. Watching the effort to bring different particle effects together, new or update artwork, new little bits of details that were half-baked thoughts said in passing made into something real and amazing.

Emma: I’ve hopped on and off active work on Cipher between other projects, and every time I came back, it was like I got to discover the game all over again as things were added, changed, and adjusted. It’s really satisfying to be able to see a game grow like that, and QA’s the place where you’ve got the best view.

Lastly, is there anything you want to tell the players (and players-to-be) of Cipher Zero?

Rose: I saw my first screenshot of this game a few weeks after I was hired at Zapdot. The art was still simple, but there was something about the muted palettes and the style that made me go, “I want to work on that game”.

I love puzzle games but I’m terrible at them; I’m much more of a Nonary Games/Danganronpa type of person. Working through the game, the puzzles, I just fell further in love with it, even the puzzles that took me a while to figure out. Don’t let it being a puzzle game stop you from playing it. CIPHER ZERO is just something special that I think a lot of people would it enjoy it if they give it a chance.

Emma: Take your time! Play through a few puzzles a day, really soak it in. I never used to be much of a puzzle gamer, but I’ve found recently that I enjoy them a lot more when I treat them less like homework and more like a daily crossword. There’s no need to rush, and nobody’s grading you. So good luck, and have fun!