Playtesting: Why It Is Worth Every Penny

80 Level Research Team presented a new report dedicated to playtesting, its importance for the game industry, types of playtesting, and how it can be used to improve marketing.

Playtest, or users/player/UX research, is an essential part of the game development process. These co-op activities provide gamers opportunities to explore the mechanics, levels, and ideas, and they help developers to make games in a shorter time with much lower costs.

To learn how developers approach playtesting, what outcomes they expect from them, what difficulties they encounter, and how they overcome these challenges the 80 Level Research team conducted a series of interviews with experts from game dev studios and agencies that specialize in playtests. Below you can find some highlights from our report. Don’t forget to download the full version here.

Playtesting helps find bugs and improve marketing

Playtests and marketing should start at the early stage of development. Ideally – the initial research should be completed before laying down the first line of code or a sketch. Based on the feedback, developers can understand the pain points, barriers, and motivation of users, build their games around these needs and even alter the initial concept.

Playtesting results affect marketing strategies and help to forecast the revenue for the game. It plays a significant role in determining the business model. Testers from the playtest target audience can potentially be the first customers (early adopters) of the game. The marketing department should be somewhat involved in the testing, as it will make the distribution and promotion process easier and more efficient in the future.

Growth marketing associate at Playdew Urooj Iqbal:

The biggest mistake made by developers is that they do not involve marketing, consumer, or another non-dev point of view. It could be an artist, UI/UX designer, a QA person, a marketer, etc. who can provide some insights from their experience. Each department needs to work together to make a perfect product.

Playtesting affects many stages of the development and distribution of games such as:

- The whole user’s experience
- The whole gameplay
- Marketing strategies
- Revenue estimates

Testing for marketing purposes may be a bit expensive. But if you don't focus on that, it can turn out to be even more expensive if, during the last stages of development, you realize that you were targeting the wrong people the whole time.

What types of playtests exist?

Playtests that are focused on the technical side of the game can be done automatically with different types of software. They’re intended to make sure that the core and server systems are responding correctly, with the main focus being on the game engine itself as well as the system integrations. User experience, gameplay mechanics, etc., where the technical and marketing sides of the game meet, are usually tested manually using, for example, surveys, interviews, and hands-on demos.

To test their builds, developers can use in-house testers (and invite colleagues from other departments) or external groups of people can be attracted. The only way to get valuable insights is to attract real players to test the game. Friends and colleagues will always be biased, and people from the gamedev industry as well as research specialists will have modified perceptions. Only players from the target audience or a wider group of potential users will help you build something new and enjoyable.

Playtests, like other research projects, can be split into two categories: qualitative (this type of research answers “how” and “why” questions) and quantitative (which answers “how many/much”). The goal of a quantitative approach in playtesting is to find numeric estimates, and the goal of qualitative playtests is to find out more about emotional and behavioral reactions, players’ motivations, and their perceptions.

In the playtesting field, an example of an instrument for studying reactions and their sources is the dual recording: i.e. recording the respondent's screen as well as their face. Before COVID, hands-on controllers could be recorded as well (how someone is holding a device, how they shake it, hit it, etc). In terms of mobile games, observations are made on where the player tries to click as well as what objects or buttons they perceive as interactive ones (when it could be a part of a background). Other components like motivation and perception can only be studied via different kinds of interview approaches.

CEO at Sense.Vision Alexander Dzyuba:

There are 2 research methodologies in playtests:

- Qualitative studies. They explain things without measuring and giving any certain numbers.
- Quantitative measure things with numbers and exact points, they allow to give marketing predictions and say what the measurement error is. Usage of in-game statistics is an example here.

A method in-between – hall/home-test: there are a lot of participants to build mathematical distributions, but at its core, it is qualitative research because it answers "how/why".

And there are 3 methods of research (detailed information about each of these methods and how they can be used in playtesting are described here):

1) interviews

2) observations

3) experiments

Do you do playtests in-house or outsource them?

Big studios assemble research teams and develop their in-house testing services, while others use external test automation software which can be added to the project (like GameDriver, Impactqa, modl AI, etc.) or outsource playtests to an agency (like Sense.Vision, PlaytestCloud, Antidote, etc.). In-house playtests are cheaper and more secure, but only if a company has 10+ projects and 3 – 4 games to test at the same time. Agencies usually have wider expertise and are able to recruit audiences and conduct playtests faster.

The process of finding external game testers can be challenging. Companies can recruit people who have already worked with them or find enthusiasts on Facebook and Twitter. It is also effective to create Discord groups for getting useful feedback from volunteer testers. There are some special platforms that can help gather focus groups. Usertesting.com connects companies and testers that fit a given profile. Developers schedule calls and pay per player. Due to the complexity of conducting stress tests with real users, a studio can also use special tools that simulate them.

CEO at Lila Games Joseph Kim:

The tools and platforms used for testing depend on the purpose of the test. At Lila Games, we used usertesting.com for small focus groups. We have searched for our own internal focus groups through Facebook ads and Twitter. The more scalable way to do this with a larger audience is to create Discord groups and get feedback through them. Discord groups’ members are usually either enthusiasts who want to know more about a game or early adopters.

The frequency of testing is dependent on the company size, the number of builds, capacity, roadmap, and budget.

Playtests have to be conducted consistently or when a developer needs to make a game-changing decision. Tests with in-house participants can be done weekly or even daily, especially when there’s an in-house software or enough budget for outsourcing, while tests with real players, especially manual tests, require additional team resources or budget for outsourcing.

Operations Lead at Super Evil Megacorp Tamir Nadav:

Playtesting is useful in a number of ways, but the biggest takeaway for the company is understanding what your product is going to be because when you are at the playtest stage, it is your last chance to form your understanding of the product concept.

For mobile games, in the stage before launching the game Tamir looks at KPI through crash percentage, retention, session time, number of sessions per day, and everything that tells about a user's first experience especially. For monetization system testing, Tamir would look at spending metrics.

Many studios don't test games often enough, as much as they would like to.

Among the main problems in conducting playtests, developers note difficulties in selecting the target audience. Developers also fear for security when they provide access to the game to testers. The testing process and results analytics are very resource-intensive, and there is also the human factor with human limitations: blurry eyes, loss of concentration, etc. Many studios don't test games often enough, even though they would like to. External playtesting agencies and platforms can help indie developers by recruiting a target audience or creating a user base for testing, as well as improving security by providing legal support for developers and testers.

Founder of Cykyria Benedikt Engelhard:

Virtual reality playtesting is difficult because today’s users have many devices, like different VR headsets, with various resolutions or different headphones. It is hard for developers to conduct tests in such a way that everything functions properly.

In Cykyria developers don’t have time to test games often. We either test stuff or work on stuff that needs to be done. We didn’t find useful platforms for playtests that are connected with VR gaming.

Automatization is the future of playtests

Interviewees see the future of playtesting in the automation of technical tests, which will free up testers' time for more creative tests that automated software cannot do. The main goal of playtesting automation in GameDriver, for example, is not to replace testers, but to free up their time so testers can be sure that the core functionality works correctly and focus on other characteristics that make a game enjoyable.

Experts also consider integrating research into the gameplay and creating a special version of a game where players will be informed of an ongoing study while playing. Sense.Vision, for example, is working in the direction of dedicating playtesting analytics to algorithms or AI (artificial intelligence). It doesn’t mean that there will be no people; they will just solve tasks of a higher level and have more opportunities to make much higher quality games.

Co-founder of GameDriver Shane Evans:

Testing was introduced in the mid to late 90s and helped to alleviate some of the pressure from software companies that are trying to ship faster and faster. The development process is very complex but right in the middle of it, there is another manual process of testing driven by people who have only two hands. Either you increase the number of hands or you make those hands more efficient. That's what I think automated testing can do for the industry (more information about automatic playtests could be found here).

The automation of the repetitive tests will become an ordinary thing, the developers will make much higher quality games because their focus will move from the work of the game to its other characteristics like making a game more fun and enjoyable.

Playtesting: Why It Is Worth Every Penny

Playtesting helps find bugs and improve marketing

What types of playtests exist?

Do you do playtests in-house or outsource them?

The frequency of testing is dependent on the company size, the number of builds, capacity, roadmap, and budget.

Many studios don't test games often enough, as much as they would like to.

Automatization is the future of playtests

Download the full version of the report here.

Join discussion

Comments 0

You might also like

We need your consent