LogInA CTO and an Integration Manager of multiplayer game session hosting Gameye talked about how the Chivalry 2 launch went, disclosed some technical challenges they faced, and explained how to manage big audiences.
Gameye Multiplayer Game Session Hosting
80.lv: Please introduce yourself and your team. Where did you study? What companies have you worked for? What projects have you contributed to?
Elmer Bulthuis: My name is Elmer Bulthuis, and I'm a CTO at Gameye, I wrote my first line of code when I was 8 years old and I’ve never really stopped, I love doing it.
Junior Mba: I am Junior Mba, an Integration Manager at Gameye. My first job in games was with a company called Game Analytics, before that I was in tech support roles for various retail and finance companies.
Gameye is essentially a provider of server infrastructure and support for multiplayer games, which we provide at scale. Most of our clients are game studios or publishers, that’s not to say there could be other use cases for our technology, but this is where our focus is right now.
As a business Gameye has been around for about 5 years. What makes us stand out is that we’re agnostic. We can fit with whatever tech you use, you don’t have to switch tech to work with Gameye, we’ll fit in with whatever you’re using.
We’ve worked on a few game launches, we have internal projects as well that we’re working on. We’re very proud to work on the launch of Chivalry 2. That’s one of the biggest launches we’ve had so far at Gameye.
Chivalry 2
80.lv: How did you get into the Chivalry development team? How did this story begin for you? What was your team focused on?
We’ve been working with the Chivalry 2 team for about two years now, so we’ve had a lot of time to integrate with them as we’ve been working together through the early game alphas and testing, etc. It’s taken a long time from first contact to the game launch, and that’s given us time to really get to know each other and understand what our needs were and build a relationship because I think that if you like each other it’s much easier to work together.
The Launch
80.lv: Let’s talk about the launch. What was it like a month before the release? What was your plan?
Elmer Bulthuis: The launch itself was great actually in many ways, it was the biggest launch we’ve worked on, so we did our test calculations and test simulations and everything looked fine for launch. We did all of our optimizations to make sure nothing would break. However, when the time comes for launch, it’s pretty stressful.
For the launch of Chivalry 2, there was a slight anomaly that no one would have even noticed. Our systems can handle thousands of game sessions starting at the same time. However, what we didn’t notice, until we saw the data afterward, was that we had so many matches starting simultaneously – literally, at the same split second, 300 matches started. Now that’s a load which is hard to test, as it’s not usual. It’s pretty rare. Our system coped with the load, and it was only looking at the data afterward that we realized that it was pretty special. But you can never really test these things, the things you test tend to go right, but there are things you don’t expect and you cannot test those.
Technical Challenges
80.lv: What were the main technical challenges? What are the tricks when launching a global multiplayer game? How did you welcome such an audience? What were the steps? How did you approach different regions and time zones?
Elmer Bulthuis: There was a bug on our part, it only happened with Chivalry 2 and not with any other titles we've been working on. This bug was really weird. There was some kind of hiccup on the server's side and it was weird because we didn’t really know where to begin to try and fix it.
Eventually, we found out it was actually a mixture of a small bug in Chivalry 2 and a small bug with Gameye. What was going on was that Chivalry was asynchronously blocking the game loop and Gameye was reading it synchronously, so there was a little bit of a delay. The event was written and there was a little delay before we read it. It’s a weird thing to come across and once we spotted it, it was simple to fix.
Challenges at Scaling
80.lv: What are the main bottlenecks when scaling a game at the beginning? How does one segment and control waves of new players?
Junior Mba: when scaling a game, first we anticipated the nature of the launch and the incoming numbers. Working with the Chivalry 2 team we created numerous scenarios of what we were anticipating from different regions. Then we allocated those players to our servers so that when we saw a spike in numbers we could scale accordingly, scaling laterally and into the cloud as well where needed.
We employ a different approach for different regions and time zones. So it’s usually a case that Europe has a peak, then, as the day goes on, that peak dips. Then a peak comes up in the US, and, when that dips, there’s a peak in Asia, and so on. For Gameye, we had to make sure we have enough servers to cope with all of those peaks. And one of the cool things about Gameye is that we don’t cherry-pick server providers, we work with as many providers as possible. That means we have backups in all of those different areas.
Managing Big Audiences
80.lv: Let’s say there’s an indie team that needs to launch a multiplayer game that might be a huge success leading to uncontrollable audiences. How difficult is it to manage this aspect today? Are there some open-source solutions that make it easier to deal with the launch period?
Junior Mba: Managing the unpredictability of people loving your game and sudden increases in player numbers is something we’re asked a lot. For example, we have one client right now worried about the cost to them if their game takes off. And we’re able to say to them, focus on making the game, you only pay for what you use. We give you the flexibility to try your game out in different areas, you can literally test it anywhere. The way we run our farm is that we allow you to try anything to see if it works for your game. You can tailor your player presence or server presence depending on what you need. We don’t charge you for X amount of servers, we only charge you for what you use.
Elmer Bulthuis: That’s one of the reasons we want to work with developers when they are in those early development phases, where we can encourage them to experiment with regions and integration with matchmakers. We can integrate with any matchmaker and it doesn’t cost to try it out, so make sure you use the one which suits your game best.
Predicting Launch Numbers
80.lv: Could we also discuss analytics? How can developers forecast the launch number? What are some of the possible ways here?
Junior Mba: It is one of those areas which is unpredictable to manage. To a certain extent, you can predict your baseline of players with alphas and betas, but then at some point, the real-world scenario kicks in. You launch your game and a popular streamer plays your game and you maybe hadn’t anticipated that. You should always overestimate so that you are ready for anything then it may settle in some middle ground somewhere.
Elmer Bulthuis: If you can imagine your worst-case scenario ‘no one likes your game’ and your best-case scenario ‘everyone loves your game’ – now, if the difference between the two is really big, then you need a lot of flexibility because the risk is big. If the distance is small, then you need less flexibility. The risk could also be realized through spikes from streamers, etc. – so it’s important to make sure you have flexibility. This is one of the reasons why we launched Gameye because you only make a first impression once and it needs to be good. When you launch your game it all must work, as people may not come back to it. So we provide that flexibility to be able to make sure your launch goes off without a hitch.
Afterword
80.lv: Any last piece of advice?
Elmer Bulthuis: Don’t believe the hype – don’t just do something because everyone else does it, only do it if you need it. Ask yourself, could we build this, or could someone else build it for you? But please don’t just do it because others are doing it. Building a game is a very expensive piece of software and you need the flexibility to make it exactly the way you want.
Junior Mba: Test your online elements as early as possible. Even just you and other devs testing the matchmaking and server orchestration. Think about it as early as possible and try it as early as possible. One may be more flexible than the other and it allows you to pick what’s best for you. Focus on your players and where they’re based, trial this through beta testing and figuring out where the majority are located, and then transfer that to your server locations so it matches up to where your players are.