First, I want to thank the community for participating in this Q&A session. I’ve always been amazed at the passion you guys show for the game. I want to be completely transparent with you about the technical issues we’re having with the game, and what we’re doing to improve things.
It’s unfortunate that we’ve let sentiment get to the point where our valued players think that we’re neglecting the game. On the contrary, we’re spending most of our engineering effort improving the stability of the game and the performance of our battles.
Without further ado, let’s get to some questions and answers!
Lag during battle can be caused by a number of factors:
We’ve also noticed this becoming more and more common. The culprit here is that our game client is using too much memory. As some of you pointed out, the client is allocating upwards of 3 Gigabytes of RAM when fully loaded. We plan to address this by reducing the amount of data needed to load the game, and by compacting the data already there. Both of those efforts are in progress now.
There are reports online about newer versions of Flash Player crashing when graphics acceleration is enabled. You could try disabling it by right-clicking on the game client, clicking “Settings” and disabling Hardware Acceleration.
In 2018 or 2019, we are going to have to transition to using a different game client technology. The least risky course of action is to convert our (significant) codebase from Flash to the programming language Haxe and use HTML5 Canvas or WebGL to do rendering. That way, we’d be able to keep the game on the web, avoid using another plugin, and take advantage of performance optimizations available in modern browsers. We also want to avoid completely reimplementing the game client as that would take years.
Going this route would reduce startup time, but it wouldn’t reduce the latency experienced during battle, which is the most detrimental type of latency for players.
Once you’ve downloaded the game client and game assets, most of that content is “cached” locally on your browser. Subsequent loads of the game will not need to download that content again, since it can be read from your browser’s cache. Web sites use this technique commonly to minimize the amount of requests the browser needs to make.
We are looking into reducing the number of messages sent from our battle servers to the client, as well as shrinking message sizes.
Combatting hacking and cheating is something we’re continually doing. The next major cheat fix is going to secure components, which is one of most cheated aspects of the game remaining.
An abridged history of cheating in War Commander:
Early versions of the game were entirely “client-authoritative”, meaning that the game’s client was responsible for data integrity. So basically the client could send any data it wanted to the server, and the server would accept it without any validation. Since it’s quite easy to hack our game client (and any game client, for that matter) and send phoney data, we needed to change this.
Over the last few years, we’ve moved to a “server-authoritative” model, making our servers responsible for validating data before we save it. So even if someone is able to hack our client and send invalid data, the servers would reject it. Last year, we made units and platoons server-authoritative. This year we secured missiles, trophy buildings, and soon: components. When we roll that out, all invalid components will be removed from all units in all platoons.
We are looking at doing that in the fourth quarter of this year. In addition, we are planning on more optimizations with our hosting provider which will make the transition easier. It will also lower the amount of downtime we experience due to bad RAM and unexpected server crashes.
We do lots of testing every time we release. Unfortunately, the emergency maintenances we’ve had recently wouldn’t be discovered by our testing because they only show themselves once we have thousands of players online. They are caused by long-running Java ‘garbage collection’ events, and tend to happen within the first hour after we bring the game back up.
We are doing a lot of garbage collection tuning, and expect this issue to be less of a problem going forward. Longer-term, we’re minimizing the amount of game data we need to keep in Java memory.
I have a lot of strong feelings about how server software should be developed. The architecture of War Commander’s servers sometimes makes my hand involuntarily curl into a fist. To be clear, War Commander has been a very successful video game, and our forefathers on this team before us who built the bones of this game deserve credit for that success. However, technology and server architecture have come a long way since the bones were laid, and we deal with a lot of what has now become technical debt.
Currently, War Commander keeps certain types of player data synchronized across all our servers. The main advantage of this it reduces load time by keeping that data accessible quickly. This strategy is analogous to a browser’s cache, where we load the data once, then save it locally so we can access it quickly. On the surface, it sounds like it should speed things up, but there are a few of problems:
In my opinion, our War Commander’s forefathers spent too much time solving intellectually challenging problems in elegant but unnecessarily complicated ways, to the detriment of delivering robust, fault tolerant software. Lots of features in War Commander (e.g. alliances, leaderboards, attack log, raids) suffered from two common software development pitfalls:
My philosophy is that we should be delivering good user experiences first and foremost. Brittle game features breed negative sentiment. Developers also need to value “non-functional requirements”: attributes of software that are unrelated to features, but are very important to the overall quality of the product. Some examples: reliability, robustness, security, readability, testability, scalability, extensibility, maintainability. In my experience, those attributes are under-appreciated by software developers, but I’ve seen the positive impact they can have on quality, sentiment, and even our team’s enjoyment of working on the game.
How We’re Changing It
We’re switching out the internals of many of our features in an effort to simplify our codebase and eliminate the requirement to keep data synchronized across N server nodes. We recently did this with leaderboards, and now retrieving leaderboard data is WAY faster. We’ll be doing the same thing with alliances soon.
In software development terms, we’re moving from a ‘stateful’ data model to a (mostly) ‘stateless’ one. With a stateless model, we can rely on well-known data storage tools to manage game data instead of relying on a complicated and brittle in-house synchronization mechanism. It might add a small bit of time to each request, but we’re talking milliseconds. (thousandths of a second). I would happily trade that for a reliable, fault-tolerant server.
Haha, this is not true, but we haven’t added new functionality to mines or mine factories for quite some time. Blame WrongThinker for that one! Would you like to see more gameplay around mines?
We probably won’t remove trophies outright, but we might set some limits on them. For example, we may only allow players to show 5 trophies in their base at once. A few trophies shouldn’t slow down battles much; the problem is animating them. Trophies use large sprites with frequent animations which means Flash spends a lot of extra time rendering.
Soon, we’re going to roll out a setting to disable trophy animations during battle.
Flash Player supports *some* GPU hardware acceleration, but it’s quite limited unfortunately. Once we move the game away from Flash to a more modern platform like HTML5 Canvas or WebGL, we’ll be able to take advantage of more GPU offloading.
100% agree. We’re going to turn off auto-registration on kixeye.com. I’ll spare you a rant about the wisdom of adding auto-registration in the first place.
This is not in the roadmap, I’m afraid.
I would love to do this, but it’s currently not on the roadmap. In theory, this shouldn’t be very hard to do. First, we’ll need to minimize the amount of traffic sent by each battle. The last thing I’d want to do is slow down battle performance due to a large number of spectators.
Game designers are primarily interested in creating fun and challenging experience in War Commander. Performance is certainly part of the equation.The engineering team gives guidelines to the design team about what limits to place on unit numbers, building numbers and tech. Historically, this has been a process of trial and error. Lately we have been more methodical with how we measure performance of certain units and tech (some units have better performance than others). In general, more units with complex tech means lower performance. Longer term, we will continue to investigate the types of tech that cause poor performance, which should loosen the restrictions on base designs.
We talk to CS daily, and we have bi-weekly meetings with CS to investigate the most commonly-ticketed issues. If players are not able to open their base, we prioritize those issues first. Honestly, we haven’t given guidance to CS on technical tasks because they already know the debugging steps we would take for a lot of player issues.
I’ve seen no evidence that a higher or lower number of sectors impacts performance substantially. Busy sectors like sector 1 will cause more data to be streamed to your client when you’re browsing the world map, but this wouldn’t affect in-battle performance. The main sources of poor performance during battle I’ve seen are:
I don’t think older features like the mine factory impacts performance that much.
The biggest impediment to transitioning to HTML5 is the sheer amount of effort required to do it. My high level estimate is 6-9 months, so certainly not a trivial project, but we will need to do it eventually.
I can’t give details about our server specs or numbers. Sorry!
We are planning on rolling out more battle servers in geographic regions. Based on your IP address, we can place the battle on the closest server. So if you’re playing in Europe, your battle server will be in Europe. That will reduce the amount of time it takes for packets (messages) from the battle server to reach your client.
As opposed to increasing bandwidth, we want to decrease the amount of data being sent per battle. That *should* reduce network-related issues also.
This is our biggest concern; improving player experience and game stability are what we’re intensely focusing on. The problem for us is that it’s difficult for us to see these disconnects on our end. I very rarely see disconnects in battle, but there are many factors involved:
We have recently added additional metrics to the game, we are analyzing that data now, and soon you’ll be able to see some of the above info yourselves. Sending that info with your tickets will help us diagnose the issue.
This is a known issue with our current implementation of alliances, where tags take forever to load in. (It’s related to my data synchronization rant from another question). This will be eliminated or minimized with our new alliances implementation, coming in a few weeks.
Thanks for the awesome questions everyone. I LOVE that so many people are genuinely curious about the game and what we’re doing to improve it. Moving forward, I’m going to write a post to you guys on a regular basis “straight from the engineering team” so we can describe technical changes that are coming up. I’d love to continue getting your feedback as we work to make War Commander better.
* ** ***ORIGINAL POST*** ** *
Hello everyone, WrongThinker here.
I’m excited to introduce our first regular Q&A! This Q&A features our Technical Director: Sobokop! Here’s how it works...
Remember to keep your questions technical (we’ll do a more design-oriented Q&A at a future time). We look forward to your questions!