Commanders,

First, I want to thank the community for participating in this Q&A session. I’ve always been amazed at the passion you guys show for the game. I want to be completely transparent with you about the technical issues we’re having with the game, and what we’re doing to improve things.

It’s unfortunate that we’ve let sentiment get to the point where our valued players think that we’re neglecting the game. On the contrary, we’re spending most of our engineering effort improving the stability of the game and the performance of our battles.

Without further ado, let’s get to some questions and answers!

Why does it seem like some players aren't affected at all by the lag/performance issues and others are?

Lag during battle can be caused by a number of factors:

1. More expensive battles running on the same battle server that are contending for resources.
2. Lots of building animations cause Flash to redraw the entire screen. We’re working on a way to optionally turn off those animations during battle.
3. Some players have large, crufty bases which contain a lot of now-unused data.
4. Battles that contain a lot of units. Our halloween events, for example, can contain hundreds of zombies, which add a lot of processing and drawing time.
5. Playing on a high-latency internet connection like mobile or satellite networks. Satellite networks, for example, take almost a second for a message to from your PC to our server. There’s nothing we can do to lower that, unfortunately.
6. Geographic proximity to our battle servers. Generally speaking, the lower the ping time between you and our battle servers, the better your experience will be.

Shockwave crash is far too common. Are there any plans to address this?

We’ve also noticed this becoming more and more common. The culprit here is that our game client is using too much memory. As some of you pointed out, the client is allocating upwards of 3 Gigabytes of RAM when fully loaded. We plan to address this by reducing the amount of data needed to load the game, and by compacting the data already there. Both of those efforts are in progress now.

There are reports online about newer versions of Flash Player crashing when graphics acceleration is enabled. You could try disabling it by right-clicking on the game client, clicking “Settings” and disabling Hardware Acceleration.

What will happen when flash is no longer supported in 2020?

In 2018 or 2019, we are going to have to transition to using a different game client technology. The least risky course of action is to convert our (significant) codebase from Flash to the programming language Haxe and use HTML5 Canvas or WebGL to do rendering. That way, we’d be able to keep the game on the web, avoid using another plugin, and take advantage of performance optimizations available in modern browsers. We also want to avoid completely reimplementing the game client as that would take years.

Could converting to a downloadable app reduce latency for players?

Going this route would reduce startup time, but it wouldn’t reduce the latency experienced during battle, which is the most detrimental type of latency for players.

Once you’ve downloaded the game client and game assets, most of that content is “cached” locally on your browser. Subsequent loads of the game will not need to download that content again, since it can be read from your browser’s cache. Web sites use this technique commonly to minimize the amount of requests the browser needs to make.

We are looking into reducing the number of messages sent from our battle servers to the client, as well as shrinking message sizes.

What are you doing about hacks in game?

Combatting hacking and cheating is something we’re continually doing. The next major cheat fix is going to secure components, which is one of most cheated aspects of the game remaining.

An abridged history of cheating in War Commander:

Early versions of the game were entirely “client-authoritative”, meaning that the game’s client was responsible for data integrity. So basically the client could send any data it wanted to the server, and the server would accept it without any validation. Since it’s quite easy to hack our game client (and any game client, for that matter) and send phoney data, we needed to change this.

Over the last few years, we’ve moved to a “server-authoritative” model, making our servers responsible for validating data before we save it. So even if someone is able to hack our client and send invalid data, the servers would reject it. Last year, we made units and platoons server-authoritative. This year we secured missiles, trophy buildings, and soon: components. When we roll that out, all invalid components will be removed from all units in all platoons.

When will we roll out regional battle servers?

We are looking at doing that in the fourth quarter of this year. In addition, we are planning on more optimizations with our hosting provider which will make the transition easier. It will also lower the amount of downtime we experience due to bad RAM and unexpected server crashes.

Why isn't there any testing done after maintenance before the game goes live again?

We do lots of testing every time we release. Unfortunately, the emergency maintenances we’ve had recently wouldn’t be discovered by our testing because they only show themselves once we have thousands of players online. They are caused by long-running Java ‘garbage collection’ events, and tend to happen within the first hour after we bring the game back up.

We are doing a lot of garbage collection tuning, and expect this issue to be less of a problem going forward. Longer-term, we’re minimizing the amount of game data we need to keep in Java memory.

RANT WARNING

I have a lot of strong feelings about how server software should be developed. The architecture of War Commander’s servers sometimes makes my hand involuntarily curl into a fist. To be clear, War Commander has been a very successful video game, and our forefathers on this team before us who built the bones of this game deserve credit for that success. However, technology and server architecture have come a long way since the bones were laid, and we deal with a lot of what has now become technical debt.

Data synchronization

Currently, War Commander keeps certain types of player data synchronized across all our servers. The main advantage of this it reduces load time by keeping that data accessible quickly. This strategy is analogous to a browser’s cache, where we load the data once, then save it locally so we can access it quickly. On the surface, it sounds like it should speed things up, but there are a few of problems:

1. Synchronizing data across N server nodes is a notoriously difficult problem to solve, with lots of edge cases and many failure scenarios. Server tools like Hadoop and Zookeeper have spent a colossal amount of time and effort solving those issues, and even still there are tradeoffs.
2. The performance advantages are debatable, especially for many of WC’s use cases.
3. Engineering time is an expensive and limited resource. Why spend time solving those problems when we could work on new game features or fixing other bugs?
4. How do you recover from a node failure? Currently, we just restart all the servers.

In my opinion, our War Commander’s forefathers spent too much time solving intellectually challenging problems in elegant but unnecessarily complicated ways, to the detriment of delivering robust, fault tolerant software. Lots of features in War Commander (e.g. alliances, leaderboards, attack log, raids) suffered from two common software development pitfalls:

1. Reinventing the wheel: That is, developing a custom, in-house solution instead of using a conventional approach or off-the-shelf toolset.
2. Pre-optimization: Spending (very expensive) development time to squeeze extra performance out of a feature before you release. It’s often better to release, THEN measure, THEN optimize. Undelivered game features are worthless.

My philosophy is that we should be delivering good user experiences first and foremost. Brittle game features breed negative sentiment. Developers also need to value “non-functional requirements”: attributes of software that are unrelated to features, but are very important to the overall quality of the product. Some examples: reliability, robustness, security, readability, testability, scalability, extensibility, maintainability. In my experience, those attributes are under-appreciated by software developers, but I’ve seen the positive impact they can have on quality, sentiment, and even our team’s enjoyment of working on the game.

How We’re Changing It

We’re switching out the internals of many of our features in an effort to simplify our codebase and eliminate the requirement to keep data synchronized across N server nodes. We recently did this with leaderboards, and now retrieving leaderboard data is WAY faster. We’ll be doing the same thing with alliances soon.

In software development terms, we’re moving from a ‘stateful’ data model to a (mostly) ‘stateless’ one. With a stateless model, we can rely on well-known data storage tools to manage game data instead of relying on a complicated and brittle in-house synchronization mechanism. It might add a small bit of time to each request, but we’re talking milliseconds. (thousandths of a second). I would happily trade that for a reliable, fault-tolerant server.

Is it really true that you guys don't know how to update the mines and mine factory?

Haha, this is not true, but we haven’t added new functionality to mines or mine factories for quite some time. Blame WrongThinker for that one! Would you like to see more gameplay around mines?

If you guys know that trophies do actually make the lag worse, will you be open to simply removing trophies from the game?

We probably won’t remove trophies outright, but we might set some limits on them. For example, we may only allow players to show 5 trophies in their base at once. A few trophies shouldn’t slow down battles much; the problem is animating them. Trophies use large sprites with frequent animations which means Flash spends a lot of extra time rendering.

Soon, we’re going to roll out a setting to disable trophy animations during battle.

Why is it that the game uses a lot of cpu power but not graphics card power?

Flash Player supports *some* GPU hardware acceleration, but it’s quite limited unfortunately. Once we move the game away from Flash to a more modern platform like HTML5 Canvas or WebGL, we’ll be able to take advantage of more GPU offloading.

KIXEYE should make a system where they don't allow players to make a new gaming account without legal email id?

100% agree. We’re going to turn off auto-registration on kixeye.com. I’ll spare you a rant about the wisdom of adding auto-registration in the first place.

Is there any chance that the game may be translated to other languages such as Spanish?

This is not in the roadmap, I’m afraid.

Could we have Spectator Mode like in other games where we could watch a LIVE battle?

I would love to do this, but it’s currently not on the roadmap. In theory, this shouldn’t be very hard to do. First, we’ll need to minimize the amount of traffic sent by each battle. The last thing I’d want to do is slow down battle performance due to a large number of spectators.

How much weight is given to a technical point of view from yourself/other engineers when designing game features, events and targets?

Game designers are primarily interested in creating fun and challenging experience in War Commander. Performance is certainly part of the equation.The engineering team gives guidelines to the design team about what limits to place on unit numbers, building numbers and tech. Historically, this has been a process of trial and error. Lately we have been more methodical with how we measure performance of certain units and tech (some units have better performance than others). In general, more units with complex tech means lower performance. Longer term, we will continue to investigate the types of tech that cause poor performance, which should loosen the restrictions on base designs.

Do engineers directly liaise and coordinate with customer service when issues are reported, especially on a large scale? Also does the technical team give guidelines to CS reps on what responses to give to certain problems. If not, why is that?

We talk to CS daily, and we have bi-weekly meetings with CS to investigate the most commonly-ticketed issues. If players are not able to open their base, we prioritize those issues first. Honestly, we haven’t given guidance to CS on technical tasks because they already know the debugging steps we would take for a lot of player issues.

From a technical standpoint, do you feel reducing the number of sectors in the game had the desired effect that was stated at the time (reduced lag, better connectivity) if so could this lead to reducing the number of sectors again in the future, if not was the number of sectors not the actual problem?

I’ve seen no evidence that a higher or lower number of sectors impacts performance substantially. Busy sectors like sector 1 will cause more data to be streamed to your client when you’re browsing the world map, but this wouldn’t affect in-battle performance. The main sources of poor performance during battle I’ve seen are:

1. Our battle servers send too much data.
2. Playing on mobile or satellite networks.
3. Wonky edge cases in unit pathfinding during battle. (e.g. trying to generate a path into an unpathable area takes too long)
4. Certain types of units and unit tech overusing CPU.
5. Trying to render too many animations.

Do older in-game items and features that are widely known to be unable to be updated (such as the mine factory) negatively impact technical performance in the game? Also do these things prevent a transition from flash to something like HTML5 or is that transition just a whole problem in itself?

I don’t think older features like the mine factory impacts performance that much.

The biggest impediment to transitioning to HTML5 is the sheer amount of effort required to do it. My high level estimate is 6-9 months, so certainly not a trivial project, but we will need to do it eventually.

Can you provide us that would understand the technical details some information on both your server specifications and how many servers there are (Battle and non-battle)?

I can’t give details about our server specs or numbers. Sorry!

Are there plans to increase the number of all server types in order to decrease the load on each server? Or an increase in Bandwidth to the servers? Or any changes in configurations that you have in mind to help with Game Stability?

We are planning on rolling out more battle servers in geographic regions. Based on your IP address, we can place the battle on the closest server. So if you’re playing in Europe, your battle server will be in Europe. That will reduce the amount of time it takes for packets (messages) from the battle server to reach your client.

As opposed to increasing bandwidth, we want to decrease the amount of data being sent per battle. That *should* reduce network-related issues also.

Why are there no concerns over the constant disconnects that players are experiencing?

This is our biggest concern; improving player experience and game stability are what we’re intensely focusing on. The problem for us is that it’s difficult for us to see these disconnects on our end. I very rarely see disconnects in battle, but there are many factors involved:

1. What rogue faction base are you attacking?
2. What units do you have deployed?
3. What is your FPS?
4. What type of network are you playing on? What is your ping time to our servers?

We have recently added additional metrics to the game, we are analyzing that data now, and soon you’ll be able to see some of the above info yourselves. Sending that info with your tickets will help us diagnose the issue.

Why, when we jump, do we lose the ability to see tags?

This is a known issue with our current implementation of alliances, where tags take forever to load in. (It’s related to my data synchronization rant from another question). This will be eliminated or minimized with our new alliances implementation, coming in a few weeks.

Wrapping up

Thanks for the awesome questions everyone. I LOVE that so many people are genuinely curious about the game and what we’re doing to improve it. Moving forward, I’m going to write a post to you guys on a regular basis “straight from the engineering team” so we can describe technical changes that are coming up. I’d love to continue getting your feedback as we work to make War Commander better.

-Sobokop

* ** ***ORIGINAL POST*** ** *

Hello everyone, WrongThinker here.

I’m excited to introduce our first regular Q&A!  This Q&A features our Technical Director: Sobokop!  Here’s how it works...

• Respond to this thread with your technical questions about the game!
• On Friday at noon PST, we’ll compile the questions and present them to Sobokop (so make sure to respond by noon that day)!
• We’ll update this post later that day with as many answers as Sobokop can write!

Remember to keep your questions technical (we’ll do a more design-oriented Q&A at a future time).  We look forward to your questions!

-WT