Q&A feat. Sobokop (Updated w. Answers) (August 2017)

WrongThinker
WrongThinker
BP Game Team
Joined Mar 2013 Posts: 1,713
edited 5 Aug 2017, 8:48PM

Commanders,

First, I want to thank the community for participating in this Q&A session. I’ve always been amazed at the passion you guys show for the game. I want to be completely transparent with you about the technical issues we’re having with the game, and what we’re doing to improve things.

It’s unfortunate that we’ve let sentiment get to the point where our valued players think that we’re neglecting the game. On the contrary, we’re spending most of our engineering effort improving the stability of the game and the performance of our battles.

Without further ado, let’s get to some questions and answers!

Why does it seem like some players aren't affected at all by the lag/performance issues and others are?

Lag during battle can be caused by a number of factors:

  1. More expensive battles running on the same battle server that are contending for resources.
  2. Lots of building animations cause Flash to redraw the entire screen. We’re working on a way to optionally turn off those animations during battle.
  3. Some players have large, crufty bases which contain a lot of now-unused data.
  4. Battles that contain a lot of units. Our halloween events, for example, can contain hundreds of zombies, which add a lot of processing and drawing time.
  5. Playing on a high-latency internet connection like mobile or satellite networks. Satellite networks, for example, take almost a second for a message to from your PC to our server. There’s nothing we can do to lower that, unfortunately.
  6. Geographic proximity to our battle servers. Generally speaking, the lower the ping time between you and our battle servers, the better your experience will be.

Shockwave crash is far too common. Are there any plans to address this?

We’ve also noticed this becoming more and more common. The culprit here is that our game client is using too much memory. As some of you pointed out, the client is allocating upwards of 3 Gigabytes of RAM when fully loaded. We plan to address this by reducing the amount of data needed to load the game, and by compacting the data already there. Both of those efforts are in progress now.

There are reports online about newer versions of Flash Player crashing when graphics acceleration is enabled. You could try disabling it by right-clicking on the game client, clicking “Settings” and disabling Hardware Acceleration.

What will happen when flash is no longer supported in 2020?

In 2018 or 2019, we are going to have to transition to using a different game client technology. The least risky course of action is to convert our (significant) codebase from Flash to the programming language Haxe and use HTML5 Canvas or WebGL to do rendering. That way, we’d be able to keep the game on the web, avoid using another plugin, and take advantage of performance optimizations available in modern browsers. We also want to avoid completely reimplementing the game client as that would take years.

Could converting to a downloadable app reduce latency for players?

Going this route would reduce startup time, but it wouldn’t reduce the latency experienced during battle, which is the most detrimental type of latency for players.

Once you’ve downloaded the game client and game assets, most of that content is “cached” locally on your browser. Subsequent loads of the game will not need to download that content again, since it can be read from your browser’s cache. Web sites use this technique commonly to minimize the amount of requests the browser needs to make.

We are looking into reducing the number of messages sent from our battle servers to the client, as well as shrinking message sizes.

What are you doing about hacks in game?

Combatting hacking and cheating is something we’re continually doing. The next major cheat fix is going to secure components, which is one of most cheated aspects of the game remaining.

An abridged history of cheating in War Commander:

Early versions of the game were entirely “client-authoritative”, meaning that the game’s client was responsible for data integrity. So basically the client could send any data it wanted to the server, and the server would accept it without any validation. Since it’s quite easy to hack our game client (and any game client, for that matter) and send phoney data, we needed to change this.

Over the last few years, we’ve moved to a “server-authoritative” model, making our servers responsible for validating data before we save it. So even if someone is able to hack our client and send invalid data, the servers would reject it. Last year, we made units and platoons server-authoritative. This year we secured missiles, trophy buildings, and soon: components. When we roll that out, all invalid components will be removed from all units in all platoons.

When will we roll out regional battle servers?

We are looking at doing that in the fourth quarter of this year. In addition, we are planning on more optimizations with our hosting provider which will make the transition easier. It will also lower the amount of downtime we experience due to bad RAM and unexpected server crashes.

Why isn't there any testing done after maintenance before the game goes live again?

We do lots of testing every time we release. Unfortunately, the emergency maintenances we’ve had recently wouldn’t be discovered by our testing because they only show themselves once we have thousands of players online. They are caused by long-running Java ‘garbage collection’ events, and tend to happen within the first hour after we bring the game back up.

We are doing a lot of garbage collection tuning, and expect this issue to be less of a problem going forward. Longer-term, we’re minimizing the amount of game data we need to keep in Java memory.

RANT WARNING

I have a lot of strong feelings about how server software should be developed. The architecture of War Commander’s servers sometimes makes my hand involuntarily curl into a fist. To be clear, War Commander has been a very successful video game, and our forefathers on this team before us who built the bones of this game deserve credit for that success. However, technology and server architecture have come a long way since the bones were laid, and we deal with a lot of what has now become technical debt.

Data synchronization

Currently, War Commander keeps certain types of player data synchronized across all our servers. The main advantage of this it reduces load time by keeping that data accessible quickly. This strategy is analogous to a browser’s cache, where we load the data once, then save it locally so we can access it quickly. On the surface, it sounds like it should speed things up, but there are a few of problems:

  1. Synchronizing data across N server nodes is a notoriously difficult problem to solve, with lots of edge cases and many failure scenarios. Server tools like Hadoop and Zookeeper have spent a colossal amount of time and effort solving those issues, and even still there are tradeoffs.
  2. The performance advantages are debatable, especially for many of WC’s use cases.
  3. Engineering time is an expensive and limited resource. Why spend time solving those problems when we could work on new game features or fixing other bugs?
  4. How do you recover from a node failure? Currently, we just restart all the servers. :(

In my opinion, our War Commander’s forefathers spent too much time solving intellectually challenging problems in elegant but unnecessarily complicated ways, to the detriment of delivering robust, fault tolerant software. Lots of features in War Commander (e.g. alliances, leaderboards, attack log, raids) suffered from two common software development pitfalls:

  1. Reinventing the wheel: That is, developing a custom, in-house solution instead of using a conventional approach or off-the-shelf toolset.
  2. Pre-optimization: Spending (very expensive) development time to squeeze extra performance out of a feature before you release. It’s often better to release, THEN measure, THEN optimize. Undelivered game features are worthless.

My philosophy is that we should be delivering good user experiences first and foremost. Brittle game features breed negative sentiment. Developers also need to value “non-functional requirements”: attributes of software that are unrelated to features, but are very important to the overall quality of the product. Some examples: reliability, robustness, security, readability, testability, scalability, extensibility, maintainability. In my experience, those attributes are under-appreciated by software developers, but I’ve seen the positive impact they can have on quality, sentiment, and even our team’s enjoyment of working on the game.

How We’re Changing It

We’re switching out the internals of many of our features in an effort to simplify our codebase and eliminate the requirement to keep data synchronized across N server nodes. We recently did this with leaderboards, and now retrieving leaderboard data is WAY faster. We’ll be doing the same thing with alliances soon.

In software development terms, we’re moving from a ‘stateful’ data model to a (mostly) ‘stateless’ one. With a stateless model, we can rely on well-known data storage tools to manage game data instead of relying on a complicated and brittle in-house synchronization mechanism. It might add a small bit of time to each request, but we’re talking milliseconds. (thousandths of a second). I would happily trade that for a reliable, fault-tolerant server.

Is it really true that you guys don't know how to update the mines and mine factory?

Haha, this is not true, but we haven’t added new functionality to mines or mine factories for quite some time. :) Blame WrongThinker for that one! Would you like to see more gameplay around mines?

If you guys know that trophies do actually make the lag worse, will you be open to simply removing trophies from the game?

We probably won’t remove trophies outright, but we might set some limits on them. For example, we may only allow players to show 5 trophies in their base at once. A few trophies shouldn’t slow down battles much; the problem is animating them. Trophies use large sprites with frequent animations which means Flash spends a lot of extra time rendering.

Soon, we’re going to roll out a setting to disable trophy animations during battle.

Why is it that the game uses a lot of cpu power but not graphics card power?

Flash Player supports *some* GPU hardware acceleration, but it’s quite limited unfortunately. Once we move the game away from Flash to a more modern platform like HTML5 Canvas or WebGL, we’ll be able to take advantage of more GPU offloading.

KIXEYE should make a system where they don't allow players to make a new gaming account without legal email id?

100% agree. We’re going to turn off auto-registration on kixeye.com. I’ll spare you a rant about the wisdom of adding auto-registration in the first place.

Is there any chance that the game may be translated to other languages such as Spanish?

This is not in the roadmap, I’m afraid.

Could we have Spectator Mode like in other games where we could watch a LIVE battle?

I would love to do this, but it’s currently not on the roadmap. In theory, this shouldn’t be very hard to do. First, we’ll need to minimize the amount of traffic sent by each battle. The last thing I’d want to do is slow down battle performance due to a large number of spectators.

How much weight is given to a technical point of view from yourself/other engineers when designing game features, events and targets?

Game designers are primarily interested in creating fun and challenging experience in War Commander. Performance is certainly part of the equation.The engineering team gives guidelines to the design team about what limits to place on unit numbers, building numbers and tech. Historically, this has been a process of trial and error. Lately we have been more methodical with how we measure performance of certain units and tech (some units have better performance than others). In general, more units with complex tech means lower performance. Longer term, we will continue to investigate the types of tech that cause poor performance, which should loosen the restrictions on base designs.

Do engineers directly liaise and coordinate with customer service when issues are reported, especially on a large scale? Also does the technical team give guidelines to CS reps on what responses to give to certain problems. If not, why is that?

We talk to CS daily, and we have bi-weekly meetings with CS to investigate the most commonly-ticketed issues. If players are not able to open their base, we prioritize those issues first. Honestly, we haven’t given guidance to CS on technical tasks because they already know the debugging steps we would take for a lot of player issues.

From a technical standpoint, do you feel reducing the number of sectors in the game had the desired effect that was stated at the time (reduced lag, better connectivity) if so could this lead to reducing the number of sectors again in the future, if not was the number of sectors not the actual problem?

I’ve seen no evidence that a higher or lower number of sectors impacts performance substantially. Busy sectors like sector 1 will cause more data to be streamed to your client when you’re browsing the world map, but this wouldn’t affect in-battle performance. The main sources of poor performance during battle I’ve seen are:

  1. Our battle servers send too much data.
  2. Playing on mobile or satellite networks.
  3. Wonky edge cases in unit pathfinding during battle. (e.g. trying to generate a path into an unpathable area takes too long)
  4. Certain types of units and unit tech overusing CPU.
  5. Trying to render too many animations.

Do older in-game items and features that are widely known to be unable to be updated (such as the mine factory) negatively impact technical performance in the game? Also do these things prevent a transition from flash to something like HTML5 or is that transition just a whole problem in itself?

I don’t think older features like the mine factory impacts performance that much.

The biggest impediment to transitioning to HTML5 is the sheer amount of effort required to do it. My high level estimate is 6-9 months, so certainly not a trivial project, but we will need to do it eventually.

Can you provide us that would understand the technical details some information on both your server specifications and how many servers there are (Battle and non-battle)?

I can’t give details about our server specs or numbers. Sorry!

Are there plans to increase the number of all server types in order to decrease the load on each server? Or an increase in Bandwidth to the servers? Or any changes in configurations that you have in mind to help with Game Stability?

We are planning on rolling out more battle servers in geographic regions. Based on your IP address, we can place the battle on the closest server. So if you’re playing in Europe, your battle server will be in Europe. That will reduce the amount of time it takes for packets (messages) from the battle server to reach your client.

As opposed to increasing bandwidth, we want to decrease the amount of data being sent per battle. That *should* reduce network-related issues also.

Why are there no concerns over the constant disconnects that players are experiencing?

This is our biggest concern; improving player experience and game stability are what we’re intensely focusing on. The problem for us is that it’s difficult for us to see these disconnects on our end. I very rarely see disconnects in battle, but there are many factors involved:

  1. What rogue faction base are you attacking?
  2. What units do you have deployed?
  3. What is your FPS?
  4. What type of network are you playing on? What is your ping time to our servers?

We have recently added additional metrics to the game, we are analyzing that data now, and soon you’ll be able to see some of the above info yourselves. Sending that info with your tickets will help us diagnose the issue.

Why, when we jump, do we lose the ability to see tags?

This is a known issue with our current implementation of alliances, where tags take forever to load in. (It’s related to my data synchronization rant from another question). This will be eliminated or minimized with our new alliances implementation, coming in a few weeks.

Wrapping up

Thanks for the awesome questions everyone. I LOVE that so many people are genuinely curious about the game and what we’re doing to improve it. Moving forward, I’m going to write a post to you guys on a regular basis “straight from the engineering team” so we can describe technical changes that are coming up. I’d love to continue getting your feedback as we work to make War Commander better.

-Sobokop


* ** ***ORIGINAL POST*** ** *

Hello everyone, WrongThinker here.

I’m excited to introduce our first regular Q&A!  This Q&A features our Technical Director: Sobokop!  Here’s how it works...

  • Respond to this thread with your technical questions about the game!
  • On Friday at noon PST, we’ll compile the questions and present them to Sobokop (so make sure to respond by noon that day)!
  • We’ll update this post later that day with as many answers as Sobokop can write!

Remember to keep your questions technical (we’ll do a more design-oriented Q&A at a future time).  We look forward to your questions!

-WT

Game Design Ninja
KIXEYE
  • Pramleader
    Pramleader
    Potential Threat
    Joined Apr 2014 Posts: 77
    Shockwave crash is far to common is there any plans to address this
  • Kar-Graal
    Kar-Graal
    Unicorn Overlord
    Joined Mar 2013 Posts: 2,638
    edited 3 Aug 2017, 9:13PM
    One Question Will The Relocation be reduced to 30 Minutes to help fix server issues if im sure
    people with free relocation is not something to be [Abusing the Gameplay in Anyway] and [its opposite of fun]
    [FYI] [People Who Were Using Dropships and relocating is causing an unfair gameplay]
    image
  • HunterHamish
    HunterHamish
    Minor Nuisance
    Joined Dec 2013 Posts: 294
    Thanks for the opportunity to post our questions:

    Why does WC differ from many other online games, whereas there is no game file download to install at our side.  Considering a lot of players have powerful PCs which are more than capable to handle an equal part of the program load, therein reducing the server side load and the reduction in data transmitted and sync'd in the game?

    Could this possibly reduce latency issues faced by players?
  • SiddyLiL
    SiddyLiL
    Skilled Warrior
    Joined Jun 2016 Posts: 596
    Kar-Graal said:
    One Question Will The Relocation be reduced to 30 Minutes to help fix server issues if im sure
    people with free relocation is not something to be [Abusing the Gameplay in Anyway] and [its opposite of fun]
    I kind of agree with this I feel there should be a timer all be it a small one with jumping from sector to sector. 
  • YamYam77
    YamYam77
    Incursion Leader
    Joined Sep 2016 Posts: 1,373
    Why isn't there any testing done after maintenance before the game goes live again? Recently with every maintenance there are several 'emergency' maintenances shortly after and the dodgy RAM excuse is wearing thin. I know trusted testers was done away with due to leaks but if there were a non disclosure agreement like there are for mods and testers in other games they could look for issues before the game is live again. Also why hasn't anyone thought of doing a staged load after maintenance so there isn't an overload of players trying to log in and overwhelming the servers?
  • Ahmed Algendy
    Ahmed Algendy
    Strike-force Captain
    Joined Jan 2017 Posts: 795

    Reduce units upgrade time

  • YamYam77
    YamYam77
    Incursion Leader
    Joined Sep 2016 Posts: 1,373

    Reduce units upgrade time

    That's not a tech question. This is about game issues rather than a wish list
  • JBSAMAR
    JBSAMAR
    Minor Nuisance
    Joined May 2013 Posts: 206
    too many have high end computers & internet connect but they still get flash crash , DC, lag, troops stuck, delay troops respond. where do you think the problem is ?. is it from our side, server side, or the scripts that checks and batch the cheats ?
  • YamYam77
    YamYam77
    Incursion Leader
    Joined Sep 2016 Posts: 1,373

    What will happen when flash is no longer supported?

    guessing html5 or kix will buy the rights to flash and keep it going. They won't admit if it's the 2nd though as that's bad business showing your hand too quickly
  • YamYam77
    YamYam77
    Incursion Leader
    Joined Sep 2016 Posts: 1,373
    Sorry if of topic but I am not happy which is apparently normal. After maintanance I attacked a shadow ops 45 base, when I went back in for second attack the base was already reset but still showing 52min until reset. I had a couple hrs repair and wasted missles. Now I am just supposed to suck it up!!!! Also the bases have become way to hard for people who don't own all the Omega units. I just attacked a Corpus 55 base and to take out 5.5 Phallanx I lost 39 lvl-11 liberators, 2 lvl-11 Vindicators, 2 lvl-10 punishers all workshopped and an Ardra. Come on kixeye give me a break here. I don't have all the toys and your making these **** basic prize bases way to difficult. Losing a ton of stuff and 10 - 20 hrs of repair is not worth it.
    yup that's off topic but if you're on facebook look up siddy3382. Loads of videos by some of the top names in the game all in 1 place
  • Micaiah Stevens
    Micaiah Stevens
    Unicorn Overlord
    Joined Aug 2015 Posts: 2,578
    edited 3 Aug 2017, 9:56PM

    Sobokop thanks for taking the time to share with us your technical knowledge.

    *I know this is a Browser based game, BUT this scenario below, is real and very bad.

    I play WarCommander on Google Chrome.  If I run into a problem with the game, Support says Clear your Cheche and that should fix your problem.

    UNFORTUNATELY for me I use Chrome for about 16 hours a day, YAH working online.  If i clear my Cache I lose ALL my internet data, passwords.  *YES I can clear certain data, BUT it destroys the Browser no matter how little of the data you remove.

    TRUE story I cleared my Cache and lost my entire War Commander account, as it was a trial account. *First month playing, MAN I hope that is not a thing anymore*

    Is there any way you can consider not making clearing the cache the technical way to solve a data problem on our end, so we don't have to clear our entire browser cache every time you guys make a mistake?* Maybe an in-game button to clear the local 'game' cache.   Even a Warcommander website button to clear the folders you guys use would be a better workaround if the Browser and local data support these features.

    It stops Support in their tracks and they act like it’s MY fault I won’t clear my cache

    *Not saying the mistake is yours, BUT it’s your data.

  • robbo40
    robbo40
    Skilled Warrior
    Joined Feb 2013 Posts: 384
    continues buffering when trying to enter workshop , tech center , platoons , warfactory , barricks , and so forth. getting red box errors , what steps are being planned to remove these lags inbase. please note i am using a MSI top grade gaming lap top and high cable networks so i belief this issue is with kix and not my lap top or internet speed
  • Maximus_Destructicus
    Maximus_Destructicus
    Unicorn Overlord
    Joined Dec 2011 Posts: 3,894

    What will happen when flash is no longer supported?


    that's my same question.  Its an important question, hope you can address it, do we have any plans?

  • jackfx4u
    jackfx4u
    Minor Nuisance
    Joined Aug 2012 Posts: 165
    When can we see a considerable increase in game performance? Give us a timeline please as I for one am about to pack it in after 5years.
  • Maximus_Destructicus
    Maximus_Destructicus
    Unicorn Overlord
    Joined Dec 2011 Posts: 3,894
    Is it really true that you guys don't know how to update the mines and mine factory?

  • YamYam77
    YamYam77
    Incursion Leader
    Joined Sep 2016 Posts: 1,373
    Is it really true that you guys don't know how to update the mines and mine factory?
    yup. This has been brought up time and again. Basically coders write in different ways, the guy who did the coding for the mine factory did it in such a bizarre way that nobody can figure it out. The only way it could be sorted would be a rewrite of the code which means deleting the whole thing then adding it as new content.....can you imagine the **** that will cause in terms of possible code clashing and the fact a lot of us have put cash into upgrades and we would lose the benefit as either we'd all have to start from scratch or everyone would just have 1 level....either way us 'veterans' lose out
  • kixeyeuser_5018480
    kixeyeuser_5018480
    Potential Threat
    Joined Jan 2016 Posts: 76
    YamYam77 said:
    Sorry if of topic but I am not happy which is apparently normal. After maintanance I attacked a shadow ops 45 base, when I went back in for second attack the base was already reset but still showing 52min until reset. I had a couple hrs repair and wasted missles. Now I am just supposed to suck it up!!!! Also the bases have become way to hard for people who don't own all the Omega units. I just attacked a Corpus 55 base and to take out 5.5 Phallanx I lost 39 lvl-11 liberators, 2 lvl-11 Vindicators, 2 lvl-10 punishers all workshopped and an Ardra. Come on kixeye give me a break here. I don't have all the toys and your making these **** basic prize bases way to difficult. Losing a ton of stuff and 10 - 20 hrs of repair is not worth it.
    yup that's off topic but if you're on facebook look up siddy3382. Loads of videos by some of the top names in the game all in 1 place
    Thx YamYam, took a look on UTube after someone in clan mentioned there was a video with zero repair for a Corpus 55. Going to have to make it a practice before attacking any base tho most of them are Hi lvl players with stuff I don't have or my toon cap not as big .... yet :)
  • kixeyeuser_5018480
    kixeyeuser_5018480
    Potential Threat
    Joined Jan 2016 Posts: 76
    sorry, meant free repair hehehe.

  • Mark4112
    Mark4112
    Potential Threat
    Joined Jun 2013 Posts: 47

    @Comm.Soviet-Union said:
    When you attack bases with lots of structures and/or units, lag always occurs making it difficult to really show your strategy maneuvering your units. What are the plans to permanently solve this issue? We've been told several things for the past 2 years, yet it seems that when something is improved, more units are introduced and we go back to square 1.
    Thank you for your time and attention.

    I agree with soviet. In the events when we just have convoy bases and you only need to kill the units and no structure's the game loading and victory pop ups seem a lot quicker than other bases.

  • Alok r
    Alok r
    Skilled Warrior
    Joined Sep 2016 Posts: 559

    As someone asked before, does some structures in specific cause or increase the lag increases in the game. For example, I do believe and have experienced when I attack bases with lot of trophies in it, it's way too laggy than other bases without trophies.
    Now most of the trophies coming out for last few months are not static ( I mean they have some kind of animation to them). I'm sure this needs more memory utilization.
    My question is if you guys know that trophies do actually make the lag worse, will you be open to simply removing trophies from the game?

  • astroturf1
    astroturf1
    Minor Nuisance
    Joined Jun 2017 Posts: 122
    Workshop tech spamming in your base after or during attack when can we expect this issue to be resolved
    I say how most players feel i defy kixeye because they always overstep thier boundaries
  • Maximus_Destructicus
    Maximus_Destructicus
    Unicorn Overlord
    Joined Dec 2011 Posts: 3,894
    what would it mean to the game to get rid of out-dated features, such as oil and metal deposits, and old un-used units such as elite laser tank?  would it help create more fluid game-play and decrease the number of disconnects, or not really?

Sign In or Register to comment.