MUST READ - World Map Loading Slowdown

«1
FriendlyFixer
FriendlyFixer
WC Game Team
Joined Jan 2018 Posts: 80
edited 8 Jun 2018, 7:03PM
Hi!

Welcome the latest installment in Friendly Fixer's Must Read series. We haven't spoken in a while. How are you? I'm great. This is a format that I've used internally for a while now and thought I'd open it up to the community, as well. So, today I bring you an explanation on what happened to our World Map loading times. At some time on Wednesday or Thursday, transitioning to the world map started taking 45+ seconds instead of 1 or 2. 

(Since you are new to this, I'll explain the format. There is a MUST READ section that is kept to the point, followed by a MAYBE READ section where I expand/explain/brag a bit. Enjoy.)

MUST READ
  • Wednesday we deployed as usual. 
  • We discovered our throttle no longer works as intended. 
    • We could let in everyone, or no one.
  • We were experiencing greater than average load on our mysql databases.
    • We 'pulsed' the throttle to avoid crippling load.
  • As a result the deploy was 'messy'.
  • Players began complaining of world map loading times.
  • We assumed this was from the DB load
  • Fast forward to Thursday. At this point the team starts noticing that ALL our development environments were loading slowly.
After much investigation, it has been determined to be caused by a change in the Flash Player plugin, which was released on June 6th and 7th. 
Adobe put out a security update to the flash player that bumped it up a major version number, from 29 to 30. This was to mitigate two things:
  • A security flaw in Microsoft Office embedded flash content
  • Security flaws due to Spectre/Meltdown
  • As part of this work Adobe changed the 'getTimer()' function at a low level. 
    • This is a core flash API function that we use for timing data. It is the closest thing to a high resolution timer that flash has and is EXTREMELY common for people to use. 
    • It's performance cost was increased at least 100 times
MAYBE READ

We have a tight loop that iterates over all 250 thousand hex map cells and populates them based on a server defined height map. This is to place mountains and other features into each hex map cell. This naturally takes a bit to process so it is deferred over several frames.

The exact mechanism to do this is a loop that has a counter into the server sent byte array. Each iteration through the loop processes exactly one hex map cell and reads one byte from the array. It then checks how long it has been since we started the loop. If that time goes over a defined threshold, which is 25 milliseconds, it breaks out of the loop. If we are not done loading yet, the next frame tick will go back into the loop and pick up where it left off, controlled by that index counter.

The problem is that we are effectively calling getTimer() 250 thousand times. In flash player version 29 the algorithm could handle 6 thousand or more iterations per 25 millisecond time slice. In the latest flash player this dropped to 300 or less. Understandably this took a LOT longer to process world map data.

I fixed it with a bit of a bodgy hack. Instead of calling getTimer EVERY iteration, I do it only every 1000. This means we only call it 250 times and process roughly 5 thousand per 25 ms. Much better. Also, simple and fast to get out to you guys!

IN CONCLUSION

There are other places in our code where we call getTimer and I will be evaluating our use. In the meantime we will be deploying this fix later today. It is a flash client only deploy and does not require down time. It might however require you to delete your cache, or at least delete the game's SWF from your cache. This shouldn't matter since we invalidate the game's SWF in our cloud storage, but I can't control browser's caching strategies.

Until next time!

(If you want to see Adobe's comments on this: https://forums.adobe.com/thread/2499947)
  • ErwinRommel2016
    ErwinRommel2016
    Strike-force Captain
    Joined Feb 2013 Posts: 767
    Question:

    Do you really have to listen to the Pokemon theme song at your desk, on repeat, all day, every day?

    Regards,
    Goldendoodle
    Well ok then :| :p :|
                           -SPECIAL FORCES ARE MY LIFE-
  • Redeemer
    Redeemer
    Unicorn Overlord
    Joined Apr 2012 Posts: 3,704

    @Goldendoodle said:
    Question:

    Do you really have to listen to the Pokemon theme song at your desk, on repeat, all day, every day?

    Regards,
    Goldendoodle

    LOL

      WC ID 3439259
  • CM Chris
    CM Chris
    KIXEYE Community Manager
    Joined Mar 2014 Posts: 3,449
    Question:

    Do you really have to listen to the Pokemon theme song at your desk, on repeat, all day, every day?

    Regards,
    Goldendoodle
    He wants to be the very best.
    WC:RA/BP Community Manager
    Community Rules: CLICK HERE
    Submit a Support Ticket: CLICK HERE
  • Axel L
    Axel L
    WC Mod
    Joined Mar 2013 Posts: 10,847
    Thanks a lot for the information. Hopefully, we will see more of these posts in the future.

    To answer your only question in the thread, I'm fine.
     <|=  0330 1323 1103 1211 0312 1201 1302 1203 0230 1220 0200  =|> 
     <|=  0302 0320 1310 1220 1211 0200 1313 1233 1302 1210 0230  =|> 
     <|=  0200 1211 0310 1303 0321 1301 1311 1221 1312 0312 1201  =|> 
     <|=  1230 1221 1211 0230 0320 0301 1232 1203 1211 0200 1221  =|> 
     <|=  1232 0200 1201 0302 0200 1210 0320 1221 1203 0232 1310  =|> 
     <|=  0232 1221 1233 1232 1201 1302 1321 0232 0232 0332 1331  =|> 
  • NemesisDarkly
    NemesisDarkly
    Incursion Leader
    Joined Mar 2014 Posts: 1,335
    Thanks for the info.

    It really helps when you guys communicate and get this stuff out to us. Silence when theres big issues only pisses everybody off.

    I just wish we could get some updates on the many other issues and bugs plauging the game atm.
    image
  • StaimB
    StaimB
    Strike-force Captain
    Joined Nov 2013 Posts: 765
    Thx for the information. And i like it to see that you employee's from Kixeye jolly each others, too. :D 
  • CM Chris
    CM Chris
    KIXEYE Community Manager
    Joined Mar 2014 Posts: 3,449
    edited 8 Jun 2018, 11:03PM
    The fix for the World Map issue is now live. PLEASE NOTE YOU SHOULD CLEAR YOUR BROWSER CACHE SO IT WORKS CORRECTLY.
    WC:RA/BP Community Manager
    Community Rules: CLICK HERE
    Submit a Support Ticket: CLICK HERE
  • K i l o
    K i l o
    Minor Nuisance
    Joined Apr 2012 Posts: 172
    edited 9 Jun 2018, 1:46AM
    The map delay is working ok now but there are still other issues that started occurring the same time the map problem came about. Going into your base from the map can result in a please wait loading screen that never loads and it is random when it happens. If your in full screen at then time it will go to normal screen after about 20 sec with the loading screen which indicates something crashed.. This also can occurred when starting an attack and once again you get the loading screen that never ends. In all cases you must refresh and I also notices that once refreshed from attack crash with the loading screen that if you selected the join battle header you will be in a loading screed again that does not end.  In this case I had to refresh and wait until the attack battle time out before doing anything.    

    It looks like the timing change in flash is going to cause many problem in this game.

    Update: The issues I'm having is with the Chrome browser 64 on the Kixeye site, ran the game in FB  gameroom and had no issues, reloaded in chrome and I get the random loading screens again.   
    LV 47
  • NemesisDarkly
    NemesisDarkly
    Incursion Leader
    Joined Mar 2014 Posts: 1,335
    Not sure if this is just the update to flash for the browser im using but since i updated not a few hours ago the whole kixeye site keeps crashing on me.
    Loaded my other browser chrome (and fgot to turn auto update off,doh! lol) and its fine but i daren't restart it now lol

    Ima install an older version to test but thought id see if anyone else has experienced the same?
    image
  • nightwolf_2
    nightwolf_2
    Greenhorn
    Joined May 2013 Posts: 1

    You know this game is not like it was 4 year's ago the loading of the game and the loading of the map loading of abase it are takes so long ever time yall do something yall **** it up more and more I will not put anymore money on this game it's getting that I might find a new game to play that the game load is faster and PvP is better

  • Dakota74
    Dakota74
    Potential Threat
    Joined Aug 2013 Posts: 49
    When you get this fixed. I'll be back.  I just spend 20 min out of 25 minutes just reloading this game.
  • Micaiah Stevens
    Micaiah Stevens
    Unicorn Overlord
    Joined Aug 2015 Posts: 2,578
    edited 9 Jun 2018, 3:14AM
    Thanks Friendly Fixer; I really appreciate the format and the depth.

    At SOME point, you could tackle why it takes 10 minutes (ok ok 1 minute) to process our rewards, YAH another Purifier unlcok.
    Probably not low hanging fruit, but the biggest slowdown.

    SINCE I am here, OMG thank you THANK you, for the 10 second before each invasion wave, shaved off 2 minutes, I know INSANE of my time. Since I was in the game is loading state for 30 seconds on each wave, and that adds up quickly! (Thereabouts)

    I went from ~#1k to ~#200, that is INSANE.

    I know it may seem like a cheat; but really it was starting the timer, and not even going.  I HOPE you can put that on other bases.
    FOr instance the Omega SS base, will drop 30s to a minute worth of Units, from the SS before I can even start.

    Amazing insane changes!  *Just seems the game is now taking longer and longer to give me JUNK.  really it does give me cool tech when I earn em from Shadow Ops, but takes a minute; not per tech, but per checking of the rewards.  I noticed it was per sets, as the invasion prizes even at 6 tech was taking a minute, than processing all the 6 tech.  Maybe do a tech reward list screen, somewhere instead of popups.  THAT way you can post, Earned 1 Purifier xx Component, and go look instead of a popup hell!
  • Dune Rader
    Dune Rader
    Strike-force Captain
    Joined Dec 2012 Posts: 801
    Interesting ..... Very interesting ..... 
    Respect, Courage and Wisdom
  • Deadly17
    Deadly17
    Minor Nuisance
    Joined May 2014 Posts: 132
    CM Chris said:
    The fix for the World Map issue is now live. PLEASE NOTE YOU SHOULD CLEAR YOUR BROWSER CACHE SO IT WORKS CORRECTLY.
    I cleaned my BROWSER CACHE, I Still have the problem, So much for the FIX

  • Sasuke970
    Sasuke970
    Incursion Leader
    Joined Apr 2018 Posts: 1,165
    Hi!

    Welcome the latest installment in Friendly Fixer's Must Read series. We haven't spoken in a while. How are you? I'm great. This is a format that I've used internally for a while now and thought I'd open it up to the community, as well. So, today I bring you an explanation on what happened to our World Map loading times. At some time on Wednesday or Thursday, transitioning to the world map started taking 45+ seconds instead of 1 or 2. 

    (Since you are new to this, I'll explain the format. There is a MUST READ section that is kept to the point, followed by a MAYBE READ section where I expand/explain/brag a bit. Enjoy.)

    MUST READ
    • Wednesday we deployed as usual. 
    • We discovered our throttle no longer works as intended. 
      • We could let in everyone, or no one.
    • We were experiencing greater than average load on our mysql databases.
      • We 'pulsed' the throttle to avoid crippling load.
    • As a result the deploy was 'messy'.
    • Players began complaining of world map loading times.
    • We assumed this was from the DB load
    • Fast forward to Thursday. At this point the team starts noticing that ALL our development environments were loading slowly.
    After much investigation, it has been determined to be caused by a change in the Flash Player plugin, which was released on June 6th and 7th. 
    Adobe put out a security update to the flash player that bumped it up a major version number, from 29 to 30. This was to mitigate two things:
    • A security flaw in Microsoft Office embedded flash content
    • Security flaws due to Spectre/Meltdown
    • As part of this work Adobe changed the 'getTimer()' function at a low level. 
      • This is a core flash API function that we use for timing data. It is the closest thing to a high resolution timer that flash has and is EXTREMELY common for people to use. 
      • It's performance cost was increased at least 100 times
    MAYBE READ

    We have a tight loop that iterates over all 250 thousand hex map cells and populates them based on a server defined height map. This is to place mountains and other features into each hex map cell. This naturally takes a bit to process so it is deferred over several frames.

    The exact mechanism to do this is a loop that has a counter into the server sent byte array. Each iteration through the loop processes exactly one hex map cell and reads one byte from the array. It then checks how long it has been since we started the loop. If that time goes over a defined threshold, which is 25 milliseconds, it breaks out of the loop. If we are not done loading yet, the next frame tick will go back into the loop and pick up where it left off, controlled by that index counter.

    The problem is that we are effectively calling getTimer() 250 thousand times. In flash player version 29 the algorithm could handle 6 thousand or more iterations per 25 millisecond time slice. In the latest flash player this dropped to 300 or less. Understandably this took a LOT longer to process world map data.

    I fixed it with a bit of a bodgy hack. Instead of calling getTimer EVERY iteration, I do it only every 1000. This means we only call it 250 times and process roughly 5 thousand per 25 ms. Much better. Also, simple and fast to get out to you guys!

    IN CONCLUSION

    There are other places in our code where we call getTimer and I will be evaluating our use. In the meantime we will be deploying this fix later today. It is a flash client only deploy and does not require down time. It might however require you to delete your cache, or at least delete the game's SWF from your cache. This shouldn't matter since we invalidate the game's SWF in our cloud storage, but I can't control browser's caching strategies.

    Until next time!

    (If you want to see Adobe's comments on this: https://forums.adobe.com/thread/2499947)
    Wow,that is some issues.I appreciate that you are trying to fix the issues.And prepare yourselves as people who don't understand a singe word you're saying become keyboard warriors on this post.I want to see more of your post in the future,they are very interesting...
  • DOUBLE DOG
    DOUBLE DOG
    Unicorn Overlord
    Joined Oct 2013 Posts: 4,422
    Well i didnt really have any major issues with the world map loading problem.But this morning tried to do wave 7 of the invasion and went to loading screen and was stuck there for a couple of hours.I tried refreshing half a dozen times,rebooted my comp and modem and router.Same thing (stuck on loading)Finally just gave up on trying to get back into the game.After a couple hours tried again and was able to enter game.What caused this?
    What has happened to this once great game.
  • Rob de R
    Rob de R
    Master Tactician
    Joined Sep 2012 Posts: 2,289
    edited 9 Jun 2018, 2:25PM
    one thing i stil don`t understand...
    • We discovered our throttle no longer works as intended. 
      • We could let in everyone, or no one.
    • We were experiencing greater than average load on our mysql databases.
      • We 'pulsed' the throttle to avoid crippling load.
    Did kixeye reduced the server capacity? as the number players dropping every day, so you expect better performance instead of less 
    meanwhile there was stated a time ago that kixeye upgraded hardware and updated software and still its runs sh!te or are you guys again behind?
    But besides all bugs are solved i wonder when you guys really get rid of all those cheatprograms... cause i hate to farm and to coin while others just pay for a hack to get it done..... and we report and report and nothing is done. 
    And try to inprove support pls as so many tickets stays answered. Unanswered tickets doesn`t make kix really trustworthly, does it? 


    You thought the imaginary wasn`t possible?
    You thought Scrooge was bad? 
    think again 
    lvl 46 now (april 2018)... lets bet .. how many years to 47? 
  • john fred
    john fred
    Greenhorn
    Joined Jan 2018 Posts: 12
    Kixeye has been telling us  for years that we were the  prob with map,  they been telling us that map is the way it is because we have so many toons on the map.  With this change to  game,  that has been taking care of because of imfimi thing.  IT HAS NOT HELPED THE MAP,  ITS NOT MY COMP,  STUCK IN ATTACK,  LAG ON MAP,  LOADING TIME OF GAME IS STILL BAD,  BP LOADS  SO FAST I WAS THINKING,  WHY IS THIS NOT LOADING.  WHY IS WC NOT LOADING THAT FAST
  • Perry299
    Perry299
    Unicorn Overlord
    Joined May 2013 Posts: 4,789
    glad it got fixed running better
    "Don't cry, Don't whine, it's a strategic game so either you GIT GUD or you Go Home"
  • GreenDust
    GreenDust
    Strike-force Captain
    Joined Apr 2016 Posts: 602

    a great fix but can this low timing synchronization effect toons functions on map?

    ฬ๏lŦรקคςк

  • Kingsman007
    Kingsman007
    Greenhorn
    Joined Aug 2015 Posts: 2
    CM Chris said:
    Question:

    Do you really have to listen to the Pokemon theme song at your desk, on repeat, all day, every day?

    Regards,
    Goldendoodle
    He wants to be the very best.
    Wold map is not reliable anymore , any new ideas ?
  • BadBoyJP
    BadBoyJP
    Potential Threat
    Joined Jul 2012 Posts: 57
    Its still not working  right  come on guys fix this please  32 min to load  gezz 

  • tamadre
    tamadre
    Potential Threat
    Joined Mar 2016 Posts: 72
    edited 10 Jun 2018, 8:30PM
    TANTA PALABRERIA PARA DECIRNOS QUE CADA DIA ANDAMOS PEOR ,AHORA ATACAN Y SIGUES EN EL MAPA,LLEGO ENEMIGO A MI SECTOR ATACABA MI PELOTON Y YO SEGUIA EN EL MAPA VIENDO COMO MI PELOTON ERA ATACADO SIN PODER ENTRAR A EL NI QUITARLO
    CADA DIA ANDAMOS PEOR YA ESTO NO ES UN JUEGO,ES UN DOLOR DE CABEZA

  • Z e u s
    Z e u s
    Minor Nuisance
    Joined Aug 2017 Posts: 177
    edited 11 Jun 2018, 3:10AM
    Thanks FriendlyFixer  , i have an idea which is used by many games which is building a gui is based on any free browser with a built in flash version that suits the game code in best way , i think  this will solve 90% of players complains or to recommend an old version of such a browser which gamers keep just for the game
    01000001 01101110 01101110 01101001 01101000 01101001 01101100 01100001 01110100 01101001 01101111 01101110
  • Steve72036
    Steve72036
    Potential Threat
    Joined Mar 2016 Posts: 47
    The world map has been loading slower for several months not just since Thursday and it never loaded in 1 or 2 seconds.
  • Ali.Abbas46
    Ali.Abbas46
    Potential Threat
    Joined May 2013 Posts: 59
    edited 11 Jun 2018, 6:24PM
    Question:

    Do you really have to listen to the Pokemon theme song at your desk, on repeat, all day, every day?

    Regards,
    Goldendoodle
    i hate you lol now i have the pokemon theme song in my head :P 
  • Tony English
    Tony English
    Minor Nuisance
    Joined Apr 2013 Posts: 134
    It irritated me because the game was loading fine. Before playing the game I always clear my browser cache, flash etc, and run my antivirus anti malware programmes so that I've done everything I can for the game to work. Very rarely does the game crash for me anymore. Anyway, everything is working fine now. If anything, it may even be ever so slightly faster
  • richard.wadzinskijr
    richard.wadzinskijr
    Minor Nuisance
    Joined May 2013 Posts: 175
    REALLY SECONDS?????   it is more like 30 minutes, 45 minutes ,1 to 2 hrs, get a grip, ya all screwed the pooch again making the game unplayable, terrible lag, continued disconnects and not able to get back in, also have new errors i've not seen !
  • KILLALLL
    KILLALLL
    Greenhorn
    Joined Sep 2014 Posts: 22
    try fix all that crap because i got 5 days allomost not do anyting in game if a coin i need suport from kixeye no ? so thanks 

    DON NUNO
«1
This discussion has been closed.