Nexon Wrote:
Hello again Maplers!

Let’s talk about server maintenances. Server maintenances, whether scheduled or unscheduled, happen fairly regularly. I think therefore, it’s safe to assume that most Maplers know what “server maintenance” means. The game is unavailable and can’t be logged into. But have you ever wondered what we do during a server maintenance and why they’re necessary? If so, read on and I’ll explain!

Step One – Preparation

Server maintenance is an “all hands on deck” situation. It requires at least a server engineer, the developers, at least one producer, QA testers, and occasionally a database administrator. It also requires intense preparation and planning to keep the maintenance as short as possible.

First, the server engineer and the developers list out all the tasks that must be performed and create an internal schedule. Next, the producer takes that schedule, adds time for testing, and adds in some extra time in case there’s an emergency. Then the producer creates a “maintenance post” that gets put up on the official Website to let the players know a maintenance is coming.

While this is going on, the developer is delivering all of the new files and a list of changes to the server engineer and the producer. The server engineer must check the files to ensure they contain no errors. Then they’re loaded onto our internal testing environment. That’s where the QA testers come in. They do some basic testing to ensure that the changes cause no serious issues. Once they’re satisfied, they give the server engineer the green light. This all happens in the two days before the maintenance itself.

Step Two – Bring it Down

When the maintenance begins, the first thing that happens is we block the login port. This keeps anyone but us from logging into the game. Then we shut down the login servers. These are the servers that maintain the players’ connection to the game servers. Once these are down, all players are removed from the game though the game worlds are still running. Finally the game servers themselves are shut down. This takes over an hour because each game server has to run a number of “synchronization checks” to make sure the game database has an accurate record of the current state of all players.

These synchronization checks are actually a large part of the maintenance period and one of the major reasons they’re necessary. If the game servers aren’t shut down on occasion to run these checks, they can become unstable. The last servers brought down are the databases. This only happens if we have to work on them though. If there’s an unscheduled server maintenance, we try to not bring down the database servers if we can help it. This helps keeps the maintenance as short as possible.

[video=youtube;xjdh4F_sL2s]http://www.youtube.com/watch?v=xjdh4F_sL2s&feature=player_embedded[/video]

Relax, the Mechanic is not involved in server maintenences.

Step Three – Update

Once the servers are down, we start updating the server code. Server code changes include minor bug fixes, add some sort of internal data tracking tools, or adjust in-game content. This is actually the quickest part of maintenance, usually taking only about five minutes.

The real time-consuming changes come when we have to make changes to the database. Even routine database work like world transfers and name changes can take about an hour and a half on average, depending on the number of requests. Sometimes though, more complicated changes are needed. Since the databases contain all the information for every character that exists in the game – every character. That character you haven’t logged into for two years? It’s in there. That Cygnus Knight you played to level 30 and never touched again? It’s in there too. That’s a lot of ones and zeros to dig through, so any change takes some time. It gets worse if a query for a database change fails and must be started over. That can add hours to the downtime.

While the servers are down, we also usually update the Cash Shop. Updating the Cash Shop doesn’t actually require a server maintenance as it’s located on a separate set of servers. That allows us to update and maintain the Cash Shop without affecting the game servers if needed, but we try to schedule Cash Shop maintenances around server maintenances.

Step Four – Bring it Back Up

Once all the changes have been made, we can restart the servers. Bringing servers back online happens in the reverse order from bringing them down. First we bring up the databases, then the game servers, then the login servers. Bringing servers back up takes about an hour and is usually the easiest part of the maintenance. We rarely encounter errors or issues during this step.

Step Five – Testing

Once all the servers are back up and running, we leave the login port blocked and enter the game to test it. We log into every channel in every world, test the Cash Shop and Maple Trading System, create and delete characters, and test new account creation. These tests are to ensure that nothing abnormal has happened to the servers. This testing takes about thirty minutes. Once it’s completed we can open the game to players.

This is often where delays happen even though we schedule twice as long as we need for the testing itself. That’s because we have hundreds of servers and sometimes one or more fails to restart properly. Sometimes there are network connection issues. This step allows us to see those problems before players experience them. If and when these very serious issues occur, we can bring the servers back down to fix it. Of course this causes the maintenance to be extended, so it’s the one thing we never want to see during this step.

Step Six – Back to adventure!

Once the maintenance is over and the game is stable, the developers and server engineer go home to sleep. Once a server maintenance begins, no one working on it leaves until it’s complete, so some of us will have been up for over 24 hours. In the mean time, the producer and QA testers monitor the game, read the forums, and hang out in populated areas of the game to make sure players are not experiencing any problems. Assuming everything went smoothly, the maintenance is all done – and then the rest of us can go home to sleep!

I hope you enjoyed this little peek into what we do during server maintenance. Quite frankly, they’re not my favorite part of my job (I know you don’t like them much either), but they’re the unglamorous necessity that keeps MapleStory running. Hopefully now they’re less of a mystery.

Until next time Maplers,

Eurydice

The MapleStory Team

Nexon Wrote:Step Five – Testing

Once all the servers are back up and running, we leave the login port blocked and enter the game to test it. We log into every channel in every world, test the Cash Shop and Maple Trading System, create and delete characters, and test new account creation. These tests are to ensure that nothing abnormal has happened to the servers. This testing takes about thirty minutes. Once it’s completed we can open the game to players.

Kimihan Wrote:How about those times when CS was broken DC'ing people, and it wasn't fixed in Scheduled SC ? HMMMM

SwordStaker Wrote:Maybe it wasn't broken for them or broken for everyone.

Kimihan Wrote:I find hard to believe that from ALL those testers to go in all chs that none of them actually dc'ed from it. Unless it was only one person going from channel to channel and world to world, which would be just dumb.

Kimihan Wrote:I find hard to believe that from ALL those testers to go in all chs that none of them actually dc'ed from it. Unless it was only one person going from channel to channel and world to world, which would be just dumb.

Kimihan Wrote:I find hard to believe that from ALL those testers to go in all chs that none of them actually dc'ed from it. Unless it was only one person going from channel to channel and world to world, which would be just dumb.

Netto Wrote:Don't forget that there were several thousand people that never experienced the CS glitch. Not only that, but the Cash shop has numerous files and finding the two files that didn't sync up wasn't exactly an easy glitch to find when there are a lot of connections.

Razmos Wrote:Step 7: Emergency Maintenance!

Kimihan Wrote:I know that, I am one that didn't DC in Cash Shop (only when my computer was lame). But I find really hard to believe that NO TESTER actually DC'ed in CS. That being said, the number that were affected was higher than the ones not affected. I believe that at least ONE of them should have that.

And just to point out, I know this is NOT necessary, but I believe that they should get one guy in each map *I know, this sounds ridiculous now* to test the maps, just to not happening that thing in Malaysia going to Vikerolas, that you had to change the graphics settings.

Nexon Wrote:That Cygnus Knight you played to level 30 and never touched again? It’s in there too.

SwordStaker Wrote:Thing is what if that doesn't happen to them? What if they can pass through just fine and thats it. NEXON has said before even after numerous testing some bugs do not appear until the game goes live.

Kimihan Wrote:Then if it appears AFTER the game goes live, someone changed the encoding. Just turning the switch ON does not make hundreds of thousands of people have a bad-encoding on CS. A bad encoding does. I am not a freaking professional on how-to create programs and such, but even I know that's not the way it works.

I'm glad they explained, just wondering why the hell happened.

Netto Wrote:Actually, that's incorrect. Let's switch from Maplestory to Southperry for a minute. I am a content tester for this website and from the things we've tested, we've ironed out all the bugs we could find. HOWEVER, the day of the upgrade, several bugs surfaced that we did not experience during the testing period.

Just saying, things DO go wrong after things go live, without changing codes.