Lessons from a failed launch

13 Sep

The relaunch of ArcheAge this weekend is snafu. There are three elements contributing to most of the player anger: a disregard for global time zones, an unusual game element (land), and inconsistent communication. I’ve been inside companies when we had to rollback a disastrous launch and I’ve been an impatient user, so I think it’s interesting to look at what went wrong and the lessons to be learned for other shared virtual spaces. There are a couple parallels to the early days of Second Life as well.

a carefully planned ArcheAge prank

a carefully planned ArcheAge prank*


It helps to know a couple things about this particular MMORPG before I continue. If you want to play the game ArcheAge in English, you play a version localized from Korean by Trion Worlds. They run clusters of North American and European servers. When they do a new release or event, they base the schedule on US Pacific time and it is simultaneous on all servers. For example, the servers were scheduled to relaunch yesterday at 1:00 pm PDT. ArcheAge is unusual among MMORPGs in that it has a player-driven economy and there is an important, limited, nonrenewable, location-dependent resource that is first-come, first-served and can be held indefinitely: land.  Having a large, well-placed farm/home can make a huge difference in the profitability of trade runs, which is a main method that players earn in-game gold for gear and supplies.

Trion’s plan to correct some imbalances was to launch three classes of server after a 48 hour maintenance period. The stakes for being in the initial login rush differed considerably between them:

  1. Legacy — No change from when the servers went down on Thursday except that they were updated to the 2.0 codebase. A delayed launch inconvenienced users on these servers because they couldn’t play, but that was the extent of the impact.
  2. Fresh Start — These are new servers that only allow level 1 characters to be created: everyone has to start over. The existing servers are grossly imbalanced in favor of older players, so theoretically, these would put everyone on equal ground again. Of course, some players were twitching to find exploits and many headed for Fresh Start servers were thinking of it as a race to the level where they could claim land.
  3. Evolution — These are new servers that each contain the population from two discontinued legacy servers. I think its fair to say that these are the angriest players. If they owned land on a discontinued server and wanted land again, they had to be prepared to sign on the moment servers came up and rush to stake their claims. We were also told that namespace conflicts would be handled in order of login: if someone on the other merging server had the same name as my avatar, the first account to sign in could keep the name but the second would have to change. So, even if the land rush wasn’t an issue, some people were anxious about keeping their names. Trion had assured us that we would be given compensation packages of in-game stuff and credits for going through the merger, but they would not reveal the contents before the servers went down. (I should say that this was all a lot of meh for me. My avatars are on an Evolution server, but they don’t own land or have names I care about. Getting a compensation package is a bonus in my situation, because all I really lost was possible playing time.)

At the scheduled launch time yesterday, the authentication servers had a problem and only about 10% of login attempts were successful. Those 10% of players gleefully grabbed land on Evolution servers and created characters on Fresh Start, but then Trion noticed the issue and took all the servers offline. They provided sporadic, uninformative updates as the players’ forum posts grew angrier. Finally, after five hours, they announced that servers would remain down and launch would be attempted again on Sunday. They also made the controversial decision to rollback the servers to their state before that small percentage of people logged in. Any land they claimed was gone, any new avatars created were erased, etc.

Some learning points from this experience:

A global userbase means that time zones must be considered. There are two issues when it comes to Trion’s management of ArcheAge: accurate communication of times and reasonable accommodation. Sure, time zones are tricky, especially with the inconsistent use of Daylight Savings Time. Trion generally lists times in both PDT and GMT and has made mistakes in translating between them in the past. The times they give are untrustworthy. This is something that Second Life has addressed by making the in-world clock consistent, pegging it to the time in California, and listing all events in SLT (Second Life Time). While this puts the responsibility for translating to local time on the users’ shoulders, most of us are capable of working through the calculations, using world time clocks, or asking. That is far better than to be unreliable.

Having users in time zones from -12 UTC to +12 UTC means that simultaneous events will always be a problem. Trion initially decided to launch North American and European servers at the same time. NA users with weekend jobs took Saturday off, EU users planned for a late night, and those in Oceania set their alarms for the wee hours. Each hour that launch was delayed upset those groups more and more. The announcement that the launch was pushed back to Sunday was infuriating to players who had planned their weekend around this (or those in Oz and NZ who had to be at work or school during the new launch, on their Monday).

For the rescheduled launch today, Trion took a better approach. They decided to start the European servers at 1:00 pm GMT: first the Legacy servers, then the Evolution servers after 30 minutes, and finally the Fresh Start server.  North American servers will follow with a similar phased rollout at 1:00 pm PDT. I’m writing between the two rollouts and this seems to be working better. There are some disconnect problems and one server went down for an hour. The European Fresh Start server reached its population cap in less than 10 minutes, triggering a login queue. Trion has already adjusted the times for the North American launch based on things they learned this morning; I’ll see how that goes in a few hours. The EU thread on the forums isn’t happy.

In Second Life, global time zones present less of an obstacle because almost all events are created and run by residents; it’s a world, not a game. As I’ve traveled around SL doing research, I’ve found Russian, Japanese, and Australian areas that are hopping when I’m staring at my coffee with bleary eyes, but are ghost towns during US prime time. Those of us with friends on other continents have learned how to negotiate and shift our schedules to spend time together.

One element can unbalance the whole world. Land is a real problem in ArcheAge because of its implementation. I don’t think it can be fixed while it has the qualities I listed above, being important, limited, nonrenewable, location-dependent, first-come first-served, and able to be held indefinitely. Land is the reason that people are crushing the servers when they go live, and losing money and time associated with land seems to be the biggest complaint.

It makes me think of the early years of Second Life, when the most desirable locations were on the SL mainland near telehubs. The land there was gobbled up and resold or leased at sky-high prices. Then, free point-to-point teleporting was opened, disrupting the importance of location, and private islands began springing up and offering rentals in a more controlled environment than the mainland. Brokers and landlords can still charge premium rates for particularly nice or high-traffic locations, but nowadays, there is so much land it’s no longer a cause of imbalance. There’s another key difference: if you don’t own/rent land in SL, you are not denied access to functions of the world. It’s easy to find free places to change clothes, open boxes, spend time with others, or create. In ArcheAge there are some products that can only be made with machines built on owned land; non-landowners like myself have to buy those items from other players or do without.

In time of upheaval and crisis, communication must be proactive. Most of the information shared with ArcheAge players comes through one senior community manager and I think she tries to do her best. Yesterday, though, updates were originally every half hour, then spaced out to an hour (and was late). Then we were promised a major update “momentarily”, which turned out to be 30 minutes. The information was usually updated in a post pinned to the top of the official forums, but that post instructed us to follow ArcheAge on Twitter for the latest news. That would be great, except that the last updates weren’t posted to Twitter until much later. If you say you’re going to give an update at a certain time, it can’t be late. If you say Twitter is the best source for news, you must keep updating it.

Another communication difficulty came from Trion’s decision not to announce the contents of the compensation packages for merging players until the servers were down. I’m seeing a lot of complaints from players saying they would have done things differently if they knew what the compensation would be; holding onto property or selling it, for example. Perhaps Trion didn’t want to give players an opportunity to try to game the system, but the result was that people had to guess and gamble.

There’s also a communication problem between Trion and the game developers in Korea. It’s rather remarkable, but I’ve seen a few occasions where Trion has announced, “This new event/item will have X qualities,” and then it launches and the details turn out to be something else entirely. Event rewards were described as aesthetic-only “weapon skins” in a live stream, but they turned out to be fully-functional weapons. We were told that if there was a namespace conflict in this merger, the first user who logged in with the shared name could keep it and the second would have to choose a new name. In the attempted launch yesterday, however, they discovered that conflicts are actually being determined by the age of the character: whoever created a character with that name first gets to keep it and the newer character’s name must be changed. Too often, Trion seems clueless about the product they’re distributing and the concept of testing.

My server is scheduled to open in 65 minutes, so I’m going to take a nap and then see how it goes. Maybe companies who try to run virtual spaces — games or worlds — in the future will understand the basics that seem to elude Trion, but I’m hoping they soon begin to learn from their mistakes.

Update: My server did open, about an hour later than scheduled, and I was one of the lucky players who received all of the promised treats from Trion immediately. Horrible UI led me to make a mistake, though: one of my characters had a name conflict, and the UI showed me the image of my female character Tsofia. “Strange that she’d have a conflict,” I thought, and I typed in a new feminine name. Much to my chagrin, it turned out to be a male alt who had the name conflict, and he was then a very masculine, tough fellow with a girly name. Grr. So, I gave him a gender reassignment. Poor fellow is now a super girly blonde with pigtails.  So far, I hate how crowded the new server is, but I’ll have to see how it is in a few days.


*In the image I used to illustrate this post, the big grey guy is a world boss. His normal position is to the left on the stone floor and he’s inactive unless you get very close. Some mischievous player carefully planted a row of trees in front of his spawn location, waited a day for them to grow, then lured the boss out. The result was that he was blocked from return, remaining active and with a larger area of effect (my avatar is just beyond the limit where he could lasso her in). A second instance of the boss respawned in his normal location.  My avatar is level 51 but I have crap gear; I was startled and killed by this guy once, but I was able to escape him on two other occasions. I can’t imagine how many lowbies he took out before someone removed the trees and he reset.

Leave a comment

Posted by on September 13, 2015 in Gaming


Tags: , , , ,

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: