Master servers considered harmful
This case study explores the aspects of video game master servers and highlights their dangerous pitfalls.
What is master server?
Multiplayer video games usually follow client-server model where server program simulates the state of the game world and transmits it to clients while clients render the world and sends input to the server. Client program runs on the player’s computer while server usually runs 24/7 in a datacenter. However, the client will only be able to access the server if the client knows the address of the server. That’s where master server comes in. Master server is a special server that keeps the addresses of other servers. It has the well known address that may be hardcoded in the client and server programs. When server program starts, it usually looks up the address of the master server and tries to connect to it to be listed. When the client program starts, it usually connects to the master server and gets the list of all servers. As you may observe, master server becomes the single point of failure.
Let’s see what the consequences of this model are based on 2 games: Team Fortress 2 and Red Eclipse.
TF2 is a proprietary video game by Valve released in 2007. The game relies on Steam DRM and uses it for the master server. It is a multiplayer first person shooter where 2 teams battle against each other. The main gimmick of the game is that the player is required to choose the class ranging from offensive ones such as Scout or Soldier to support ones such as Medic or Sniper. All classes have different stats and weapons and work in a rock-paper-scissors way.
The game used the model of previous Valve titles were Valve relied on the community to host its servers. The EULA allows you to download the TF2 dedicated server free of charge and run it without serious restrictions. The server also has semi-official way to load plugins at runtime. While there was no official support for plugins, the community embraced the opportunity and started work on many of them. There is now several popular plugins which can do a lot of things and most of them are OK with Valve as long as they don’t remove the DRM.
When it was released, the game didn’t offer any progression and the only thing player could earn was achievements. So it was easy to modify all aspects of the game. In particular, I’ve found that it is possible to use console to spawn any weapon and pick it up as any class. This allowed to play the game in a new way and I quickly made a SourceMod plugin which allowed people to buy weapons that don’t belong to their class. Then in 2008 Valve announced the Gold Rush update which promised new weapons and so I thought that they would be just like any other and prepared the GUI of my plugin for them.
However, when update was released, I’ve found that new weapons share the same classname with old ones and it was not possible to spawn them. I have used the
GiveNamedItem function from the vanilla Source engine which exists in any Source game. By disassembling server binary I have found that a new function was added which took a pointer to
CScriptCreatedItem but reverse engineering further was far beyond my skills. So I was not able to spawn new weapons.
Now, there are a few very important points to note. Before the update, all content of the game was available at the start. But new weapons had to be unlocked by earning achievements which were non-trivial. This had a side effect of people starting to host “achievement servers”, servers where players didn’t play the game but were helping each other earn achievements. This was the first sign of the downfall.
After that, a few other updates were released which added more unlockable weapons and spawned more achievement servers. However, after some time, a peculiar update was released. Valve was frustrated that people farm achievements and wanted to combat this. So they designed a new system which awarded player unlockables based only on played time. This resulted in a very negative reaction from players and Valve reversed the decision for weapons. However, a new type of unlockable was added - hats. They didn’t have any gameplay value and the only way to earn them was playing a lot. Every once a while the game would roll the dice and if the player was lucky, it would award the hat. The consequence? Idle servers - when players start their client, join the server and keep the game running indefinitely. The winning move is not to play.
After some time a person with the alias DrunkenF00l created a program which would fake the player playing on a server, saving the need to actually run the game. He ran a special server which had a very large player limit and it was possible to see thousands of people “playing” on it. This resulted in Valve announcing that it would remove all items earned by idling and give players who didn’t idle a halo. Since the game used Steam spyware, it was easy for Valve to see what the players were doing. This was dubbed a “Halocoust” and again resulted in a very negative reaction from player base.
And somewhere near that time, there was a breakthrough in SourceMod community. Someone reverse engineered
CScriptCreatedItem. This made possible to give items which were not available before, such as unlockable weapons and hats. Quickly, a TF2Items extension was released which provided a simple API to give any items to anyone playing on the server with it. Valve’s response? Adding a special DRM in client which would make a request to Steam server to see if the player actually earned the item they were wearing and, if they didn’t, not rendering it. So even from the server’s point of view, the items were legit, client-side they weren’t rendered. Of course, “NoSteam” servers didn’t have this problem.
This was a strong indication that Valve is obsessed with their total control and would not give anyone freedom. Their ultimate goal was revealed shortly. Valve sponsored the contest held at Polycount which promised winners that their items will be included in the game. The result? “MannConomy update” - selling items for real money. Not only Valve were now selling items, their prices were extremely high, a single kit from Polycount was more expensive than the game itself.
The most insulting thing were crates that are earned the same way as hats - just by playing a lot. However, to unlock the crate you need a key and the key is sold for real money. And crates were the only source of “unusual” items which were very highly priced between players.
And this whole chain of events had a single cause - Steam. Only by having any step to be approved by Steam it was possible to make this Orwellian scenario possible. The direct solution? Don’t use Steam.
RE is a free/libre/open source video game developed by Quinton Reeves, Lee Salzman et al. The game code is licensed under zlib license and assets use various CC licenses (but all of them are free culture compatible). It is also a multiplayer first person shooter. It used arena style gameplay similar to Quake 3. The main gimmick of the game is parkour tricks which allow you to travel fast and reach otherwise unreachable areas.
The game also uses clients-server model, however, most of the calculations happen on the client and server acts merely as a proxy to transfer data. Upon request to the master server, it responds with a script which will be interpreted on the client. This means anyone who connects to any master server is vulnerable to remote code execution.
By default, the game used master server hosted at play.redeclipse.net. When you first start the game and select “play online” you are presented with multiplayer guidelines. The 2 points are important:
- If the source code of the server is modified substantially you must contact the Red Eclipse Team to check that the changes are permitted.
- The server must honour the auth system, allowing global bans and grant the correct access by the Red Eclipse Team, or moderators assigned by them.
The first point basically prohibits freedoms 1 and 3 of Free Software definition. The second point is more interesting. The game uses so-called “global auth system” were there are players with global accounts which have various auth levels. When you run your own server, you can give out local auth levels such as Supporter, Moderator, Operator and Administrator. There are people out there which have global auth levels such as these. In essence, you are forced to give out the control of your server to random people on the Internet. In practice, if you are using official master server, you are giving a control of your server to trolls who blindly worship the main developer.
Here the global administrator Acerspyro tries to troll Xonotic IRC channel on Freenode.
You can remove all control from levels up to Administrator but you can’t deal with global Administrators such as the one you see on that screenshot. To put insult on injury, there are 2 levels higher than Administrator: Developer and Founder. They have more control of your server than you are. I have contacted Richard Stallman to discuss this and his response is “this is going too far”.
If you remove the global auth from your server and connect to play.redeclipse.net, your server will be banned. It has occurred before. One server owner made every kill called First blood and his server was banned. Apparently, making any kill first blood is “substantial modification”.
Last point, when you use global auth and connect to any server, a connection to master server happens to prove your account level. This allows owners of master servers to spy on anyone with account.
You can connect to another master server, but since the community is small, it means losing almost all players. Promoting a new master server can be compared to promoting a separate game. It is a very hard goal.
All in all, using any master server in Red Eclipse is dangerous. Forking the project and fixing the mentioned issues is the only way to go.
We have observed that master servers are a single point of failure and people who run them get unjust control over their users. In fact, any system with a central authority has this weakness. So we need to remove this central authority. We need a peer-to-peer system. Since there are not a lot of people who play libre games, implementing such system with single game will not produce a robust network, the system should be able to handle many games at once with developers of each game running their server and contributing to the whole network. The good effect of this is that small dev teams without budget for servers will be able to use the network to promote their game.
There are a lot of technical details to be ironed out and the work on this system has begun. Expect more information in the future.