"Run an OSRS LAN," they said. "How hard can it be?" they said.
Ahead of the Deadman All Stars live finale, Mod Hamster was handed a deceptively simple challenge: run OSRS entirely from inside a theatre in Chicago. In this Spudworks, we'll peek behind the (literal) curtain to explore the technicalities of delivering OSRS-in-a-box.
Why LAN?
LAN stands for Local Area Network. Running an "OSRS LAN" means operating a game world, its supporting infrastructure, and the competitor game clients all in the same room and all on the same local network.
Running the finale locally wasn't convenient (using the live servers would have been much simpler), but a ticketed, live competitive event has a specific set of requirements:
- Extremely low and consistent latency for a level playing field
- Complete independence from internet connectivity, should the venue internet fail to materialise or be unreliable
- Confidence that no external service outage could interrupt play
Historically, DDoS attacks have been a real bugbear for this kind of event, so that last bullet point was a major one for us to address.
The answer: put OSRS in a box. One that was hosted on-site but an approach with its own challenges, as you'll see...
Microservices, Microservices Everywhere
Unlike Ironmen, a live OSRS game world doesn't exist in isolation. Behind every login, membership subscription, Grand Exchange trade, hiscore, bond redemption and friends list is a web of supporting services and databases. Altogether there are more than 200 back-end microservices involved.
The system in its entirety is both too big and too complex to reasonably reproduce on a single physical server to be supported by a small set of J-Mods. So, to make the problem tractable, we distilled our requirements down to a minimum viable set of services:
- rs2engine - the OSRS game server
- authserver - the authorization (and in the absence of Jagex accounts, authentication) server
- loginserver - the confusingly misnomered service responsible for character saves
But even this minimal set presented a problem: authserver and loginserver had their own dependencies, pulling in a tree of other microservices and databases.
Simplicity was our watchword, so we swapped authserver out for fakeauth, a mocked (test software which imitates production software in limited ways) stand-in which we normally use for test purposes during development.
While it solved our dependency tree problem, fakeauth wasn't a perfect fit. In truth, it was inappropriate in several ways:
- It doesn't actually check usernames / passwords - the username is accepted as is in combination with any password.
- It doesn't support clean display names - entering Mod Hamster would result in an in-game name of mod_hamster.
- The permissions / entitlements for characters are set at fakeauth launch time - we needed concurrent support for competitors with standard world permissions and referees, spectators, and J-Mods.
Clearly, fakeauth needed some work so in the run up to the event we added support for password checking, configurable mapping of character name to display name, and a simple "character role" authorisation scheme.
Selecting The Hardware
With our essential services identified, we set about selecting the hardware for our local server. With such a minimal set of software components, we didn't need much. But, there were a few items on our shopping list:
- Error correction at CPU and Memory level - Reduce the risk of hardware glitches or pesky cosmic rays crashing our server during the event
- 4-6 core x86_64 CPU - We can run rs2engine on aarch64, but not all micro-services have been validated
- >= 16GiB RAM
- Enterprise warranty with rapid onsite support
- Compact / easy to transport
We went for a Dell PowerEdge T160 Server, Intel® Xeon® 6 6315P, 16GiB ECC memory, 2TB mixed use SSD. Basic, but robust with a good on-site warranty repair service.
Selecting The Operating System
Linux of course! Are you kidding me?
Ubuntu 24.04 LTS specifically.
Don't Leak The Game Server
Shipping a box around containing everything someone would need to run legit OSRS game worlds is a bit scary. To protect our intellectual property, we use password-based full disk encryption (LUKS) with a physical hardware security key (Yubikey) to store and input the decryption key password.
While the server was shipped everywhere by courier, the Yubikey was carried separately by a J-Mod.
Importing Character Saves
In the run up to the finale, the teams were building their characters up on a live game world. Behind the scenes, we store character saves as loosely structured data keyed by unique character hashes in a NoSQL database, which is better for handling this kind of data at scale. Specifically, AWS DynamoDB.
Obviously, we can't bring the real Amazon Web Services on-site with us, but we could follow the same pattern as with fakeauth by using local development software. Amazon provides DynamoDB Local for development, making it a natural fit for our offline environment.
Prior to the finale, we exported the DMM All Stars table items as JSON and imported it into our local server.
RuneLite
In addition to all of the Jagex components being capable of working entirely offline, we also wanted to ensure that the competitors' chosen client could function should the venue internet be down or unreliable.
RuneLite is much more dependent on an internet connection than you may initially assume. Each time you launch RuneLite, you are invoking the RuneLite Launcher; a wrapper which talks to runelite.net to download the latest client files. Installing/validating plugins is also internet dependent, something which would happen a lot at our event as competitors were sharing PCs and switching RuneLite profiles between each match.
To address this, the RuneLite maintainers gave their generous support to help us mirror everything we would need on-site and host it via an Nginx Docker container.
DNS Hijacking
Both Jagex's own components and RuneLite have hard coded assumptions about the DNS names / domains where services can be found.
To support these assumptions, we configured Dnsmasq on our OSRS-in-a-box server to be the DNS resolver for our competitor PCs. This allows us to direct any and all relevant lookups to our server.
address=/jagex.com/172.16.32.1
address=/runescape.com/172.16.32.1
address=/static.runelite.net/172.16.32.2
address=/repo.runelite.net/172.16.32.2
For jagex.com and runescape.com, Jagex controls the domains and bringing valid TLS certificates with us was straightforward.
For runelite.net, we looked at adding our own private certificate authority to the appropriate trust stores on the competitor machines. However, as Jagex were using a third-party to provide and configure these machines and we control both a trusted network and DNS, we opted instead to simply disable certificate validation in RuneLite. This was acceptable because the LAN was physically controlled, isolated from the public internet, and used only for the event.
Remote Access
The final piece of the puzzle was providing a mechanism for the wider team of J-Mods back in the UK to provide technical support (internet allowing) and ship hot fixes to the game content if needed.
To support this, we added our server to AWS Systems Manager (SSM) as a Hybrid Managed Node. This allowed remote management without a VPN, including having our regular build pipelines deploy hotfixes directly to the local server through a mixture of AWS S3 storage for artefacts and SSM Documents to drive deployment.
The Final(e) Architecture
Putting everything together, the OSRS-in-a-box deployment looked like this:
It'll Be Alright On The Night
The best-laid plans of hamsters and men often go awry, and so it was when our plans made contact with reality. On the day, unexpected gotchas and fixes included:
- RuneLite profile limit capped at 20: Our initial plan was to have the profiles of all competitors pre-loaded onto all PCs, but as the astute among you may have noticed 30 > 20. The fix here was to ensure that team members 1 through 5 only sat at a pre-assigned PC (1 through 5) on stage left or stage right, thus capping the permutations at 10 profiles per PC.
- Version mismatches: The RuneLite version, rs2engine version, and game content version are tightly coupled. Between freezing the versions of these components before shipping the server and the day of the event, we J-Mods had inadvertently introduced content and engine drift which broke RuneLite compatibility. Thankfully, the RuneLite maintainers came in clutch with a freshly minted RuneLite build which would work for our specific setup. GG RuneLite team!
While we had some techincal hurdles and hiccups, overall we're really happy with our first live US event and we hope you enjoyed it too!
Let us know if you enjoyed this instalment of Spudworks and if there are other topics you would like to see covered in future.
You can also discuss this update on the
2007Scape subreddit,
the Steam forums,
or the official OSRS Discord.
For more information, check the official Old School Wiki.
Mods Adad, Bash, Boko, Cky, Haydon, Ibex, Kraken, Maniac, M0iqp, Qwert, Galgo, Titus, Vallcore, Vxp ... and ????
