RuneNation
— users online
RuneNation
Join our Discord for weekly events & prizes!
Take a look behind the scenes with our infrastructure team!
Given the number of potato-related comments we’ve seen recently, we’ve no doubt that you’re all aware of this year’s maintenance roster. We thought it’d be a good idea to give you more insight into the what, why, and how of infrastructure maintenance at Jagex.
Since this is all about infrastructure, there’ll be a bit of tech-jargon to deal with, but we’ll do our best to keep things simple and straightforward. Let’s get started!
Infrastructure maintenance is the (we reckon) unfairly maligned cousin of Game updates. It refers to the upkeep, patching, optimisation, and scaling of the technical foundations that power the worlds of Gielinor!
Infrastructure maintenance includes activities like:
There’s a Dan Simmons quote about entropy which we shan’t repeat here – but the gist of it is that all systems eventually decay, no matter how well built they are. Just like roadworks, we need ongoing maintenance to keep our infrastructure functional. Our work keeps our games stable, fast, and secure, and ideally it does all that without you lot even noticing.
We’d like to share some specific instances when infrastructure maintenance has benefited players in recent years.
We're sure you remember the Login Lockout situation. It's been very important to all of us that these types of situations are prevented as far as possible! Which is exactly why one of the key areas we've been focusing on is improving our backup and continuous data protection features, to help mitigate the risk of anything like that happening again.
Here are a few more examples:
We’re always thinking about the future of our infrastructure, and learning wherever we can. For example, the team working on the now-defunct Project Zanaris did some fantastic work that will help us modernise and migrate more systems into AWS, reducing the need for offline maintenance.
As previously said, ticks are king here, so we're looking to change our virtualisation platform for Game Worlds to move from a co-scheduling CPU scheduler (ESXi) to an asynchronous scheduler (Linux CFS/PreemptRT tuning under KVM). While there's a lot of technical wording here, the key takeaway is that doing this will help reduce missed game ticks under high load conditions.
Downtime is great for wrapping up chores and actually getting work done, but we know you’d rather be back to second-screening your skills. While our work is important for the long-term health of the game, we want to be open with you about some of the most common pain points we've seen when the game comes down.
You might be surprised to know that it generally doesn't. Most of our infrastructure maintenance activities are achieved without any downtime at all.
However, there are certain activities – more than we would like – which cannot be achieved while keeping our games online. In part, this is the nature of working on fantastic games with a 24-year history; some software components have been with us for a long time and are not as horizontally scalable or dynamic as we would like.
In some cases, this precludes approaches like blue-green deployments (seamless switching between two versions) or rolling upgrades (live update parts of a system bit by bit).
This doesn't mean we're complacent about downtime. We're always trying to ensure that our updates and changes impact the fewest people for the shortest amount of time.
Look, we all want Nan out of that cage as quickly as possible.
When we schedule downtime for infrastructure maintenance, rather than doing it online behind the scenes, this is often because we are working on parts of the infrastructure involved with state persistence. The main two areas for this are billing and entitlements and, most importantly for you, player saves.
When Jagex performs significant works on these systems, some of which store data in Jagex's own proprietary formats, our overriding focus is not on speed but accuracy. Most of the maintenance window is taken up validating that player saves and the game state are correct before opening the games back up to players.
Nobody likes a rollback, and part of the reason downtime takes so long is that we’re doing our best to make sure rollbacks are never required due to infrastructure work.
Regardless, for each piece of key maintenance we perform, especially when it involves downtime, the team always comes together for a retrospective review to see what we can improve on next time!
OSRS and RuneScape are fairly unique in that our players are not region locked. A consequence of this is that game state and other backend services are global constructs. When we perform disruptive maintenance to those global backend services, global downtime is required.
A lot of the region-specific maintenance we do is already transparent to players. We aim to do more things on a per-region basis where possible.
Our maintenance is normally timed between 08:00 UTC and 14:00 UTC, which matches our working hours here in the UK. We know it’s frustrating for those of you in far-flung time zones, but these hours mean we have access to the whole team to help when emergencies arise.
We have looked at other potential time windows to occasionally shift the impact between demographics, but these are areas that still need further discussion.
That’s a wrap! As always, you can keep up-to-date on upcoming maintenance windows through our socials, Player Support articles or the Game Status page.
We hope this gives you some better insight into our continued efforts to improve your experience and why it's sometimes necessary to put the world on snooze. If you enjoyed this style of newspost, then let us know! The team is really keen to share more potato peelings in future “Spudworks” posts.
We appreciate everyone for taking the time to read. May RNG be ever on your side!
Mods Kraken, Vxp, Bash, Maniac, Vallcore, Haydon, Maniac, Cky, Qwert, Drax, M0iqp, Ibex, Adad, Roman ... and ?