This week’s post has been brought to you by Dave!
We’ve all been there: You’ve just built your first ship and ascended gloriously from the shipyard. You’ve got the wind in your hair, you can’t wait to reach the next island, and then BAM! Your ship has mysteriously jolted forward or backward, throwing you off. There’s a reason World’s Adrift veterans stay grappled to their ships at all times…
Well you’ll be happy to know that for the past month I’ve been working on fixing this situation, that we call ship rubberbanding. There were a number of issues causing it and I believe I’ve fixed almost all of them, but there is at least one remaining so don’t forget to stay grappled just yet.
So, a little bit of background about how the ships work and why:
Worlds Adrift is built on technology called SpatialOS which distributes simulation of the world across multiple servers. Each of these servers is responsible for simulating physics for each entity inside a particular area of the world and then sending players’ the position of that entity. The problem with this is that simply sending positions to players suffers from latency.
‘Latency’ is the amount of time it takes for a message to reach a player’s game after the server sends it. It’s not uncommon for latency to be as high as 100 milliseconds for players living further from the game servers. For a ship travelling at 20 meters per second, this means that a ship would appear to be 2 meters behind where it actually is on the server.
Now, for many of you, your favourite pastime is ship fights, where ships’ positions on your client is paramount to your piloting and cannoning tactics. It would be pretty annoying if all positions were actually a couple meters wrong! To compensate for this, we actually send the position where the player will be in the future. This means that most of the time the player will receive this message before they need it and therefore can position the ship in the exact position where it is on the server right now.
Determining where a ship is going to be in the future is achieved through a process called ‘extrapolation’. We take the ship’s current position and take into account its current speed and direction of travel to estimate where it will be in half a second. This method has its own set of problems, particularly when turning or colliding with other objects, but we think the tradeoff is worth it.
In fact, an attempt to solve one of these problems is what was causing the enormous jerks that ships would display when inside the weather walls. Extrapolation in fact introduces a small error in the ships path when turning due to the fact that the ships velocity is changing.
In this image, the green line an exact path taken by a ship and the red line is the extrapolated path.
In order to reduce this error, we were previously taking the ships acceleration into account. However, inside weather walls the ships acceleration is constantly changing as it gets blasted by the wind. This small change in the acceleration resulted in huge changes in the extrapolated path. We no longer take acceleration into account so we will have errors in the ships path now, but in practise it’s not really a problem because we send updates for the ships position so regularly that the error will never be very big.
Additionally, we were also extrapolating the rotation of ships, which is why ship collisions would produce such crazy results! Weather walls would also cause the ships angular velocity to change rapidly and therefore cause large rotation jerks. During battle ships don’t tend to rotate very quickly, therefore it was unnecessary to extrapolate their rotation, so this behaviour is now also improved.
The next big group of issues that I found was determining at what point in the ‘timeline’ to synchronise with the stream of positions coming from the server. This kind of thing is really hard to get your head around, especially when considering that in the most complicated case you have at least four different ‘timelines’ to reason about; player one, player two, server one, server two. This is exacerbated by the fact that each of these timelines runs at a variable rate!
In Worlds Adrift, when a server’s framerate starts to slow down we actually begin to slow down the passage of time itself! The way computer simulations are run is in discrete chunks of time called ‘ticks’, so when it takes longer for the server to simulate one tick than the amount of time that tick represents, you are now effectively running in slow motion.
In order to have a player’s computer coordinate with a server that is running on a completely different timeline we need to somehow synchronise the timelines. Originally we were using our own algorithm to synchronise each player with the server they were on, but it turns out synchronising time accurately over a network with highly variable latency is not a trivial problem! So in order to rule out any mistakes that we made we completely replaced it with a third party solution. That’s not to say there aren’t any, but at least it was created by experts in the field.
There is however a problem with using a timeline that runs in ‘realtime’ and is not pegged to the simulation – if the player’s machine experiences a framerate hiccup their physics will suddenly jump forwards, meaning that the ship they were standing on would disappear from under their feet! In cases like this we just glue the player to the ship and teleport the ship forwards.
There are still some outstanding issues that remain – transitioning between two physics servers still sometimes causes a rapid acceleration. I believe this is due to synchronisation issues between the servers which we want to address in the future, but for now we have used a workaround – when switching to a new server, the ship will slow to a stop for a second then resume course. While this isn’t ideal, it certainly beats getting thrown off your ship in the middle of nowhere…
The other issue occurs when there is another physics object colliding with your ship while you cross the boundary. When an object is not being controlled by a server, but it still knows about it, it’s physics is set to ‘kinematic’. This means that the object isn’t really simulated, it’s just receiving position and rotation data from another server. The problem with this is that it’s as if the object is unmovable. Therefore if any other object (like a ship) collides with it then that object will be forced to move to stop collding with it. In the case of ships crossing server boundaries with a Manta attached, or a broken panel hanging around, this can have drastic consequences. I’ve seen ships literally flip upside down due to this issue. Unfortunately this problem is very deeply rooted in the way the game works so we will probably only revisit this system as part of a major server refactor which is ongoing for at least the next year. I have however attempted to mitigate this issue by reducing the rate at which ships can ‘depenetrate’ other physics objects for five seconds after they switch server.
So there it is, 4 weeks of my job condensed down to around 1000 words! My work on the ships essentially amounted to a complete rewrite of the systems that govern the transmission of ship movement. Overall it’s going to be much, much easier to work with the system as a whole now, so hopefully we can iron out the remaining issues after getting some data from the new public test servers.
Caption: A graphing tool I developed to help discover exactly what was going on under the hood.