The unlikely story of Teardown Multiplayer

(blog.voxagon.se)

122 points | by lairv 3 days ago

10 comments

  • jmgao 1 hour ago
    > For the longest time (and for good reasons), floating point operations were considered unsafe for deterministic purposes. That is still true to some extent, but the picture is more nuanced than that. I have since learned a lot about floating point determinism, and these days I know it is mostly safe if you know how to navigate around the pitfalls.

    If you're only concerned about identical binaries on x86, it's not too bad because AMD and Intel tend to have intentionally identical implementations of most floating point operations, with the exception of a few of the approximate reciprocal SSE instructions (rcpps, rsqrtps, etc). Modern x86 instructions tend to have their exact results strictly defined to avoid this kind of inconsistency: https://software.intel.com/en-us/articles/reference-implemen...

    If you want this to work across ARM and x86 (or even multiple ARM vendors), you are screwed, and need to restrict yourself to using only the basic arithmetic operations and reimplement everything else yourself.

    • turtledragonfly 11 minutes ago
      I'm sure you know, but for others reading: even on the same architecture, there is more to floating point determinism than just running the same "x = a + b" code on each system. There's also the state of the FPU (eg: rounding modes) that can affect results.

      On older versions of DirectX (maybe even in some modern Windows APIs?) there were cases where it would internally change the FPU mode, causing chaos for callers trying to use floats deterministically[1].

      [1] https://gafferongames.com/post/floating_point_determinism/ (see the Elijah quote, especially)

  • kajkojednojajko 2 hours ago
    I enjoyed playing Teardown when it first came out. It was already a technological marvel back then, so it's even more impressive that they managed to make the simulation deterministic to add multiplayer several years after release. Clearly top tier engineers.

    Cool to see that the game is owned by Coffee Stain now, too. Satisfactory has been handled well by them, so I'm optimistic about the future of Teardown as well.

    • VoidWhisperer 44 minutes ago
      It seems like Coffee Stain generally handles the games they publish pretty well. They also have Valheim and Deep Rock Galactic published under them, both of which have been reasonably successful and long running games
  • HexDecOctBin 1 hour ago
    If someone from Teardown dev team is here, did you guys ever tried to do the physics in voxel space? If I understand correctly, Teardown convert each physics chunk into a mesh and feeds all the meshes into a traditional physics engine. But this means that the voxel models remain constrained to the voxel grid only locally but not globally.

    I have been trying to figure out a way to do physics completely in voxel space to ensure a global grid. But I have not been able to find any theory of Newtonian Mechanics that would work in discretised space (Movable Cellular Automata was the closest). I wonder if anyone in the Teardown dev team tried to solve this problem?

    • DecoPerson 1 hour ago
      (I’m not a Teardown dev!)

      I tried this on a local project. It looks very jank and the math falls apart quickly. Unfortunately, using a fixed axis-aligned grid for rotating reference frames is not practical.

      One to thing I wanted to try but didn’t, was to use dynamic axes. So once an entity is created (that is, a group of voxels not attached to the world grid), it gets its own grid that can rotate relative to the world grid. The challenge would be collision detection between two unaligned grids of voxels. Converting the group to a mesh, like Teardown does, would probably be the easiest and most effective way, unless you want to invent some new game-physics math!

  • bjord 4 hours ago
    I highly respect the implicit decision to forgo repeat purchases by merging into the original game, considering how much work was clearly involved. I haven't played it, but I hope for sustainability's sake, there are sufficient (purely cosmetic) microtransactions to cover their development costs.
    • prox 3 hours ago
      He is already talking about the new engine at the end, so maybe that means a new game/version.

      Anyway I recently bought it because of multiplayer. Can’t wait to try it out.

  • tietjens 3 hours ago
    In my opinion one of the most impressive independent games published on Steam in the last years.
  • esperent 3 hours ago
    Looks like someone needs a better web host.

    > Due to protection of web servers from repeated attacks, we were forced to restrict access to administrative interface of web pages to selected countries. If you are currently in a foreign country, please sign in to WebAdmin, proceed to your domain management and disable this GeoIP filter in OneClick Installer section.

  • Sweepi 3 hours ago
    "1.Serialize the entire scene, compress the data, and pass it to the joining client. We already do full scene serialization for quicksave and quickload, so this is possible, but the files are large: 30-50 MB is common, often more, so transfer would take a while.

    [...]

    3. Record the deterministic command stream, pass it to the joining client, and have that client apply all changes to the loaded scene before joining the game. The amount of data is much smaller than in option 2 since we’re not sending any voxel data, but applying the changes can take a while since it involves a lot computation.

    Once we started investigating option 3 we realized it was actually less data than we anticipated, but we still limit the buffer size and disable join-in-progress when it fills up. This allows late joins up to a certain amount of scene changes, beyond which applying the commands would simply take an unreasonably long time. "

    So [1] is not an option for players who want to do it that way?

    • florbo 3 hours ago
      fme, it's only kind of inconvenient. By the time the scene gets to the point where join-in-progress is disabled it's complete chaos anyway. Might as well restart the scene.

      That said I haven't played any of the more intricate mods out there, but I can how it would become more of an issue.

  • DimitriBouriez 2 hours ago
    Wouldn't it have been simpler (even if technically heavy) to host the game on a single machine and just stream each player's camera? That way all the physics would be computed in real time on one computer, and each player would just receive a different video stream.
    • dwattttt 2 hours ago
      Video streams are not known for their low bandwidth needs, let alone adding in RTT latency for inputs.
      • DimitriBouriez 2 hours ago
        That's true, I'm not saying it comes without trade-offs. But in return you get a perfectly consistent and physically accurate simulation. It would mostly be expensive, I think, but it's technically feasible (services like Shadow or GeForce Now already demonstrate that).
        • filcuk 2 hours ago
          Which one of your friends can host an mp physics heavy game with a number of low-latency high-resolution video streams? I would estimate the average answer to be zero.
        • kg 41 minutes ago
          Most consumer GPUs have a limit on the number of video streams their hardware encoder can handle at once, and in some cases the limit is as low as 2.
  • xecaz 3 hours ago
    Looking forward to play the MP version with my son who also loves the game. I worry that the game will get heavier and maybe no longer work of the steamdeck, will it?
  • stainlu 1 hour ago
    [dead]