Implementation

Colyseus consists of approximately 8,000 lines of non-whitespace C++ code, with an additional 9,500 lines for Mercury and its networking library. The implementation of Mercury we use does not yet perform automated load balancing so each node is currently assigned an equally sized range from each hub. Hence nodes handling more popular portions of each hub tend to be more loaded with routing traffic.

We modified the original Quake II FPS game to run as a distributed game over Colyseus. We represented an object's area-of-interest with a bounding box, which we precomputed as the visible volume for different locations in a map. Publications and subscriptions are routed along a single Mercury hub using the $y$-coordinate, since we found that it was almost always the most selective in our particular workload. However, publications are matched to subscriptions using the intersection of all three $x$, $y$, and $z$ attributes. Mobile objects (such as players and missiles) use predictive publications to publish to a small region around their current position and re-publish when they leave that region; this allows re-publication to occur once every few seconds. Similarly, an object submits a new subscription whenever its last subscription does not contain the currently visible bounding box. Non-mobile objects (such as items) publish every 30 seconds since their locations do not change. Although we emperically validated that the particular values we chose allow for a reasonable re-publication and re-subscription rate (i.e., once every few seconds rather than hundredths of a second), we believe that the object manager could estimate these values for individual objects based on history to achieve a more percise tradeoff between consistency and communication overhead. This is one topic of future research.

These additions are accounted for by approximately 3000 lines of code, including a liberal amount of debugging code (but excluding serialization code, which we generated automatically). This is in contrast to the original codebase's approximately 190,000 lines. The majority of the additions deal with interfacing Quake 2's data structures with Colyseus' adaptor interfaces and computing, saving, loading, and checking bounding boxes. Moreover, except in two special cases⁹, all our additions were inserted in the top-level of the main event loop without changing any of the internal game logic.

We also note that Colyseus is not yet optimized in terms of communication cost (for example, ACKs are not piggybacked and Mercury routing attributes are represented in a string format). Moreover, because we automatically delta-encode and serialize Quake 2 objects without hand-tuning, the average object delta size in our implementation is 145 bytes as opposed to the more selective average 22-byte delta that Quake 2 sends to clients. We believe hand-optimization could yield a similar selective delta size since many fields in Quake 2 objects are scratch variables, can be derived from others, or are simply not needed in replicas. Nonetheless, the numbers we report are measured values and not estimated optimal sizes. Different games will have different delta sizes, and we expect that delta sizes will grow as client bandwidth increases to facilitate experiences with higher fidelities.

Despite these limitations, during many of our experiments, we verified that real clients can connect to our distributed servers and play the game with a lag similar to that obtained with a centralized server.