Beneficial Distractions

OK, so I have a legit excuse for my posting absence this time. It turns out the force directed graph (FDG) engine I have been developing to power the graphics of HTP has drawn some interest from the data visualization market. So myself and couple friends are spinning up a start-up company around some of this tech. It's exciting stuff, but has also torn my away from working on the gameplay side of HTP for a couple months now.

Warning: This is going to be a fairly technical post.

On the positive side, the FDG engine has been improved by an order of magnitude. It's gone from being naively simulated on the CPU, to simulated on the CPU using a Barnes-Hut octree solution with constraint relaxation used to bring nodes together as opposed to the previous spring solution (which was much more prone to 'exploding' during low frame rate conditions). Finally I just migrated the whole simulation to the GPU using a compute shader and reverted to the brute force approach, which, it turns out, works fantastically with massively multi-threaded capability of a GPU. In short, I can simulated ~100 fold more nodes than before:

So that's one of the changes to the core tech. There are 2 significant others, and they were also partially born out of this start-up endeavor. First, till recently, all game data that was procedurally generated was just stored in memory in a whole pile of different specialized classes, leaving me with this significant challenge of figuring out how to serialize it all to disk without losing any state. Then having to deserialize all that data and reinitialize any unsaved data to just the right state for things to carry on as expected. Hard given the scale of the simulation going on in HTP.

Then it kind of re-dawned on me, something I had discounted early on, because it didn't feel 'right' for a game. Just use a big ass NoSQL database. It's scheme-less document nature made it perfect for storing a shit-ton of different kinds of entities without spending a bunch of time developing a strict schema that would be rigid and resistant to future changes. It initially felt wrong to have a full blown database server running locally behind a game. But HTP has grown in scope to such an extent that it's not really your run-of-the-mill game any more.

The benefits of this change is huge, and it was driven by the need for a FDG engine agnostic to the dataset it is presenting. First of all, saving and loading are completely free. I simply don't have to worry about serialization or deserialization at all.

Everything in the world is an instance of the Entity class, which is supported by a backing document (in this case a MongoDB document). The class overloads the index operator [], and all data access for each entity is done by indexing it like a string object dictionary (plus a few convenience properties for things like name, type, etc). All these entities are kept in game memory, any time a change is made to any entity via the index operator, it is marked as 'dirty' and added to a set of dirty entities. A background thread which maintains a connection to the backing database pulls from this queue of dirty entities and updates/inserts them into the database. This results in a constant persistent version of the entire game state which is only ever a second or so old.

Gameplay wise, this is also interesting, because it completely removes the concept of saving and loading games, which is an aspect I've always been keen on for HTP. I want player actions to always carry weight, much like a rouge-like, though that's not quite what I'm trying to make with HTP. The idea that the game world and everything you do in it is a permanent action can be quite compelling.

Secondary benefits to this shift to a database backed game engine are equally significant. From a developers perspective, it allows me to create tools (which I've already done) that connect to the same local database and have a constant live perspective into everything about a running game's state. A potent tool for debugging and accelerating development. Secondly, and this is a big one, it could potentially (with a lot of design consideration) lead to free multiplayer, of a potentially MMO scale if designed carefully. With all game clients reading and writing to the same database.

So that's the first of the two major engine changes that have been recently, the second is equally significant and was again born out of the need for a dataset agnostic data visualization engine. It involves removing all the logic for the creation and updating of entities and their relationships from the core engine and moving them into a scripting engine, in this case, one of my favorites: Python (well, IronPython with its C# driver, but it's basically the same).

This separation of concerns allows the engine to remain lean and clean and delegate all of the game logic out into high level scripts, which can even be hot swapped at runtime, allowing for extremely rapid turn around times on code changes. It also means the final game will be extraordinary moddable, which could potentially be modded into entirely different games with no input from me. Part of the programmer in me desperately wants to open source this thing, but the other part is screaming "YOU'RE 2 MONTHS BEHIND ON YOUR RENT!", so closed it shall remain against my best wishes.

All this scriping is still a work in progress, but I've created a basic finite state machine (FSM) framework in Python that allows entities (so far I'm only really simulating people and mobile devices,  as this has been effectively a complete game rewrite) to inherit from base types, but override certain behaviors. For example, a CEO goes to bed at night just like anyone else, and heads to work like anyone else, but their behaviors at work are very much different than say, a programmer, or a janitor. So their behavior script can override the AtWork state accordingly. These scripts make it very easy to add new complex behaviors to entities, especially given the scheme-less nature of the backing database, as any script can introduce new data to an entity with easy. For instance, when I added mobile devices, it was just a few lines of script code to give them batteries that drain and are charged depending when their user plugs them in. Yes this is a potentially pointless piece of gameplay logic, but the easier it is to add all these small details, hopefully the easier it will be for players to discover emergent gameplay, which has always been my goal with HTP.

So for the TL;DR types: Development was sidetracked with good reason, which benefited the game engine significantly, and will hopefully continue to. Gameplay logic is being totally rewritten in Python script executed by a leaner, cleaner core game engine. Hopefully rapid development will ensue baring other work commitments.

As always: