What's this Online Terra Mystica thing?

For the last couple of years my main hobby hacking project (over a thousand commits, and probably an order of magnitude more time spent on it than all other non-work projects combined) has been an asynchronous multiplayer web implementation of the brilliant board game Terra Mystica (Feuerland Spiele, 2012). At the moment it's roughly 2/3 Perl, 1/3 Javascript, and uses Postgres as the data storage.

It's been a fairly successful project for something that was originally intended as a one-off. The usage statistics at the end of November 2014 are:

• Almost 6000 registered users
• About 1200 monthly active users (as in playing at least one game; not passive use like looking at the statistics pages).
• 14000 moves executed on a normal weekday (10000 on weekends)
• 16500 games either ongoing or finished.
• Bi-monthly online TM tournament run by Daniel Åkerlund with 400+ players.
• 1038 commits as of this writing.

This was not supposed to be a general use program. It was originally a one night hack to help keep track of a hand-moderated play-by-forum game of TM, which was obviously headed for failure due to the massive amount of errors people were making while describing their moves in natural language or when manually tracking their resources in the game.

From there the project snowballed, slowly gathering features including just about everything I ever marked in the TODO as being 'out of scope'. Since I often had only very limited amounts of time to work on this, and my expectation was always that the interest in the site would soon fizzle out, the project management method was to always get the maximum short-term bang for the buck.

A project whose direction is literally guided by 'what can I get done in the next two hours' is of course massively path dependent; the early decisions made with very little consideration had outsized influence on where the site ended up. Sometimes the expedient gambles on 'do the simplest possible thing' failed, and the results were just rubbish. At other times things ended up at a slightly odd local maximum. And in some rare cases the gamble turned out to produce wonderful and unexpected results.

Timeline

Future posts will discuss the actual lessons learned; what didn't work and what did work - both in the mechanics of programming and in the peculiarities of online boardgames. But in this one let's just have a look at the history of the site, how long it took for it to get features that one might consider absolutely necessary, and how amazingly bad user experience people are willing to put up with when it's the only way they can play their favorite game online.

Feel free to skip past the bulleted list if you get bored, it's still a bit long even if I include only changes I consider fairly major (indeed, a lot has to get filtered out given it's 1000+ commits).

2012

• December - Early January: The smallest program that did anything useful related to a game. I'd enter moves into a text file and run the script to produce the final game state as JSON. This JSON was rendered to HTML + Canvas by some Javascript code that was half ripped off from an old project. There was some minimal rules checking and automation, and support for only 5 out of the 14 factions in the game. Users of the current site might want to see the old look.

• January: A rudimentary dynamic web site, implemented simply as a wrapper CGI script around the JSON generator script. After that a clumsy web-based editor was added for game files (a textarea that could be used to edit specific files in a git repository, no authentication except for each game having a random 160 bit identifier as part of the URL). This allowed other people to moderate their own games, as long as I created a game for them and sent the link with the secret embedded. Players would post / email a natural language description of their move to the moderator, who would then enter the moves into the admin tool using the correct syntax. Amazingly some 20 games were run using this insane system, while by all rights the project should have died there.
  
  This version of the software had automation for resolving the effect most game events, but did very little validation to notice completely invalid moves.
• February: Added an ability to easily rewind the game state back to any time in history, to help with post-game strategy analysis. Also added a way for players to enter their own moves (a textarea in the main game view, a preview button and a save button, and some verification to make sure they could only enter their own moves). Again there was no real authentication here, just links with an embedded faction token derived from the per-game secret key.
• March: The hackiest email integration in the world: Store the email addresses players in the same text file with the commands. After a player has entered a move, the software would create a mailto: link with prefilled subject, content and receivers (the other players). The player would clicks on the mailto: link, the email loads up in their mailer (even GMail), and they'd press send.
  
  Compute and display a VP projection on the last round assuming no further moves, to give players some idea of who is really winning.
• April: I continued to resist adding any user management or authentication. But my friend Gareth wanted a better way to manage his ongoing games than a spreadsheet, and wrote a small App Engine site into which players entered their secret game URLs. His site then used my site's API to figure out which games the player needed to act in. And it went even a bit further, by embedding the move entry UI into the same app.
  
  After a few weeks of using Gareth's site, I had to admit that he was totally right about this being required functionality. So I finally added a DB to the project for storing user accounts and game metadata, and a 'your games' list on the front page after login. It's also only at this point in the lifetime of the site where I added a UI for people to make new games. Until then every game was created by somebody asking for a new game via email.
  
  Finally, this month also saw the addition of a statistics page on how often each faction was winning (since balance was a hot topic on the BGG forums of Terra Mystica right from the start), and soon after a list of achieved high scores for each faction and player count.
• May: This month mostly introduced all kinds of stricter validation, as the reduced barrier to entry for playing was causing significantly more illegal moves to be entered (early on players were enthusiasts of the game and thus had good knowledge of the rules; at this point people started to learn the game through the site, which was quite scary).
  
  The main new feature of the month was the 'planner', an alternate text entry box which could be used to enter commands arbitrarily far into the future, and check that the moves are valid and what kind of effect they have. This is useful for example for checking that you have sufficient resources for making certain moves without manual computation. Another use is leaving 'notes to self', so that the player doesn't need to re-evaluate the board for every single move. (Some people were suddenly playing tens of games at a time, so this was a real problem).
• June-August: This time period saw only minor fixes and improvements from the user's point of view. There was a bit of infrastructure work behind the scenes, such as moving the actual game moves into the database, though they still remained just plaintext.
• October: The mini expansion for TM was released at the Spiel fair in Essen. I implemented the new features the very next morning in lobby of my hotel at Essen, with a ChromeBook, a ssh connection to the production server, and and the world's worst WiFi. After some reflection I decided not to make the change visible to the public before getting back home and a more reliable work environment :-)
• November: I finally made the site automatically send email notifications, rather than require players to jump through the fragile mailto: hoops to let other players know whose turn it is. Replacement of the mailto-style notification of moves also required the addition of an in-site chat feature for communication.
• December: Another consequence of the real email support from the previous month was that players no longer needed to expose their email addresses to other players. This finally made it possible to allow players to create 'public games' that anyone can join, rather than only play people with whom they've done some kind of an out-of band email address exchange. (At this point 1500+ games had been started, amazing how far such a kludgy system could go).
  
  At the time 25-30% of moves were being entered from smartphones or tablets. But the move entry interface was typing commands like 'convert 2pw to 2c. upgrade d3 to tp' into a text box. What's wrong with this picture? :-) In the month we finally got a slightly friendlier UI, though the textual command representation still remained the canonical one.
  
  The site finally got a ranking system: a multi-iteration version of the ELO algorithm, which computed not only player strengths but also faction strengths, and credited good results with the weaker factions more than good results with the strong ones.
  
  Finally, in very late December I went on a big refactoring spree to move the game from CGI scripts to a more persistent application server (FCGI with Plack and CGI::PSGI, but no framework). Eradicating all global data and all modification of literal data structures was way too much work, those were not corners worth cutting in the first place.
  
  The new UI went live a year from starting the project (almost exactly; from December 22nd 2012 to December 21st 2013), and is the point where I'd consider the site to be actually usable by mere mortals.

• February: Support for variant maps, for testing parts of the upcoming Terra Mystica expansion for the designers. I also added a map editor that could import map definitions from Lode's TM AI, which the design team had been using for the map. The online playtest team proceeded to play 100 games with different map versions before the expansion finally went to print.
• April: A bunch of work on the expansion, which was still being kept under wraps. So the support for the new final scoring types and four of the new factions was not visible to most users at this time.
  
  The main user-visible change was automatically dropping players from games after a week of inactivity, to support the inaugural season of the online Terra Mystica tournament. People's irritation about others playing slowly had been constant ever since the addition of public games (95% of my games are private with a few separate groups of friends, so I'm pretty isolated from this myself). Unfortunately this change appears did not appear to help enough.
  
  This month also saw the addition of individual profile pages, showing all kinds of statistics for each player (games started, finished, performance with given factions, performance and play counts against specific opponents, etc).
• September:The next attempt at reducing the anguish caused by slow players was to allow setting shorter move timers than the default one week (from 12 hours to 14 days). Lots of people started 12 hour deadline games, and moved on to complaining about so many people dropping out. Sometimes you just can't win.
• October:Public support for the two new expansion maps, as well as the new final scoring types.
• November:Public support for all six new factions from the expansion, as well as the variable turn order variant.

I find it interesting that it really did basically take a year of real time (and maybe 2 months of hacking time) before the implementation was in a shape where I would've thought about publishing it. And there's no way I'd put that amount of time into a project like this up front. Usually these projects are active for a couple of weekends before getting abandoned; fun parts are done but all the hard work of making it really usable remains.

In this case people were eager to use even the incredibly crude early versions, so I got over that hump very quickly. And at that point every incremental improvement to the site was affecting tens, hundreds, or thousands of people. This is of course always more motivating than working on polishing the perfect piece of software that nobody is using.

There were many architectural and design decisions done along the way that I ended up deeply regretting, and which cost me lots of time later on. But without all those early shortcuts there would've been no implementation at all. Easily the best example of Worse is Better that I've been personally involved with.