Stockfighter , the first starfighter game for programming, is currently in closed beta testing. We're moving into a dark launch (for clients and select friends of vict^h^h^h) next week. We will launch publicly immediately after the servers give us the marking that people will not burn in the fire. I would expect 3 weeks from now, and you can imagine the engineering schedules being done.
(Starfighter is a company trying to solve the problem with engineers. Puzzles where to play think minecraft meets lego meets world of warcraft and cs class - we offer people a fun little place where you can play by programming with specific pre-set goals and then see what happens. On our website or here if you're curious.)
I wanted to talk about what we've created and why, as out of general interest, likewise because i'm aware that some people dream of clearly understanding how to prepare. I'll take the liberty of not talking about levels because they're a lot of fun to unlock, but i'll focus on the setting, building and interaction, and design wishes.
Picking the fiction / setting
We chose "wall street cops and robbers" as the setting of our first game - making money through legal means (such as charging for liquidity) and illegal means (such as stealing). While my co-founders thomas and erin have done a lot of work in their careers to ensure the safety of real stock exchanges, i don't have much experience in finance. We are also primarily interested in general engineering skills, but not, for example. Quantitative operation specifically. So why start with finance?
Basically, the rotation of the player through a number of wall street institutions gave us carte blanche to parse almost any engineering goal into a more or less coherent narrative. Forex strategies deal with low-level coding, networks (all seven layers of the osi hello model), api, end-user user interface, databases, embedded systems, enterprise web frameworks, advanced research in software languages, big data, state management, etc. Etc. Etc.
Fiction also provides us with a reason to be naughty, which is what people think absolutely all programmers are attracted to. Many ctfs (slots that practice security skills) portray the player as a hacker hacking something that works up to a certain point. You have the opportunity to bring about a similar contest in the official fashion of wall street by making things often, but not strictly breaking them. (Although, ahem, not what you build in our levels is strictly speaking legal. That's also part of the fun. The coding community is also helpful for our florists in terms of motivating players. Others learn what the big institutions of wall street do for a living, and come to a finer understanding of their position in society.)
There are natural anchors in the universe as well as for competitions among participants (which are our confectioners will not be produced at the time of release, but soon), and for direct work with computers. Controlled opponents. Finally, wall street is one big computer-controlled adversary.
Some people have been left impressed with everything that every game is nothing more than a game. Couldn't we have done, for example, an rpg or a strategy game or something like that? Yes, definitely, but there were some social concerns.
Cards on the table: before doing business, i ran a world of warcraft raiding guild. (There are fewer dragons and more loot in the cases.) My co-founders, on the other hand, feel about games in much the same way as i do about ballet: it's cool that there is a partner in reality and i'm glad that people find pleasure in the forum, and such a move not for me. This position is shared by many talented engineers and would-be engineers.
As of 2015, we are not opening a connection with the technology industry out of obvious enthusiasm for ballet. It is important for the russian founders that the industry is not closed down by demonstrative enthusiasm for qua-gaming games or associated cultural geek signs, such as, for example. Fantasy or sci-fi setting.
Starfighter never wanted to build gates (gates close, that's what they're for). We're blowing up the gate.
Tech trees
Stockfighter is split into tech trees.The technology tree is a series of tasks united by the development of storytelling and pedagogy. Learn a new concept; pass the level; let's take a look at advanced levels where you can build on what you've built, even what you've learned. (It's a fun game interior challenge how players don't have to return you to an empty editor between each level without making it feel like the user has already done it.”)
From an architectural standpoint, a tech tree is usually need additional internal engineering work on top of our shared portal as well as other external engineering work to get it playable through a browser window. (More on the process later.)
At launch, we import two tech trees: automated trading, and emulators and compilers. My co-founder erin has already discussed building our emulator, so for the time being, i'll only talk about the trading tree.
Automated trading
Automated trading is a classic. The problem people think of when they hear wall street. Relevant: flash boys (a book about high-frequency trading) hit the spectrum of new york times bestsellers. He exhibits in himself the fantasy of "making a quasi-million dollars using only his wits". ("Fantasy" in this sense is an art term referring to the creation of games: strongly as if someone were telling a story or script, the game designer wishes to have a deliberate regulation of the player's emotional state. When you want to familiarize yourself with this, it is vital "fun theory » raf koster.)
In stockfighter's automated trading tree, each level provides you with a comfortable scenario (randomly generated and assigned to a tiny copy of the world available only to the customer and a plausible set of goals. You achieve the goals by controlling our rest api using any language/stack/tools you are comfortable with.
What is open to support trading:
- A limit order book written in go - such an organization with a rest api interface written in go - a settlement system written in go - a simulation of the world for a universe of fictitious companies written in ruby- several automated traders written in ruby - master server game p that exposes a rest api written in go- an internal message bus, more on that later- a blotter interface (similar to a retail trader's web app - think what i see when i log in to etrade"), built with react and using the exchange api and the gamemaster
Most of this was written by me as thomas and erin were doing the technically ambitious parts of stockfighter. No, actually, against the backdrop of their work, building a stock exchange from scratch is actually pretty easy.
Writing a stock exchange
Since 2010, my main stack was ruby on rails. I love rails and we use it for a human user web application, but rails is not the best choice for sculpting a stock exchange. The main reason was concerns about raw results.
Stockfighter does not engage in "high-frequency trading" as wall street understands the term, although it supports (and you do) the fact that some non-professionals understand the meaning of hft.
Difference in degree. Many non-specialists are extremely inclined to believe that any ordering algorithm is hft. There are a lot of prop-shops - it is better to say hybrid technology / financial companies that use the organization's funds for algorithmic trading - that have received a time scale of the order of days, but not of the order of nanoseconds. Their individual claims, and the logic behind them) are more active than a person can reasonably follow, but this is still not hft. (Acid test: if you never worry about the speed of light between your systems, you never do hft. Amateur trading from london on the chicago mercantile exchange while using ms-freaking-excel to place trades? Not hft guys.)
We abstract a bit from the notion of time, but it was clear that each exchange had to be able to handle thousands of orders in a normal second.
That would be… difficult in rails . In my experience with rails, low complexity operations that require logging into the database typically take around 300ms. (Better, more advanced commands like the guys at basecamp on the list, in classic cases have reached 50~100ms.) We could do thousands of these operations per second, and, apart from a typical web application, we are: - heavy, non-cached workload by nature and b) have some serious limitations in the direction of requests affecting one another.
Why? Well, this is the stock market. The order book - a list of how many shares at what prices are now realistic to buy and sell - is (conceptually) a loner behind a global lock.(Each stockfighter exchange will simulate thousands of exchanges at the same time, but from an architectural point of view, it needed to first consider the use case “thousands of people go to one exchange at the same time.”) Which facilitates interaction between the player and the bot. The matching mechanism must be fully executed on a particular order before the next one can be processed.
We have solved this in a similar way to a real stock exchange: the order book is completely preserved. In memory, in go.
Stockfighter is my first production system written in go. The google performance wizards (and a few others.) Absolutely did the trick. The language screams, and luckily a mere mortal gets to put a web server with it. (Pay close attention to the server, not the app. More on that later.) Rest api or our internal message bus are passed to 1 goroutine that manages entry and exit to the complex. Book. This goroutine writes the results of these requests to a series of tapes. The tape is, physically, a buffered channel with a maintenance routine. The tapes are then consumed by other parts of the system, eg. Send execution emails to players/bots, update quotes displayed via api/ui, etc.
Our exchange is incredibly fast: we can process fairly complex orders similar to a full glass of orders, in the order of 100-200 nanoseconds. Read operations are even faster. (Maybe over time i'll come up with some way to change the exchange architecture so that reads don't touch the matching mechanism at all, but right now a lot of them do. Oops.)
This intelligently, this gives users considerable freedom to experiment with trading strategies (including those that push the boundaries of performance), gives us the ability to run dozens or hundreds of bots per player, and reduces our infrastructure costs to "less than a small fortune." (We are taking out the trash on ec2. By release hour, while waiting to understand how many visitors there are and what their typical behavior is, i assume we will have a huge number of t2.Micro instances for the stock exchanges. )
Making it work was a bit monstrous. Our order book is represented internally in the form of a red-black tree (go implementation via gollrb). I have never used a red-black tree since i got my cs degree. (Okay, they're in mysql or postgres somewhere, but i've never had to think about them before). For pedagogical purposes, modeling the complexity of the stock market is a real engineering exercise compared to, for example, a typical crud application. . This is the basis of many trading strategies. In case you're going to speed up golf loops to build your code faster, this will a) be of direct benefit to you in the later levels of stockfighter and b) provide a fun way to play games with algorithms and data structures.
A quick example of using llrb for an order book data structure: they are great at carrying arbitrary account and takeaway, sorting, and allow you to do things like "iterate pin up casino over every price point on the tree from the cheap cost side and give me everything for now, while you're counting so many shares and doing it fast.”
My main problem in this part of the development was concurrency control. Had i learned the wisdom of the common go pattern "create a server object with a single control routine and loop it on a single channel of incoming requests" earlier, i probably would have avoided a lot of deadlocks. Crashes and concurrency errors caused by e.G. Reading the order book for get /venues/foo/symbols/bar while the match was in progress after someone placed a new order.
Mimic stock exchange
There is a significant prior level of equipment in this simulation of funny money stock organization. Most of them are clever tricks - in reality the person is not trading with another actor. When you place an order you get the results offered by the stock market simulation. Amalgamated cat food (acf) is on sale today you paid $45.32 for perfect promotions! But you did not buy logs from others. Your account has been boosted today, but there is no matching sale anywhere. Your counterparty is a sham.
For other things, there are good engineering moments and i will not refute their analysis, but the stockfighter had other goals.
Why? Well, a lot of the fun engineering problems in trading are caused by the fact that it's actually an unreliable case where you submit an order and the risk is price-matched seamlessly."Markets are distributed systems. The exchange's view of reality and the view of your trading system are necessarily separated by the great firewall known as 'physics'. To get the best possible results you should be able to do repair things like accurately predicting the future state of the exchange, because the order you place immediately will reach the future, but not the current one, while being aware that your current representation of the state of the exchange is in fact de facto is the past of the stock exchange.
Sounds complicated? It's difficult. You don't have to pay enough attention to it for the first few levels... Well, actually, let me take it back. We purposely don't tell you about the speed of light as the limiting factor at the start of stockfighter... But humans aren't prepared to eliminate latency as that's the actual limit that distributed systems have to overcome.
Here's an example. How it manifests itself: at some point during the development process, i was extremely frustrated while debugging a level that i wrote. The blotter interface showed a quote with an offer of 500 shares available for $100. (Customer quickly learn the market jargon - an ask is an offer to sell a few at a selected price. If they want to buy this is called a bid.)
I sent in a market order for monitoring. A market order is a specific type of order that does not specify a price - it says "i will pay literally no limit, i just want guaranteed execution of this order immediately." (Also: stockfighter will quickly teach players intuitively why wall street thinks people who place market orders are idiots. Use a limit order more often. -Cancel with a reasonably generous but not unlimited price attached.)
My order came back saying i bought 0 shares. And the quote from the stock exchange now reads: “500 shares are acceptable at $120.”
What?!
After a lot of searching, i figured out what was going on. : At our level, there was only one bot on the side of the order book. The mentioned bot had an outfit for a book displayed on my screen. Over the period however, as i pressed the key and my order hit the exchange, the bot clicked on the rest api to withdraw my order. Then my order arrived in an empty book, swallowing 0 shares. It then remained canceled (because market orders are instantaneous). The bot then changed its order, during which the exchange issued a new quote, which returned to my computer before i could blink. From my imperfectly attentive human point of view, it seemed that the exchange and the bot deliberately colluded to insult me.
This is not true.
But such interactions - orders lying in the book, cancellations, difficulties over time, different perspectives on reality, profiling the incoming order flow (as far as it is reflected in quotes or tailoring tapes and using it to transfer or cancel orders faster than the direct incoming flow can set up, use of execution from one exchange to decide whether to send orders to another exchange, etc. It's so much fun to play along with that, and other agents involved in the exchange with clients are also suitable for this, but not the exchange in single-player mode.
Introduce bots.
write bots
Because cooperation between players provokes many social problems that people were not willing to develop under manufacturing, and since users are not always in access to the popular game a to your quality of knowledge b) right now, we are aware that we want the ai to trade against most levels. (Among other things, we will always support pvp at some levels in subsequent chapters for all the clients who like it. If you just want to be called stockfighter as an intellectual exercise and never worry about many other people for the duration of your experience, this is the game mode we always we will support.)
It fell to me to write several trading algorithms. For context: i have the skill of cooperating with artificial intelligence (sufficient to know that it is largely not applicable to the problem at hand), and a good half of my investing experience has been in this, in order to simply buy and maintain, but not trade actively. (Also, retail investors shouldn't trade actively. If a person thinks the stockfighter has instilled the confidence to try trading actively in the meantime, i'll hit my head against the wall a few times and wonder how we propose to make the next chapter clearer just making rofl will be trampled at all if you try to do it.)
Fortunately, in the field of trade there are participants working at all skill levels.Markets are price signal aggregation mechanisms and exhibit surprisingly complex behavior even when composed of relatively uncomplicated individual agents. As soon as we had several bots using different strategies, everyone began to notice funny emergent behavior.
An example, i hope not to spoil too much: i made a technological preview of the game with a friend who gained experience in the profession on wall street. The only purposeful level in the game at the time was our tutorial, where you're just supposed to buy 100% campaigns to prove you're mechanically ready for it.
(Note: i can describe level 1 like "fizzbuzz for the rest api" but we don't see that. Fizzbuzz is widely needed to exclude non-programmers from hiring engineers as soon as possible, but the practice of asking the customer to implement this is a bit repulsive to us. Giving people a real fun goal is to demonstrate mastery in this allows us to covertly test basic engineering skills without saying "this is a basic engineering skills test" at the top of the page.)
Anyway, my friend from wall street went over the textbook asked how to proceed. I told him, “well, i don’t have anything in terms of content. However, you are connected to the exchange, and all other participants are idiots. What do you think would be interesting to do?” How dumb are they? "Pretty stupid." “So stupid that they would follow stocks to zero if someone manipulated them?” "Umm... I don't know." "Let's see."
Five minutes later, we've verified that unplanned