Finicky Whiskers (pt. 4): Spin, Containers, Nomad, and Infrastructure
Matt Butcher
finicky whiskers
nomad
aws
krustlet
Finicky Whiskers is a fast-paced game about feeding a picky kitty.
It’s also the world’s most adorable manual load generator.
It showcases how WebAssembly modules can be started, executed, and shut down in the blink of an eye.
In a typical game of Finicky whiskers, where a player scores 100 points, 260+ WebAssembly modules are started, run, and terminated.
Contrast that with current container technology, and you’ll quickly understand why we believe WebAssembly is critical for the next wave of cloud computing.
This is the fourth and final part of our Finicky Whiskers series.
We started our series with an overview of the world’s most adorable manual load generator.
Next, we looked at how Finicky Whiskers serves static files quickly and easily.
From there, we dove into code, taking a look at the five microservices that drive Finicky Whiskers.
In this post, we’re going to dive into the Finicky Whiskers infrastructure.
The Lowest Level: Our Amazon setup
We run Finicky Whiskers on AWS. But we only use the basic services common to most cloud platforms.
We use compute, load balancing, and a little bit of storage.
That means that we’ve avoided any special services that might lock us in to just one cloud vendor.
We could just as easily have run on Azure, Digital Ocean (like our Spin demo), or any other major cloud provider.
As we’ve touted before,
we run Finicky Whiskers replicated on three AWS t2.small instances.
To put that in perspective, an average empty Kubernetes cluster starts at large instance sizes.
That immediately translates to cost savings for us.
Under observed load, these little VM instances never break a sweat.
We typically see them consuming around 50% of the memory and 10% of the CPU.
We’ve never seen an outage or even a slowdown.
The Orchestrator: Nomad
Long ago, we wrote a Kubernetes kubelet for WebAssembly called Krustlet.
It was a fun project to work on, but after several months of active development,
it became clear to us that there was a world beyond Kubernetes that we needed to explore
as part of this next wave of cloud computing.
So we looked for an alternative. And Nomad won our hearts almost immediately.
We had spent many months and tens of thousands of lines of code to get Krustlet stable enough to run a few WebAssembly modules.
But in just a couple of days we had Wagi (and then Spin) running in Nomad.
It was easier to configure, faster to run, and more reliable too.
We just used Nomad’s exec driver as-is, and had it invoke Spin.
We traded in gobs of code for a couple short HCL files.
Over the last few months, we’ve been working on our Nomad configuration.
Right now, we are running an HA cluster with redundant Consul, Vault, and Nomad instances.
And when we deploy Finicky Whiskers, it is replicated across three worker nodes (all of which are t2.smalls).
The Nomad cluster has been rock solid, and running it in an HA configuration is definitely overkill.
But we do it because we feel it is important to garner knowledge about production-grade infrastructure.
That said, it does feel a little bit funny that right now our control plane is bigger than our data plane
(six VMs on the control plane, versus the three on the data plane).
The control plane is responsible for keeping a cluster operational. The data plane is responsible for running workloads (containers, webassembly binaries). The control plane manages the data plane.
Non-Spin Workloads: Traefik, Bindle, Redis
In addition to the Spin instances that run Finicky Whiskers, we have a few more things sitting on the data plane.
We use Traefik, a proxy, to balance load across our three worker nodes.
Traefik is loaded with great features, and has been easy for us to configure and maintain.
Redis is used by the Finicky Whiskers game.
We run Redis in a container managed as a Nomad job.
This has been one advantage to Nomad. We can manage and run both our container workloads and our WebAssembly microservices,
and both feel right at home in Nomad.
Bindle is a package management backend for WebAssembly. Not long ago, we wrote a post explaining how Bindle is designed to store WebAssembly applications, where an application is a collection of WebAssembly microservices or binaries.
Finicky Whiskers is a good example of this: The seven microservices that jointly compose Finicky Whiskers (along with all of the static assets) can all be packaged up into a single bindle and stored in the Bindle server.
Most of the time, we run our non-Spin workloads in containers. This allows us to easily encapsulate dependencies for services that were not designed to be compiled to WebAssembly. Again, as we like to say, we view containers and WebAssembly as complementary technologies.
Deploying the Infrastructure
Most of the time, when we are “deploying” Finicky Whiskers, we are just installing or upgrading the WebAssembly microservices. But to make things easy on ourselves, we have a Terraform project that can create an entire cluster capable of running Finicky Whiskers. In fact, this same setup is used to manage the infrastructure for Fermyon.com, where we run our site on Bartholomew.
It might be evident, then, that we run Finicky Whiskers in its own cluster. Why do that? We could, after all, easily run Fermyon.com and Finicky Whiskers in the same cluster.
The main reason lies in the purpose of Finicky Whiskers. We’re not (just) joking when we call it the “world’s most adorable manual load generator.”
We keep a close eye on the Finicky Whiskers cluster to understand how it performs under various conditions.
When we talk about it at a conference, we tend to see load go way up.
During an internationally recognized holiday weekend, we tend to see load go way down.
In each case, we gain some visibility into how the cluster behaves, how much it cost to operate, and how we can optimize things.
Conclusion
Our infrastructure layer has served us well, not just for Finicky Whiskers, but also for Fermyon.com.
We are looking forward to sharing more about how to build a scalable, cost-effective, energy-efficient cluster for WebAssembly microservices.
Stay tuned!
Are you a finicky fan? We’ll be giving away Finicky Whiskers t-shirts and stickers at Open Source Summit in Austin.
In Summary:
Finicky Whiskers (pt. 1): “The World’s Most Adorable Manual Load Generator.”
Finicky Whiskers (pt. 2): “Serving the HTML, CSS, and static assets.”
Finicky Whiskers (pt. 3): “The Microservices.”
Finicky Whiskers (pt. 4): “Spin, Containers, Nomad, and Infrastructure.”