Building Your Own WebAssembly Cloud
David Flanagan
platform
wasm
webassembly
cloud
Did you know that the technology we use to power Fermyon Cloud is open-source?
Fermyon Platform is built on-top of Nomad, as the orchestrator and scheduler, with open-source components Bindle, Traefik, and Hippo.
We provide a repository, installer, that has all the Terraform you need to run our platform on AWS, GCP, Civo, and more.
Recently, I spent some time working on adding Equinix Metal to the installer.
Why?
Equinix Metal is one of the few cloud providers that exposes BGP to its customers, meaning we can hook into the fabric of the internet and do some very interesting routing to multiple Fermyon Platforms.
Let’s take a look.
Show Me the Code
Just want the code and not the chat?
No problem
Understanding BGP
Let’s try to understand how the internet works, by correlating it to some real world examples that everyone can understand.
You’re in your home city. A tourist stops you and asks, “How do I get to XYZ address?”. As a local, you point and direct, and the tourist marches on pretty happily. The internet isn’t much different. Routing on the internet happens through autonomous systems (AS). These ASs are the locals. They understand their neighborhood (peers) and can direct requests that are asked of it.
Let’s expand this example a little further.
Let’s assume the tourist asks a slightly different question, “Can you help me get to Starbucks?”. If your city is anything like mine, you need to put some serious thought into the answer.
Why?
There’s a whole bunch of Starbucks.
While you COULD direct them to the other side of the city, why should they walk for 40 minutes when there’s a Starbucks around the corner? You need to take into consideration where you are, where the closest Starbucks are, and give the appropriate directions.
This applies to the internet too. When you request google.com from Seattle, do you want a server in Seattle to respond, with its 10ms latency, or one in London, with its 400ms latency?
This is how BGP helps keep the internet fast.
BGP allows us to advertise a single IP address from multiple regions around the world. When you request an IP address, say 8.8.8.8, the AS closest to you will route you to the closest AS that is advertising that prefix. Meaning requests in the US go to a server in the US, and requests in the EU go to a server in the EU.
This is sometimes called Anycast and it’s how the internet giants, like Cloudflare, Google, and Amazon, provide the best services to their customers everywhere in the world.
So how can you use this to deliver the best WebAssembly service to your customers using Fermyon Platform?
Getting Started
You’ll need to clone the installer repository to your machine and jump into the Equinix Metal Terraform directory.
git clone https://github.com/fermyon/installer
cd installer/equinix-metal/terraform
In-order to deploy this, you’ll also need an Equinix Metal user API key.
export METAL_AUTH_TOKEN=XYZ
We can run terraform apply and watch as we deploy the Fermyon Platform to Dallas and Amsterdam. If you want to change the region, modify the metros variable in variables.tf.
When this has finished, we’ll be presented with some Terraform outputs. For today’s mission, we’ll need the regional_ip_addresses output. The bootstrapping happens AFTER the Terrraform apply has finished, so please give it 5 minutes to become healthy.
Warning:
The installer exposes Consul without any authentication. Ensure you shutdown your instance as soon as you’re finished playing around.
Hippo is also exposed on the public internet. However, you’ll need authentication to deploy your Spin applications.
What’s Different?
In order to make this demo work, I needed to add a few things to the installer.
First, I inject a new user-data script via cloud-init. I do this so that I can ensure we have bird running on the operating system, which handles our BGP advertisements.
Next, I hacky-patch the Hippo and Bindle Nomad jobs to ensure they can be routed to via the server IP address and the BGP address.
Caveat: While I do this for the Hippo and Bindle jobs, so that you can use spin login and spin deploy, I don’t do this for other Nomad jobs. This means you’ll need to use the BGP address to access your applications.
Deploying an Application
Let’s create a simple Spin application, build it, and deploy to our Fermyon Platform.
The example below assumes you entered “hello-world” for the name of the application.
spin new http-rust
cd hello-world
spin build
Now that our application is built, we need to push it to each of our regional platforms. We can get the IP address for each server like so:
terraform output -json | jq -r .regional_ip_addresses.value
We’ll also need the username and password for the hippo server:
terraform output -json | jq -r .hippo_admin_username.value
terraform output -json | jq -r .hippo_admin_password.value
Which means we need the following deployment steps:
# Run this for each IP address in regional_ip_addresses
spin login --url http://hippo.X.X.X.X.sslip.io
# Select 2, login with username and password
# Hippo username and password from terraform output commands above
# Bindle URL is http://bindle.X.X.X.X.sslip.io/v1, where the IP address is the regional one for this deployment
spin deploy
Testing the Latency
So what does this mean for our latency?
With both servers advertising the BGP route, from my local laptop we see httping showing us a time of in and around 70ms. From Scotland to the Netherlands, this ain’t too bad.
❯ httping http://spin-deploy.hello-world.hippo.147.75.40.12.sslip.io
PING spin-deploy.hello-world.hippo.147.75.40.12.sslip.io:80 (/):
connected to 147.75.40.12:80 (90 bytes), seq=0 time= 69.47 ms
connected to 147.75.40.12:80 (90 bytes), seq=1 time= 72.28 ms
connected to 147.75.40.12:80 (90 bytes), seq=2 time= 67.36 ms
^CGot signal 2
--- http://spin-deploy.hello-world.hippo.147.75.40.12.sslip.io/ ping statistics ---
3 connects, 3 ok, 0.00% failed, time 2643ms
round-trip min/avg/max = 67.4/69.7/72.3 ms
We can confirm the routing with mtr. You’ll notice the am6 in a hostname within the trace, which shows we hit the Equinix facility in Amsterdam.
❯ sudo mtr -4r 147.75.40.12
Start: 2023-02-23T22:13:40+0000
HOST: MacBook-Air.lan Loss% Snt Last Avg Best Wrst StDev
1.|-- amplifi.lan 0.0% 10 2.3 3.0 2.3 4.8 0.7
2.|-- 192.168.0.1 0.0% 10 4.8 4.4
... random hops ...
10.|-- 464.et-0-0-2.bsr1.am6.pac 0.0% 10 29.1 33.5 29.1 43.5 5.5
11.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
12.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
13.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
14.|-- 147.75.40.12 0.0% 10 32.6 30.5 27.8 36.3 2.6
Let’s shut down bird on our Amsterdam server and see how this affects our results.
systemctl stop bird
Now our tests show:
❯ httping http://spin-deploy.hello-world.hippo.147.75.40.12.sslip.io
PING spin-deploy.hello-world.hippo.147.75.40.12.sslip.io:80 (/):
connected to 147.75.40.12:80 (90 bytes), seq=0 time=342.05 ms
connected to 147.75.40.12:80 (90 bytes), seq=1 time=256.82 ms
connected to 147.75.40.12:80 (90 bytes), seq=2 time=307.51 ms
connected to 147.75.40.12:80 (90 bytes), seq=3 time=305.30 ms
^CGot signal 2
--- http://spin-deploy.hello-world.hippo.147.75.40.12.sslip.io/ ping statistics ---
4 connects, 4 ok, 0.00% failed, time 4716ms
round-trip min/avg/max = 256.8/302.9/342.1 ms
Just for clarity, our mtr now looks like:
❯ sudo mtr -4r 147.75.40.12
Start: 2023-02-23T22:22:06+0000
HOST: MacBook-Air.lan Loss% Snt Last Avg Best Wrst StDev
1.|-- amplifi.lan 0.0% 10 4.6 3.1 2.7 4.6 0.6
2.|-- 192.168.0.1 0.0% 10 5.3 4.8
... random hops ...
10.|-- 465.et-0-0-1.bsr1.ny5.pac 10.0% 10 136.0 72.2 28.5 140.7 50.3
464.et-0-0-2.bsr1.am6.packet.net
11.|-- 434.et-0-1-3.bbr1.da11.pa 60.0% 10 125.1 126.4 124.3 131.1 3.2
424.et-0-0-5.bsr1.jfk3.packet.net
12.|-- 903.et-0-0-0.bsr1.ny5.pac 90.0% 10 133.4 133.4 133.4 133.4 0.0
13.|-- 434.et-0-1-3.bbr1.da11.pa 90.0% 10 135.1 135.1 135.1 135.1 0.0
14.|-- 147.75.40.12 70.0% 10 131.8 133.0 131.8 135.0 1.8
As we can see, getting our Spin applications as close to our customers as possible doesn’t take too much effort, but delivers a huge performance benefit; lowering response times by over 4x. When your customers are online and worldwide, your infrastructure should be too. With these gains and Fermyon Platform’s abiltiy to schedule and run your WebAssembly applications in under a 1ms, you can deliver a truly global experience.
Granted, we’re not concerned ourselves with the wonderful world of stateful applications … but that’s a story for another day.
Summary
Using awesome internet technologies like BGP allows us to do some amazing things with wonderful open source software, such as Fermyon Platform. If you want to build your own WebAssembly cloud that serves your customers no matter where they are, we’ve got you covered.
Of course, if you want to let us do the heavy lifting; we got you. Remember to checkout Fermyon Cloud, our managed WebAssembly Cloud for all your Spin applications.
Have a great day.