image credit Rahul Chakraborty (@hckmstrrahul), via Unsplash
Do you have an Amazon AWS account? Do you with things were more automated and just easier to manage? Want to do that using BOSH, but wish there was a simple way to deploy BOSH? Me too. So, let's do that.
Before you can do anything in the Genesis ecosystem, you really need a BOSH director. But why is Genesis so enamoured with BOSH?
Simple:
BOSH is, first and foremost, an orchestrator of virtual machines. You tell it what you want deployed, and it handles all the nitty-gritty details of spooling up compute and memory resources, storage volumes, etc.
We're going to use Amazon AWS EC2 instances in this tutorial, but that's not the whole of BOSH's unique bag of tricks. Through the abstraction of a cloud provider interface (or "CPI" if you prefer TLAs), BOSH supports a whole range of cloud infrastructures, including Microsoft Azure, Google Cloud Platform, VMware vSphere ESXi, and more. The best part? You don't have to do anything terribly special to move a deployment manifest (the recipe you give BOSH that tells it how to deploy stuff) from one cloud to another!
But BOSH does way more than just stand up new VMs. It keeps a watchful eye (a tedious task that only a robot does well) on all the machines it deploys, and if they go away, it replaces them with new instances. This is critical for running large distributed systems (like Cloud Foundry), since individual VMs can go missing for a variety of reasons: the cloud provider loses the instance, software crashes, etc. BOSH notices these types of outages and happily provisions replacement machines and puts them into service.
Finally, separation of concerns becomes increasingly important as we transition into the day 2 operations phase of running a platform. With BOSH, there are two types of updates: OS updates and release updates.
The majority of security patches are going to come through the former channel. As upstream OS vendors (Canonical, Microsoft, etc.) upgrade core system components like the kernel, OpenSSL libraries, etc., the BOSH team is building updated stemcells that roll all of those changes up into a nice, versioned package that operators can deploy. These stemcells are wholly independent of any of the software we deploy (like Kubernetes).
Application-specific security patches, as well as bug fixes and new features, come through the latter channel. For example, when the Cloud Foundry UAA team patches UAA, they release a new BOSH release that we can then deploy on top of our existing stemcells.
Taken together, these two avenues of updates allow operators to choose when and how to patch their systems, based on business need, security posture, and risk tolerance. When you're running a platform with thousands of customers, atop hundreds of VMs, this becomes a luxury you'll not want to go without.
That's why Genesis loves BOSH so much.
We are going to assume that you are starting out with nothing deployed in Amazon AWS, and work our way up to a BOSH director that you can use (in concert with Genesis) to deploy all kinds of other fun things.
To that end, we will build the following:
This is the beginning of an entire Genesis Control plane, but we'll save the rest of the deployments for future tutorials.
The first four things on our list are deployed via Terraform. To make this easier (and since few truly enjoy writing Terraform configurations), we are going to use the Genesis Terraforms project:
$ cd ~/ops $ git clone https://github.com/genesis-community/terraforms $ cf terraforms/lab/aws
Before Terraform can do its thing, we need to provide it some credentials
for accessing Amazon AWS on our behalf. Those will go in the aws.tfvars
file:
$ cat >aws.tfvars <
The only two values you really must change are the aws_access_key and
aws_secret_key.
We also need to generate an EC2 SSH keypair, which will go on the Bastion
host that we are about to deploy. You may want to refer to these
step-by-step instructions. Make sure the private key exists
in the terraform/lab/aws directory, and is named according to whatever you
put in the aws_key_file variable.
To deploy, we've provided a helpful Makefile, so all you have to do is
type:
$ make
This spits out a lot of output, mostly just Terraform explaining all the work its about to perform. At the end, you should see this:
Plan: 14 to add, 0 to change, 0 to destroy.
Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve.
Enter a value:
Type yes (Terraform will be happy with nothing else) and it's off to the
races! And by races, I mean the long and (relatively) slow process of
standing up infrastructure inside of Amazon's data centers. Go grab a cup
of tea and let Terraform do its thing.
When all is said and done, your termainal should say something like this:
For configuring your proto-BOSH:
Static IP: 10.4.0.4
Range (CIDR): 10.4.0.0/20
Gateway: 10.4.0.1
DNS: 10.4.0.2
AWS Region: us-west-2
VM Sec. Group: open-lab
BOSH Sec. Group: open-lab
Subnet ID: subnet-01c354c3403d4149e
To access the bastion host:
ssh -i genesis-lab1.pem ubuntu@35.164.43.170
(your IP address and Subnet ID will be different.)
The bastion host has all the software you need to run Genesis, including
Spruce, Vault, the BOSH and CF CLIs, and, of course, genesis itself.
The first time you log into the bastion, you're going to have to do a little bit of configuration.
$ sudo jumpbox system
The first command sets up global jumpbox-y things. It installs a slew of
useful utilities like safe, spruce, jq, and more.
It also installs required commands like bosh, cf, and genesis.
To see what's installed, just run jumpbox with no arguments.
$ jumpbox
Checking jumpbox installation jumpbox installed - jumpbox v55 ruby installed - ruby 2.3.1p112 (2016-04-26) [x86_64-linux-gnu] bosh installed - version 6.1.0-9c1c210c-2019-09-18T17:33:51Z cf installed - cf version 6.46.1+4934877ec.2019-08-23 jq installed - jq-1.6 spruce installed - spruce - Version 1.22.0 safe installed - safe v1.3.4 vault installed - Vault v1.2.3 genesis installed - [installing genesis runtime] sipcalc installed - sipcalc 1.1.6
git user.name is 'B. Astion'
git user.email is 'ubuntu@lab.genesis'
To bootstrap this installation, try `jumpbox system`
To set up your personal environment: `jumpbox user`
To update this copy of jumpbox, use: `jumpbox sync`
To create a new local user account: `jumpbox useradd`
One final note before we jump into BOSH and Genesis: tmux is an amazing
piece of software.
It's a terminal multiplexer that lets you run lots of terminal sessions, across a single SSH link. If you get disconnected, have to reboot your laptop, or just want to unplug for a long weekend, your tmux session will still hum along happily on the remote server.
To use tmux, just run tmux from the bastion host (that's important!)
It should look something like this:
For the rest of this article, I'm going to assume you're doing everything from a tmux session. There's a tmux cheat sheet out there if you aren't 100% comfortable in tmux.
Genesis leverages Hashicorp Vault to generate and securely store passwords, SSH keys, X.509 certificates and more. Once we get a BOSH director up and running (our very next task, actually), we will use it to deploy a Vault instance. Until we get that far, however, we need an initial Vault to jumpstart the process.
This is where safe proves valuable.
Safe (an alternative command-line interface for Vault) provides the means to run a local Vault instance, on loopback. It's so easy, we won't have to worry about configuration, process management, or unsealing / initializing our nascent Vault.
So, without further ado:
$ safe local --memory --as init Now targeting (temporary) init at http://127.0.0.1:8201 This Vault is MEMORY-BACKED! If you want to retain your secrets be sure to safe export. Ctrl-C to shut down the Vault
This gives us a memory-backed Vault, named init. If we kill the process,
we will lose all the secrets. That might seem like a bad thing, but we only
really have to get through two deployments (about 2 hours wall time), and
a memory-backed initial Vault won't leak credentials as easily.
Once the local Vault is up and running, we can interact with it via safe:
$ safe tree $ safe paths
etc.
Before we can begin deploying things with BOSH, we need a BOSH director.
This will be our very first Genesis deployment in this environment, and
we'll use the BOSH create-env tooling (under the hood) to get up and
running.
(If you don't know what create-env is, or have never stood up a BOSH director, don't worry; we'll walk through every step of the process.)
In Genesis, a proto-BOSH is the initial deployment. It is used, in turn, to deploy both the operations tier (Vault, SHIELD, Concourse, etc.), and also each of the other environment BOSH directors. Having BOSH deployed by another BOSH seems a bit weird at first, but it turns out to be a surprisingly useful optimization. For one thing, it makes for faster BOSH director updates.
To deploy something with Genesis, you need a Kit. Kits roll up most of the tedious bits of configuring BOSH releases into usable deployments. They can be a bit opinionated at times, but we think that makes for better software and systems.
Specifically, we are going to use the (unsurprisingly-named) BOSH Genesis
Kit. Since this our first deployment of BOSH, we're going to initialize a
new set of deployments, using genesis init (remember to do this from a
tmux session):
$ mkdir ~/ops $ cd ~/ops $ genesis init -k bosh --vault init Downloading Genesis kit bosh (latest version)...
Initialized empty Genesis repository in /home/ubuntu/ops/bosh-deployments using the bosh/1.5.0 kit.
The genesis init command creates a deployments directory for the type of
thing you are deploying (in our case, BOSH), and then downloads a Genesis
Kit from GitHub.
Next, we need to write an environment file for our new BOSH director.
$ cd bosh-deployments $ genesis new jhunt-aws
Genesis will then ask a whole bunch of (pertinent) questions about your configuration, infrastructure / cloud provider, and your preferences. First up, Genesis needs to know if you want a proto-BOSH or a regular BOSH. We don't yet have a BOSH director, so we need to deploy a proto-BOSH. Say "y" here.
Is this a proto-BOSH director? [y|n] > y
After that, Genesis asks a series of questions about the networking for this director. These answers come from our understanding of the Terraform configuration we deployed. You can use these values verbatim.
What static IP do you want to deploy this BOSH director on?
10.4.0.4
What network should this BOSH director exist in (in CIDR notation)?
10.4.0.0/20
What default gateway (IP address) should this BOSH director use?
10.4.0.1
What DNS servers should BOSH use? (leave value empty to end) 1st value > 10.4.0.2 2nd value >
To properly configure the BOSH director, Genesis needs to know where we are deploying (what cloud provider / IaaS). This is AWS, so answer "2".
What IaaS will this BOSH director orchestrate? 1) VMWare vSphere 2) Amazon Web Services 3) Microsoft Azure 4) Google Cloud Platform 5) OpenStack 6) BOSH Warden
Select choice > 2
Now that Genesis knows we're in Amazon, it can ask more pointed questions about how we expect the BOSH director to access AWS on our behalf.
What AWS region would you like to deploy to?
us-west-2
What is your AWS Access Key?
... type your key here ...
What is your AWS Secret Key? secretkey [hidden]: secretkey [confirm]:
Next up, you will be asked for security groups, and the subnet ID to deploy the director to.
What security groups should the all deployed VMs be placed in? (leave value empty to end) 1st value > open-lab 2nd value >
What is the ID of the AWS subnet you want to deploy to?
subnet-01c354c3403d4149e
What security groups should the BOSH Director VM be in? (leave value empty to end) 1st value > open-lab 2nd value >
Hint: you can find the AWS Subnet ID in the output of make info.
(The open-lab security group allows all inbound and outbound traffic, on all ports, to and from all hosts, all protocols. It won't pass muster on a production system, but it sure is handy in lab environments.)
That concludes the question & answer session.
Finally, you'll be asked if you want to edit the environment file. You can safely say "no" here; we won't be customizing this proto-BOSH.
Now it is Genesis' turn to do some thinking and processing. You should see something that looks like this (depending on what you named your environment):
- auto-generating credentials (in secret/jhunt/aws/bosh)...
- auto-generating certificates (in secret/jhunt/aws/bosh)...
New environment jhunt-aws provisioned!
To deploy, run this:
genesis deploy 'jhunt-aws'
Genesis just generated all of the random passwords and required X.509
certificates that a functioning BOSH director needs. Isn't that neat?
There is one thing to be done manually in the AWS console, and the Genesis
new wizard helpfully explains the what, why, and how.
Before we go off and genesis deploy this BOSH director, scroll back
through the output in your terminal until you find this:
Before deploying, please be sure to import the keypair generated for you from Vault into AWS console.
First run the following command to get the public key:
safe get secret/jhunt/aws/bosh/aws/ssh:public
Then go to EC2 > Key Pairs > Import Key Pair and:
1. Type 'vcap@jhunt-aws' in the 'Key pair name' input box
2. Paste the safe command output into the 'Public key contents' input box
3. Click 'Import' button
Now you can SSH into VMs deployed by this director using the generated key.
Now that the keys for the SSH EC2 keypair have been generated in the Vault, it's time time import them into AWS itself.
So go do that.
Finally, we can run that genesis deploy command.
$ genesis deploy jhunt-aws
When that finishes, follow the on-screen instructions and log into the BOSH director:
$ genesis do jhunt-aws -- login Running login addon for jhunt-aws Logging you in as user 'admin'... Using environment 'https://10.4.0.4:25555'
Email (): admin Password ():
Successfully authenticated with UAA
Succeeded
The BOSH director is deployed, but before we can use it to deploy something, we need to do two things.
The BOSH Genesis Kit sports this wonderful addon called upload-stemcells that provides a menu driven interface for selecting and uploading stemcells (the stuff deployment VMs are made of). It properly limits which stemcell variants are considered, based on the IaaS you configured the director for.
Since we're going to need a stemcell, let's take it for a spin!
$ genesis do jhunt-aws -- upload-stemcells
This will show you the major and minor versions of the Xenial stemcell series (Ubuntu 16.04 LTS) for AWS, and let you pick which you want to upload. The upload is handed off to the BOSH director, and when you get back to a prompt you should be good to go.
I tested with Xenial 456.x (latest minor revision).
BOSH Cloud Config is one of the more arcane and esoteric parts of BOSH. To help make this easier, the Genesis Terraforms repository can build a rudimentary, baseline cloud config, if you ask it to.
From wherever you ran the Terraforming process, run:
$ make cc
That will print out a cloud config YAML document that you can transfer to
the bastion host, and use that to bosh update-cloud-config.
This cloud config defines a default network, some generic VM types (small,
medium, large), a default disk type, and a single availability zone (z1).
It should suffice to get us through the next deployment, and since this is a
lab, you're probably going to modify it eventually anyway!
Now that we have a BOSH director, a stemcell, and a cloud config, we can
deploy a real Vault to replace the ephemeral safe local Vault we've been
using up to now. We'll repeat the same dance of genesis init / genesis
new, only this time, for the Vault Genesis Kit.
$ cd ~/ops $ genesis init -k vault $ cd vault-deployments $ genesis new jhunt-aws
The Vault kit is mercifully less chatty than the BOSH Director kit, so there
isn't much in the way of questions to answer during genesis new. We will
be asked if this is the Genesis Vault. It is, so answer "y" to that one.
If you try to deploy the Vault now (go ahead, I'll wait) it will fail, because the Vault Kit expects to deploy a highly-available cluster across three different availability zones. Our lab only has one AZ, so it doesn't work.
Not a big deal, all we have to do is open up the jhunt-aws.yml file (in
vault-deployments/) in our favorite editor, and add this to the params:
stanza:
params: availabilityzones: [z1] stemcellos: ubuntu-xenial
Changing stemcell_os is also necessary, because (at least as of 1.2.1) the
Vault Kit expects to deploy to the latest Ubuntu Trusty (14.04 LTS), but we
only uploaded Xenial stemcells.
Now we can deploy.
$ genesis deploy jhunt-aws
After the deployment succeeds, we need to initialize the Vault:
$ genesis do jhunt-aws -- init
This sets up a safe target in your local configuration, named jhunt-aws
(after the environment). We are now able to transfer all of our
credentials from the ephemeral init Vault to their forever home.
$ safe -T init export | safe -T jhunt-aws import
It is now safe to kill the safe local process.
Congratulations!
🎉
Hopefully you've had fun setting up this little AWS lab. The BOSH director we deployed has Credhub installed, so you're able to deploy whatever you want to it, with or without the power of Genesis.
If you do think Genesis is awesome sauce, check out the other kits and see if anything tickles your fancy. Official Genesis Kits live on GitHub, in the Genesis Community organization. Notable Kits include:
Concourse - Deploy a CI/CD pipeline and get started with automating your Genesis deployments!
SHIELD - A data protection solution for the cloud. Schedule backups and perform restores on databases, key-value stores, even file systems.
Cloud Foundry - The Cloud Foundry PaaS itself. Now deployed via Genesis.
Blacksmith - Data services, on-demand, leveraging BOSH. Available for CF marketplaces and Kubernetes!