image credit Francesco Gallarotti (@gallarotti), via Unsplash
Everybody needs a place to play, to experiment, and to try wacky things with new technology. It's the same when you're just getting started with Genesis — you need a lab environment!
For this to work, you're going to need an Amazon Web Services (AWS) account, both parts of an Amazon Access Key ID / Secret Access Key set, and an EC2 (SSH) key pair (which we can generate).
We've just released Genesis Terraforms, a (hopefully growing) collection of purpose-built Terraform configurations that make it easy to spin up different bits of cloud infrastructure, on different providers. Want a production operations tier on GCP? Genesis Terraforms. Want to train a bunch of people on Concourse? Genesis Terraforms. Trying to follow along in a blog post about setting up a lab in AWS? Genesis Terraforms.
First, you're going to want to clone the git repository.
$ git clone https://github.com/genesis-community/terraforms
The particular configuration we'll be targeting is in the lab/aws
sub-directory. It requires a few variables, so we'll have to create an
aws.tfvars file inside there.
$ cd terraforms/lab/aws $ vim aws.tfvars
Your tfvars file will look something like this:
awsaccesskey = "AKI..." awssecretkey = ".. your actual secret key ..." awsvpcname = "jhunt-genesis-lab1" awskeyname = "jhunt-genesis-lab1" awskeyfile = "jhunt-genesis-lab1.pem"
(I'll be flattered if you name your first-deployed VPC after me, really I will.)
The Access Key and Secret Key will be unique to you. You can set the VPC name to anything you want; I like to make mine descriptive so I know what I can and cannot delete in the AWS console.
That aws_key_file needs to be set to the path of the EC2 key pair's
private key component. If you don't already have one of these, they are
easy enough to generate in the AWS console. You may want to refer to these
step-by-step instructions.
Once you've got all that, command the combined power of a thousand robots with this one simple incantation:
$ make
That will compile the Terraform plan, taking into account the variables we just set, and provide you a summary of what is to be done to AWS. Type "yes" when prompted and Terraform will go off and deploy a VPC, some subnets, security groups, a Linux bastion host with an elastic IP, and some other stuff.
When that's all settled, you should see this:
Apply complete! Resources: 12 added, 0 changed, 0 destroyed.
... some more output ...
For configuring your proto-BOSH:
Static IP: 10.4.0.4
Range (CIDR): 10.4.0.0/24
Gateway: 10.4.0.1
DNS: 10.4.0.2
AWS Region: us-west-2
VM Sec. Group: open-lab
BOSH Sec. Group: open-lab
Subnet ID: subnet-07c0644b5296640a9
To access the bastion host:
ssh -i jhunt-genesis-lab1.pem ubuntu@some.ip.addr.ess
Congratulations, you've successfully configured all the IaaS bits!
So what have we just deployed? It looks a little something like this:
There are two networks, DMZ and Lab, which appropriate gateways and
NAT devices to make the Internet traffic work. Inside the DMZ network
(a 10.4.255.192/26), we have a bastion host, which has a public IPv4
address. This is how we'll get into the VPC to administer it, configure
BOSH, run genesis commands, etc.
The two networks can see each other completely; traffic from the DMZ can pass freely into the interior Lab network, and vice-versa. Since we're testing, the security groups are wide-open and the network ACLs (that firewall out front) are virtually not there.
From the bastion host, we will deploy our (proto-)BOSH director into the Lab network, and use that BOSH director to deploy our Vault node(s), and anything else that strikes our fancy.
The bastion host has all the software you need to run Genesis, including
Spruce, Vault, the BOSH and CF CLIs, and, of course, genesis itself.
The first time you log into the bastion, you're going to have to do a little bit of configuration.
$ sudo jumpbox system
The first command sets up global jumpbox-y things. It installs a slew of
useful utilities like safe, spruce, jq, and more.
It also installs required commands like bosh, cf, and genesis.
To see what's installed, just run jumpbox with no arguments.
$ jumpbox
Checking jumpbox installation jumpbox installed - jumpbox v55 ruby installed - ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux-gnu] bosh installed - version 3.0.1-712bfd7-2018-03-13T23:26:43Z cf installed - cf version 6.40.1+85d04488a.2018-10-31 jq installed - jq-1.5 spruce installed - spruce - Version 1.18.2 safe installed - safe v0.9.9 vault installed - Vault v0.9.6 ('7e1fbde40afee241f81ef08700e7987d86fc7242') genesis installed - Genesis v2.6.12 (563bd7a7ee) build 20180926.165834 sipcalc installed - sipcalc 1.1.6
git user.name is 'James Hunt'
git user.email is 'jhunt@starkandwayne.com'
To bootstrap this installation, try `jumpbox system`
To set up your personal environment: `jumpbox user`
To update this copy of jumpbox, use: `jumpbox sync`
To create a new local user account: `jumpbox useradd`
One final note before we jump into BOSH and Genesis: tmux is an amazing
piece of software.
It's a terminal multiplexer that lets you run lots of terminal sessions, across a single SSH link. If you get disconnected, have to reboot your laptop, or just want to unplug for a long weekend, your tmux session will still hum along happily on the remote server.
To use tmux, just run tmux from the bastion host (that's important!)
It should look something like this:
For the rest of this article, I'm going to assume you're doing everything from a tmux session. There's a tmux cheat sheet out there if you aren't 100% comfortable in tmux.
Genesis leverages Hashicorp Vault to generate and securely store passwords, SSH keys, X.509 certificates and more. Once we get a BOSH director up and running (our very next task, actually), we will use it to deploy a Vault instance. Until we get that far, however, we need an initial Vault to jumpstart the process.
This is where safe proves valuable.
Safe (an alternative command-line interface for Vault) provides the means to run a local Vault instance, on loopback. It's so easy, we won't have to worry about configuration, process management, or unsealing / initializing our nascent Vault.
So, without further ado:
$ safe local --memory --as init Now targeting (temporary) init at http://127.0.0.1:8201 This Vault is MEMORY-BACKED! If you want to retain your secrets be sure to safe export. Ctrl-C to shut down the Vault
This gives us a memory-backed Vault, named init. If we kill the process,
we will lose all the secrets. That might seem like a bad thing, but we only
really have to get through two deployments (about 2 hours wall time), and
a memory-backed initial Vault won't leak credentials as easily.
Once the local Vault is up and running, we can interact with it via safe:
$ safe tree $ safe paths
etc.
Before we can begin deploying things with BOSH, we need a BOSH director.
This will be our very first Genesis deployment in this environment, and
we'll use the BOSH create-env tooling (under the hood) to get up and
running.
(If you don't know what create-env is, or have never stood up a BOSH director, don't worry; we'll walk through every step of the process.)
In Genesis, a proto-BOSH is the initial deployment. It is used, in turn, to deploy both the operations tier (Vault, SHIELD, Concourse, etc.), and also each of the other environment BOSH directors. Having BOSH deployed by another BOSH seems a bit weird at first, but it turns out to be a surprisingly useful optimization. For one thing, it makes for faster BOSH director updates.
To deploy something with Genesis, you need a Kit. Kits roll up most of the tedious bits of configuring BOSH releases into usable deployments. They can be a bit opinionated at times, but we think that makes for better software and systems.
Specifically, we are going to use the (unsurprisingly-named) BOSH Genesis
Kit. Since this our first deployment of BOSH, we're going to initialize a
new set of deployments, using genesis init (remember to do this from a
tmux session):
$ mkdir ~/ops $ cd ~/ops $ genesis init -k bosh Downloading Genesis kit bosh (latest version)...
Initialized empty Genesis repository in /home/ubuntu/ops/bosh-deployments using the bosh/1.2.1 kit.
The genesis init command creates a deployments directory for the type of
thing you are deploying (in our case, BOSH), and then downloads a Genesis
Kit from GitHub.
Next, we need to write an environment file for our new BOSH director.
$ cd bosh-deployments $ genesis new jhunt-aws
Genesis will then ask a whole bunch of (pertinent) questions about your configuration, infrastructure / cloud provider, and your preferences. First up, Genesis needs to know what Vault to store your credentials in:
Known Vault targets - current target indicated with a (): () init (insecure) http://127.0.0.1:8201
Which Vault would you like to target?
init Now targeting init at http://127.0.0.1:8201
The next question is about proto-BOSH vs. regular BOSH. We don't yet have a BOSH director, so we need to deploy a proto-BOSH. Say "y" here.
Is this a proto-BOSH director? [y|n] > y
After that, Genesis asks a series of questions about the networking for this director. These answers come from our understanding of the Terraform configuration we deployed. You can use these values verbatim.
What static IP do you want to deploy this BOSH director on?
10.4.0.4
What network should this BOSH director exist in (in CIDR notation)?
10.4.0.0/24
What default gateway (IP address) should this BOSH director use?
10.4.0.1
What DNS servers should BOSH use? (leave value empty to end) 1st value > 10.4.0.2 2nd value >
To properly configure the BOSH director, Genesis needs to know where we are deploying (what cloud provider / IaaS). This is AWS, so answer "2".
What IaaS will this BOSH director orchestrate? 1) VMWare vSphere 2) Amazon Web Services 3) Microsoft Azure 4) Google Cloud Platform 5) OpenStack 6) BOSH Warden
Select choice > 2
Now that Genesis knows we're in Amazon, it can ask more pointed questions about how we expect the BOSH director to access AWS on our behalf.
What AWS region would you like to deploy to?
us-west-2
What is your AWS Access Key?
AKIyourkeyhere... Now targeting init at http://127.0.0.1:8201
What is your AWS Secret Key? secretkey [hidden]: secretkey [confirm]:
Next up, you will be asked for security groups, and the subnet ID to deploy the director to.
What security groups should the all deployed VMs be placed in? (leave value empty to end) 1st value > open-lab 2nd value >
What is the ID of the AWS subnet you want to deploy to?
subnet-0b79be9eaeb7d164b
What security groups should the BOSH Director VM be in? (leave value empty to end) 1st value > open-lab 2nd value >
Hint: you can find the AWS Subnet ID in the output of make info.
(The open-lab security group allows all inbound and outbound traffic, on all ports, to and from all hosts, all protocols. It won't pass muster on a production system, but it sure is handy in lab environments.)
That concludes the question & answer session.
There is one thing to be done manually in the AWS console, and the Genesis
new wizard helpfully explains the what, why, and how.
Before deploying, please be sure to import the keypair generated for you from Vault into AWS console.
First run the following command to get the public key:
safe get secret/jhunt/aws/bosh/aws/ssh:public
Then go to EC2 > Key Pairs > Import Key Pair and:
1. Type 'vcap@jhunt-aws' in the 'Key pair name' input box
2. Paste the safe command output into the 'Public key contents' input box
3. Click 'Import' button
Now you can SSH into VMs deployed by this director using the generated key.
So go do that.
Finally, you'll be asked if you want to edit the environment file. You can safely say "no" here; we won't be customizing this proto-BOSH.
Now it is Genesis' turn to do some thinking and processing. You should see something that looks like this (depending on what you named your environment):
- auto-generating credentials (in secret/jhunt/aws/bosh)...
- auto-generating certificates (in secret/jhunt/aws/bosh)...
New environment jhunt-aws provisioned!
To deploy, run this:
genesis deploy 'jhunt-aws'
Genesis just generated all of the random passwords and required X.509 certificates that a functioning BOSH director needs. Isn't that neat?
Literally all you have to do is run that genesis deploy command.
$ genesis deploy jhunt-aws
When that finishes, follow the on-screen instructions and log into the BOSH director:
$ genesis do jhunt-aws -- login Running login addon for jhunt-aws Logging you in as user 'admin'... Using environment 'https://10.4.0.4:25555'
Email (): admin Password ():
Successfully authenticated with UAA
Succeeded
The BOSH director is deployed, but before we can use it to deploy something, we need to do two things.
The BOSH Genesis Kit sports this wonderful addon called upload-stemcells that provides a menu driven interface for selecting and uploading stemcells (the stuff deployment VMs are made of). It properly limits which stemcell variants are considered, based on the IaaS you configured the director for.
Since we're going to need a stemcell, let's take it for a spin!
$ genesis do jhunt-aws -- upload-stemcells
This will show you the major and minor versions of the Xenial stemcell series (Ubuntu 16.04 LTS) for AWS, and let you pick which you want to upload. The upload is handed off to the BOSH director, and when you get back to a prompt you should be good to go.
I tested with Xenial 97.x (latest minor revision).
BOSH Cloud Config is one of the more arcane and esoteric parts of BOSH. To help make this easier, the Genesis Terraforms repository can build a rudimentary, baseline cloud config, if you ask it to.
From wherever you ran the Terraforming process, run:
$ make cc
That will print out a cloud config YAML document that you can transfer to
the bastion host, and use that to bosh update-cloud-config.
This cloud config defines a default network, some generic VM types (small,
medium, large), a default disk type, and a single availability zone (z1).
It should suffice to get us through the next deployment, and since this is a
lab, you're probably going to modify it eventually anyway!
Now that we have a BOSH director, a stemcell, and a cloud config, we can
deploy a real Vault to replace the ephemeral safe local Vault we've been
using up to now. We'll repeat the same dance of genesis init / genesis
new, only this time, for the Vault Genesis Kit.
$ cd ~/ops $ genesis init -k vault $ cd vault-deployments $ genesis new jhunt-aws
The Vault kit is mercifully less chatty than the BOSH Director kit, so there
isn't much in the way of questions to answer during genesis new. We will
be asked if this is the Genesis Vault. It is, so answer "y" to that one.
If you try to deploy the Vault now (go ahead, I'll wait) it will fail, because the Vault Kit expects to deploy a highly-available cluster across three different availability zones. Our lab only has one AZ, so it doesn't work.
Not a big deal, all we have to do is open up the jhunt-aws.yml file (in
vault-deployments/) in our favorite editor, and add this to the params:
stanza:
params: availabilityzones: [z1] stemcellos: ubuntu-xenial
Changing stemcell_os is also necessary, because (at least as of 1.2.1) the
Vault Kit expects to deploy to the latest Ubuntu Trusty (14.04 LTS), but we
only uploaded Xenial stemcells.
Now we can deploy.
$ genesis deploy jhunt-aws
After the deployment succeeds, we need to initialize the Vault:
$ genesis do jhunt-aws -- init
This sets up a safe target in your local configuration, named jhunt-aws
(after the environment). We are now able to transfer all of our
credentials from the ephemeral init Vault to their forever home.
$ safe -T init export | safe -T jhunt-aws import
It is now safe to kill the safe local process.
Congratulations!
🎉
Hopefully you've had fun setting up this little AWS lab. The BOSH director we deployed has Credhub installed, so you're able to deploy whatever you want to it, with or without the power of Genesis.
If you do think Genesis is awesome sauce, check out the other kits and see if anything tickles your fancy. Official Genesis Kits live on GitHub, in the Genesis Community organization. Notable Kits include:
Concourse - Deploy a CI/CD pipeline and get started with automating your Genesis deployments!
SHIELD - A data protection solution for the cloud. Schedule backups and perform restores on databases, key-value stores, even file systems.
Cloud Foundry - The Cloud Foundry PaaS itself. Now deployed via Genesis.
Blacksmith - Data services, on-demand, leveraging BOSH. Available for CF marketplaces and Kubernetes!