Overview

Cloud Foundry has recently improved support for secure inter-process communications, which has been reflected in the cf-deployment release. This change in v1.8.0 is to better approximate the certificate tree with regards to the CA used to sign each certificate, as well as CN and SANs used. This version also contains future-looking configuration to support extended key usage that is supported but not validated currently in Cloud Foundry.

Impact

The impact of this feature is minimal to the runtime and stability of your Cloud Foundry deployment. It requires rotating the certificates and associated CA certificates. Genesis provides the tooling to do this.

There is no downtime incurred by applying this feature.

The Process

Once you update your environment to use the v1.8.0 kit, you will need to update your certificates. There are two ways to do this:

The Quick and Dirty Method

The fastest way to proceed is to regenerate all your secrets using the genesis rotate-secrets <env> -f command for your target environment after you update its kit version. The -f option is needed to force the regeneration of the CA certs, which are otherwise left unrotated.

However, this has the side effect of rotating ALL your secrets, including the credentials used for passwords. This generally has no impact because these are used internally, but If you have external tools or users that use these credentials to interact with your deployment, you will need to propagate these updates to your users and tools.

IMPORTANT UPDATE/WARNING

There are a handful of secrets that CF cannot handle being changed. These are the following:

• cc:directory_key
• cc:db_encryption_key
• diego/encryption_key:value
• uaa/encryption_key:current
• uaa/jwt_signing_key:private
• uaa/jwt_signing_key:public

If you want to perform the Quick and Dirty method, ensure you have a copy of these secrets and manually restore them after rotating. These will be found in save under the path secret/<env-slug>/cf/ - see below for definition of env-slug.

The In-depth Method

If you'd rather keep your credentials, you can surgically remove just the certificates used and add them back in using genesis add-secrets <env>.

All certificates generated by the cf-genesis-kit are stored under the following path:

secret//cf//certs/

The <env-slug> is your environment file name, but with hyphens converted to slashes. For example, if your environment file was mycorp-aws-uswest1-staging.yml, the path root for this kit would be

secret/mycorp/aws/uswest1/staging/cf/

Using this information, find the certificates using safe and remove them:

pathroot='secret//cf/' # as described above for s in $(safe paths $pathroot | grep '.\/certs\/.') ; do echo "removing $s"; safe rm -f "$s" ; done

Warning: Ctrl-C will not stop that command, so ensure you've correctly entered your path_root.

Once that is complete, update the kit version in your environment file and run genesis add-secrets <env> to build the required certificates. This method is preferred as it clears out all the old defunct certificates that are no longer used, but it doesn't hurt to leave them in.

Verification

As always, you can use genesis check-secrets <env> to verify your secrets have been created.

One Final Note

Genesis currently does not have the ability to have a CA certificate signed by another CA certificate, but this is needed for the diego_instance_identity_ca. To that end, when you first run a command that looks at building a manifest (manifest, deploy, lookup, yamls, etc...), the blueprint hook script with check that the CA is signed correctly, and if not, will renew it with the correct signatory CA. If this happens in a pipeline, it will lack the required permissions to change the secret, so make sure you run genesis manifest <env> prior to triggering your pipeline run.

Help & Support

If you have concerns about the impact of this migration process, or need assistance running through it, please don't hesitate to find us in #help on Slack.