Supports pages to ci.yml for better pipeline customizations. These pages can be either overlay-style (spruce) or go-patch style. These extra pages will be merged over the pipeline generated by Genesis.
Because concourse syntax allows non-homogeneous structure, you can't always use spruce operators to overlay the right location (ie adding an additional param to a specific task config) In these cases, you can use go-patch.
Example: ```
pipeline: <... all the normal ci.yml configuration values ...>
type: replace
path: /jobs/name=
```
Note that the first page must be in overlay style, not go-patch, because it must stand as a valid yaml hash on its own.
For vaults that have different secrets mount, you can set VAULTSECRETSMOUNT in the pipeline job to specify an alternative location where the secrets are found. This fixes the issue where the pipeline can't authenticate to vault because it can't find handshake secret under /secrets/
Pipeline fixes to authenticate to the vault when running errands
Pipeline jobs can now use email for status notifications.
Pipeline can now specify an alternative image registry for resources and workers, for supporting closed networks. Supports both insecure and authentication by username/password.
Clarified create-kit help description regarding the -n option to indicate that it automatically adds "-genesis-kit" suffix
The generated scaffolding now has the base manifest default to use bionic stemcells
genesis bosh now supports all the basic options, including
--[no-]color, -q|--quiet, -D|--debug, -T|--trace, and -S|--show-stack.
Add the yaml files used during the check process of deployment. This is especially useful when running deployments in pipelines, and when running the spec tests. To enable it, set the environment variable GENESISCHECKYAMLONDEPLOY to a truthy value (1, yes, true, Y) and export it (or set it in params section of the deploy task).
Add pre-deploy and post-deploy reactions on a per-environment basis.
Sometimes different environments need to perform tasks prior to and after doing a deploy. This feature allows you to specify scripts that will be run in those circumstances. Add the following structure to your environment file:
genesis:
reactions:
pre-deploy:
- script: put-up-maintenance-page
- script: update-jira
args: [ 'some-argument', '$SOME_ENV_VAR' ]
post-deploy:
- addon: valid-addon-for-kit
- script: remove-maintenance-page
The scripts are located in the bin/ dir under the repository root directory, and are propagated via the pipeline cache system.
The scripts have access to the following environment variables:
GENESISPREDEPLOYDATAFILE -- file path that contains any data gathered by the predeploy hook
GENESISMANIFESTFILE -- file path to the full unredacted unpruned manifest for the current deployment
GENESISBOSHVARSFILE -- file path to any BOSH variables for the deployment
GENESISDEPLOYOPTIONS -- JSON representation of the options passed to the deploy call
GENESISDEPLOYDRYRUN -- "true" if the deployment is a dry-run, "false" otherwise
GENESISDEPLOYRC -- return code of the BOSH deploy call. 0 if successful, 1 otherwise. Only available for post-deploy reactions
You can now specify where your alternative BOSH director's credentials can be found.
By convention, your BOSH environment (the director you are using to
deploy your environment) is the same name as the environment you are
deploying, and its credentials are stored in exodus in the same
vault.
This isn't possible when what you're deploying is another BOSH
director, so Genesis has always made it possible to specify another
BOSH environment alias, via the genesis.bosh_env parameter, and
before that, via the params.bosh in the bosh kit.
Furthermore, if you deployed your BOSH director with a non-standard
kit, you could specify that deployment-type as such:
<alias>/<deployment-type>. By default, that deployment type is
bosh
When v2.8.0 came around, we started storing access for the credentials
to the BOSH director in vault. While we always recommend using a
single vault for all deployments in your deploy repo, some companies
have policies that don't allow that. To this end, you can now specify
which vault and exodus base path under which your BOSH director
credentials can be found. This extends genesis.bosh_env with the
following pattern:
<alias>[/<deployment-type>][@[http(s?)://<host>[:<port]/]<base-path>]
We still recommend a single vault and single exodus base path whenever possible, as most of the existing kits that use exodus data do not have the ability to specify an alternative vault or path. This simply allows the layering of nested BOSHes to work within the confines of corporate security policies.
Adds a --fix-releases option to deployment for when you need to re-upload
a bosh release. This is rarely needed, mainly if you change the underlying
stemcell to a new OS.
Support --recreate-persistent-disks on deploy for both deploy and create-env environments.
genesis decompile-kit will now accept file with .yml suffix
Added printaddondescriptions and runextendedaddon helpers to support extended addon scripts.
Rather than have all the addons in the hooks/addon file, you can add
individual hook/addon-<name>~<shortcut>. To do so, you need to
follow the following conventions:
The script gets passed as its first argument the action to take. This is either help or run. Help prints out the description to be used in list, and run causes the script to run.
The main hooks/addon script must use printaddondescriptions in its list command block, with alternative arguments of the base commands and descriptions. Example:
list) print_addon_descriptions \
"command" "description of command" \
"another" "description of another command" ;;
*) block.You can now find more genesis environment information in the manifest.
In addition to the existing keys under genesis., you can now find
the vaultified environment (dashes into slashes) in
genesis.vault_env, and the full path for the secrets and exodus
base: genesis.secrets_base and genesis.exodus_base respectively.
genesis.secrets_base is the same value as meta.vault but with the
trailing /.
When using legacy secrets provider, if the current safe target has the same URL as another safe target, a confusing error would be printed. This change provides a much more informative error message.
When using a specified secrets provider for a deployment repo, a more instructive error message is provided when multiple safe targets with the same URL is specified.
Genesis doesn't like the https_proxy or HTTPS_PROXY environment
variables being set, but BOSH was happy when they were set to "".
However, upstream BOSH now considers that an error, so Genesis is
going to simply unset them.
If you need to use HTTPS proxy, you can still set GENESIS_HONOR_ENV
to a truthy value, and it will keep HTTPS_PROXY and https_proxy
environment variables from the calling scope, but only do this if
you encounter an error.
No longer errors when running genesis version if prerequisites are
missing (the lone exception of course is if perl itself is missing)
Clearing of the secrets provider (genesis secrets-provider --clear)
now works. This puts genesis in "Legacy Mode", meaning it will use
whatever your current safe target is as the source for your genesis
secrets. Not recommended, but there are specific use cases that
required it (for now).
Support --recreate for create-env environments. This was already supported for deploy-based environments, but was erroneously denied for create-env based environments (ie proto-bosh)
Added genesis lookup --env to retrieve environment variable values
used by Genesis for the specified environment. Like all lookup
commands, '.' will return everything in JSON format, single value will
be returned as an unquoted string, and default if specified, will be
returned if environment variable specified doesn't exist.
Added --no-bosh and --no-vault to genesis env-shell command, so
that you can access a shell without forcing connections to those
services.
Fixed output of genesis env-shell so that the preamble header
doesn't go to stdout. This better allows running a user-provided
executable as a shell (--shell path/to/executable option) to run
arbitrary commands inside "Genesis-space", then capturing its output.
Added -p|--pre flag to genesis update to allow checking for or
upgrading to prereleases.
genesis decompile-kit can now take just the kit name or version.
Specifying the name will decompile the latest local version, and
specifying the version doesn't need the kit name if its unambiguous.
The version can handle the v prefix in genesis decompile-kit
You can specify latest as the argument if only one kit type is in
use (which is the way Genesis is designed to function)
Show the version of Genesis when doing a deploy; helps with diagnoses when user submits deploy output when encountering an error.
Ignore any 'v' prefix when specifying the version in genesis
compile-kit
Fixes issues encountered when creating new environments based on the
bosh Genesis kit. This also removed the confusing bosh_env: --
entry under genesis in environment yml files.
Future-proofing for creating new environments based on
bosh-genesis-kit v3.0.0 (--[no-]create-env option to genesis new)
Improved error messages for create-env/bosh_env conflicts.
genesis new now supports environment-first call style, allowing
users to type genesis <env> new ... to create the new environment.
Fixes another issue with genesis new when genesis bosh --connect
has been used in the terminal session.
genesis new will explicitly prune genesis.bosh_env when making a
new create-env based (aka proto) environment.
When preparing exodus data after deploy, properly handle undefined values.
The genesis bosh command now runs in the users calling directory, so
relative paths to files work as expected.
Fixed bug where some systems could not validate SSH keys due to permission issues with here-strings.
This is a prerelease - please see commit messages for changes
Download v2.8.3-rc.4 from GithubCommand line can now take the path/env before the subcommand, without using -C.
Command line can now take the enviornment before or after the command, and no longer needs the -C. Moving forward in 3.0, the preference will be prefix instead of postfix, and the -C option will be dropped.
Unlike the postfix method, the prefix method can take a path component.
Unlike the -C option, the .yml doesn't need to be included (unless it
is ambiguous, such as your file is named the same as a genesis command).
Before:
genesis -C ../alt/path deploy my-env
Now:
genesis ../alt/path/my-env deploy
Before:
genesis -C deploy-dir/env.yml deploy
Now:
genesis deploy-dir/env deploy
User is now notified if the local kit version has changed from what
was deployed when running genesis info
With the changes in 2.8.2, a quazi-regression was introduced regarding BOSH targeting. If the BOSH environment variables were set in an environment, Genesis would use those to target the BOSH director instead of what was specified in the environment file.
The target BOSH and the manner in which it is targeted has been in flux for a number of years. Genesis 2.8.x was meant to unify this, but there are multiple edge cases that we've worked through, and a lot of habits that need to be unlearned. The following is the intended best practices, that if followed, will keep you out of trouble.
genesis bosh <env> <bosh opts and args> to
interact with the BOSH deployment associated with UNLEARN:
Genesis is 'deployment-focused', while the BOSH cli is 'director-focused'. When you target a Genesis environment file, you need to stop thinking about which director you are using, but which deployment you care about; Genesis will figure out the associated BOSH director to talk to. One of the reasons for this paradigm change is so users that have access to a deployment's env file, but not the corresponding bosh deployment's repo can still access the BOSH director needed to manage their deployment.
Example: genesis bosh -A <env> <bosh opts and args>
If you need to use the BOSH cli directly, you have two choices, each somes with side effects:
Set the environment variables to connect to bosh. This is what Genesis does internally, but if you do this, be aware that it lasts for the lifetime of the shell you are in, which can lead to unexpected side-effects. You can do this by running
eval "$(genesis bosh --connnect [-A] <env>)"
where the optional -A specifies use
You can also use the bosh-genesis-kit's printenv addon, if you
have access to that deployment environment file. While it is more
powerful, including support for socks5 proxy, it also impacts the
shell more. See the bosh kit for more details.
Use BOSH alias configuration and log in (daily) using the bosh deployment configuration. This is the method Genesis used prior to v2.8.0, and still works if you're using bosh-genesis-kit v2.2.0 or greater, and bosh-cli 6.4.4 or greater. This has the advantage of needing to be explicitly stated with the -e option on the bosh call, and allowing multiple bosh directors to be authenticated against at the same time. The down side is you'll need to re-authenticate periodically.
NOTE: If you have the BOSH environment variables set, Genesis will now ignore them when operating on a genesis deployment environment, but will issue a warning that it is doing so. The environment's explicitly or implicitly specified bosh environment will be used.
Fixes outdated references to pre-2.8 BOSH library calls.
Resolves runtime error message when not specifying kit on genesis init
Resolves runtime error message for unspecified use_create_env property (2.8.0 feature)
Correctly allows recreation of existing kits with --force option in compile-kit
Fixes some internal kit hook bash wrappers around genesis callbacks
Fixes race condition when creating a new bosh enviornment where it doesn't know the bosh director or if it is a proto bosh or not before it asks for those values.
Reverts a change in v2.8.2, which would set the enviornment variable used internally and by the hooks to use the BOSH direcctor identified by BOSHENVIRONMENT. The normal environment variable to do this is GENESISBOSHENVIRONMENT, but the previous change erroneously also included BOSHENVIRONMENT as a source for setting the BOSH director.
Fixed bug Can't locate object method "base_url" via package "Genesis::Kit::Provider::GenesisCommunity"
that occurred when genesis repipe was run
Replaces error Can't call method "execute" on an undefined value
with more useful information on why BOSH director cannot be accessed.
Suppress repetitive deprecation messages for the environment when processing various genesis commands.
The existing bosh wrapper function in Genesis::Helpers was not
compatible with the Genesis v2.8 internals regarding BOSH connectivity.
If the environment file exists, helpers will simply use genesis bosh
... calls to make it Just Workâ˘. For the rare situation where the
enviroment file doesn't exist, a much more complicated validation and
connectivity routine is employed.
When generating the BOSH director for an environment, and is using the environment variables to determine the BOSH director configuration, make sure to set the deployment name.
Added version_check bash function to check that the version of Genesis
meets or exceeds your hook's required version. If you use this, make sure
your kit specifies genesis_version_minimum of 2.8.2.
Improvements to genesis compile-kit interface and functionality:
Now creates a local rc release when no version is specified, for the next patch level above the highest remote or local version found.
Can specify --target|-t to have the compiled kit placed in the desired directory. If the directory is a genesis repo, it will place it under the directory's .genesis/kits path.
When using -v
When not specifying the explicit version, you can bump the major or minor version instead of the path by using --major|-M or --minor|-m options. The --final option will create the next version without a -rc# component.
Improved feedback while processing.
Genesis v2.8.0 reworked how Genesis interacted with BOSH, getting the login information from exodus data. However, when the exodus data is not available, it lost its ability to specify the BOSH connectivity via environment variables. This has now been resolved.
Fixes error Can't locate object method "from_envvars" via package
"Genesis::BOSH::Director" when generating new genesis environment.
Fixes error when running genesis bosh -h, documents how to get help
for bosh commands instead of help for genesis bosh
Auto-connect to BOSH director.
Genesis can now automatically connect to the correct BOSH director for the specified environment, without need to set up a bosh configuration or manually logging into the director.
Caveat: The BOSH director that deployed your environment MUST have been deployed by Genesis. If not, you will still have to target and log into the BOSH director manually.
Added genesis bosh command.
Instead of worrying about logging into the right BOSH director, and translating genesis environment into BOSH environment and deployment, let Genesis do the heavy lifting.
If you want to know something about your concourse vms, for example, you
just run genesis bosh my-concourse-env vms --vitals and you'll get the
right output. Genesis will determine which bosh director deployed
my-concourse-env, how to connect to it, what the BOSH deployment is named,
and return the results of the command you specified.
By default, the genesis bosh command is deployment centric, but sometimes
you need to talk to the BOSH director itself. In that case, you can specify
the BOSH deployment itself, and use the -as-director (or -A) option to
indicate that the environment is the director, not a deployment on a
different director. For proto-BOSH environments, this is required.
See genesis bosh -h for more usage details.
Connect to BOSH director without using the BOSH deployment kit.
If you still need to connect to a BOSH director directly, you can configure
your shell by using eval "$(genesis bosh my-target-env --connect)".
Please note that this connects to the BOSH director that deployed the
my-target-env. If my-target-env is a bosh deployment, the command will
authenticate to the BOSH director that deployed it, not itself. In this
case, you need to use the --as-director option to explicitly state that
this is the BOSH environment you wish to connect to.
Removes legacy support for kits without new or blueprint.
As of v2.8.0, kits must use new and blueprint hooks. All modern kits support this model, so anyone actively keeping current with genesis would not be impacted.
Remove subkits, error if used in env file.
Subkits have been deprecated since at least 2.6.13, so its time to remove them.
Remove --environment global option.
The environment option has been replaced with --bosh-env option to remove
confusion regarding the overloaded 'environment' name. Henceforth,
environment refers to just the deployment environment. The -e shortform
is still usable.
Starting in v2.8.0, kits that indicate that they are compatible with
v2.8.0 can support use_create_env as their method of indicating
their ability to use create-env instead of deploy for deployments.
Valid values are yes for this kit always uses create-env, no for
kits that cannot use create-env, and allow for kits that permit
either.
The kit will have to detect genesis.use_create_env in the
environment to determine blueprint and other create-env kit
behaviour. Genesis will manage the actual deployment and state file
associated with it.
Genesis properly handles both legacy features (create-env,bosh-init,
proto) and v2.8.0's genesis.use_create_env attribute to determine is
create-env is to be used. It will also detect when an incompatible
mixture of using create-env and specifying a genesis.bosh-env
attribute, and fail with an appropriate error message. A quick update
of the environment file by the user will resolve this issue when
encountered.
Since BOSH does not provide a confirmation after a diff on the environment to be deployed, Genesis provides this based on a previous deployment, assuming it was added to the repo.
If the exodus data from the previous deployment doesn't match the local
archived manifest, a warning will be stated, or in the case of running
under -y, the deploy will be aborted.
This does require that you ensure you commit your repo after doing deployments, which you should be doing regardless of this feature, as a method to record your deployment history.
```
And in an inheritting file:
``` kit: overrides:
Improve params handling for use in new hook.
Support multiline values in keys and arrays in param_entry.
Echo param_comment to screen so the same information that prefaces the
value in the env.yml file can be used as an explanatory blurb prior to the
prompt.
Improve decompile-kit
Can decompile kit to directories other than ./dev (--directory <dir>)
Can specify an environment YAML file to identify which kit to decompile, instead of explicitly stating the kit/version.
Support downloading multiple configs.
Before, if you specified to download a config type (ie cloud or runtime)
without a name, it would download only the unnamed default config of that
type. Experience has shown that the more correct interpretation should be
to download all configs of that type, and merge them (in the order they are
specified in the BOSH director). The old behaviour can still be used by
explicitly stating default as the name.
Added stderr output capture for Genesis::Vault#get.
Rather than have the raw safe stderr leak out to the user, we now capture it and write it to the DEBUG stream.
Improve deploy help output for create-env environments
There are difference behaviours for create-env environment than from environments deployed via a BOSH director. This update calls them out and gives instructions on their use.
Improved error message when kit.features is incorrectly formed in environment YAML files.
Better diagnosis messages when running PING in debug mode.
Kits can now specify required prerequisites via hook.
Although the prereqs hook was documented, it was only partially
implemented. This commit completes that implementation, so that if
hooks/prereqs exists in a kit, it will be run prior to usage.
Fix bug where default config type was lost when fetching configs from BOSH director, resulting in bad messages and error reports.
Export GENESIS_EXODUS_MOUNT to hooks environment. This was missing from the
set of vault path environment variables available to hook scripts.
Add Genesis::BOSH for controlling BOSH director.
This is a major refactor to facilitate better operation and connections to BOSH directors. Connections to Bosh will use exodus data to determine the corresponding BOSH director address and connection details.
Set GENESIS_BOSH_COMMAND when checking BOSH cmd
Also removes check_bosh_version subroutine that was replaced by
Genesis::BOSH#command
Change needs_bosh_create_env to use_create_env
Allow Genesis::Kit#metadata to take args.
If Genesis::Kit#metadata receives no arguments, it will return a hash reference for all metadata for the kit. If it gets a single value, it returns the metadata corresponding to the key of that value. Finally, if it gets a list of values, it will return a list of the metadata corresponding to the keys of the given values.
Lazy-load kit provider types as needed.
By using require instead of use for loading the
Genesis::Kit::Provider::* classes, they are loaded as needed during
runtime instead of fully loaded at compile-time.
Memoize Genesis::Env params and actual_environment_files.
Reordered lib/Genesis/Env.pm, with folding and grouping.
This is done to be consistent with the other lib files. No actual code changed in this release, but the diff is massive because blocks were moved into logical groupings.
Privatized Genesis::Env#validate_name.
validate_name method should only be used internally, so it was renamed
_validate_env_name.
Fixes inconsistant private variable naming.
_configs has been changed to __configs in Genesis::Env, to match all
other private instance variables.Update Genesis bosh activities to work with BOSH cli v6.4.4
As of BOSH cli 6.4.4, BOSH_* variables set to "" cause errors, so now they are simply unset.
genesis init can now accept a local compiled kit file as an argument to the -k option, allowing locally compiled (or stored) released kit files instead of relying on the local uncompiled dev directory.genesis kit-manual now works on dev kits, or environments that use dev
kitsAdd update genesis command to get new version
Usage: genesis update [-c|--check] [-v|--version VERSION] [-d|--details] [-f|--force]
-c, --check Check if a newer version (or the specified version) is available.
-v, --version
-d, --details Print the release notes of the versions between the current version and the latest (or the release notes of the specified version)
-f, --force Replace the current genesis executable. Otherwise, confirmation will be requested.
Improve genesis kit-provider -v status output
Improve genesis list-kits hint to include -r flag.
Make it easier to trace the stack
Adds the -S|--show-stack debug option, and gives better context of traces and errors.
Behaviour Changes:
internal trace output will show the location in the code were it was
called. If --show-stack option specified, it will show the full stack
instead of just the location.
internal dump_vars output will now show the full stack instead of
just the calling location if --show-stack specified (along with
--trace or --debug)
internal bail will show calling location if --trace or --debug is
specified, or the full stack if --show-stack is specified (with or
without --debug or --trace). This allows you to see where genesis
terminates without wading through all the trace output.
internal bug will always full stack where the bug was encountered.
This allows the user to paste the stack into a defect report against
the genesis project.
Limited false positives of detected changes in pipeline notify summaries.
Fix getting non-production github release version
If user specified version by name, it should include draft and prereleases when looking for it. This was the initial intent, but the wrong options were set when attempting it.
Fix error when calling genesis man on a kit that does not exist.
Fixes issue where proto-bosh deployment did not have up to date state file.
-Can now update subject, SAN, and usage of vault-stored certificates without generating a new key with genesis rotate-secrets --renew This allows users to correct warnings that show up in the check during deployment without breaking mutual TLS. (Note this will also renew the TTL for the certificates)
Download v2.7.31 from Github