lukegbcom: add a long rambly post about my Vault setup
This commit is contained in:
parent
b238831963
commit
ff665ab50f
1 changed files with 225 additions and 0 deletions
225
web/lukegbcom/posts/2022-04-07-vault-and-me.md
Normal file
225
web/lukegbcom/posts/2022-04-07-vault-and-me.md
Normal file
|
|
@ -0,0 +1,225 @@
|
|||
---
|
||||
title: Vault and Me: Taking It Too Far
|
||||
date: 2022-04-07
|
||||
layout: Post
|
||||
---
|
||||
|
||||
Recently I've been thinking about how I could distribute secrets to my NixOS
|
||||
machines in a... relatively... decent way.
|
||||
|
||||
---
|
||||
|
||||
At the same time, I've been wanting to move to using an SSH CA to issue myself
|
||||
credentials, rather than a variety of SSH public keys.
|
||||
|
||||
In any case, my Vault setup ends up looking like this:
|
||||
|
||||
- Vault instance, running on [Google Cloud Run](https://cloud.google.com/run)
|
||||
with autounsealing from Cloud HSM, backed by Cloud Storage
|
||||
- Vault configuration using [Terranix](https://terranix.org/) (Terraform, but
|
||||
with the config in Nix)
|
||||
- Vault App ID credentials on each machine, with Vault Agent used to
|
||||
automatically auth
|
||||
- `tokend` for issuing service credentials
|
||||
- `secretsmgr` for managing SSH CA host certificates and ACME certificates
|
||||
- `access` for issuing SSH CA user certificates
|
||||
|
||||
Let's go over these one at a time from a relatively low-level perspective - and
|
||||
I'll describe in more detail the mechanics of SSH CAs in a separate post. If
|
||||
you're interested in some [good](https://www.lorier.net/docs/ssh-ca.html)
|
||||
[docs](https://blog.habets.se/2011/07/OpenSSH-certificates.html) on how SSH CAs
|
||||
work, written by some colleagues of mine - then I refer you to those instead,
|
||||
since I'll primarily be focusing on my specific setup with Vault rather than a
|
||||
more general introduction.
|
||||
|
||||
## Vault instance on GCP
|
||||
|
||||
First off: why on GCP when I have a bunch of physical boxen I'm managing?
|
||||
|
||||
The reason is simple: if I have a lot riding on it, I'd rather that it doesn't
|
||||
have the same failure domain as the stuff I'm hosting. It makes it much easier
|
||||
to recover if I don't have to rely on having my own infrastructure up in order
|
||||
to recover it.
|
||||
|
||||
On the other hand... it does mean I'm putting my root of trust "out of my
|
||||
hands". I'll take that risk.
|
||||
|
||||
Broadly speaking, my setup roughly mirrors Kelsey Hightower's [Serverless Vault
|
||||
with Cloud
|
||||
Run](https://github.com/kelseyhightower/serverless-vault-with-cloud-run) -
|
||||
although I build the Docker container [using
|
||||
Nix](https://hg.lukegb.com/lukegb/depot/-/blob/branch/default/nix/docker/vault/default.nix).
|
||||
|
||||
It's a relatively neat setup, although... it turns out to be expensive. Maybe
|
||||
I'll move it to Oracle Cloud's free tier running on one of their ARM64
|
||||
instances.
|
||||
|
||||
### vault-acme
|
||||
|
||||
One important thing to notice is that I install the
|
||||
[`vault-acme`](https://github.com/remilapeyre/vault-acme) secret engine for
|
||||
issuing SSL certificates using ACME.
|
||||
|
||||
This allows me to just store the Let's Encrypt stuff within Vault and not have
|
||||
to distribute my DNS server's credentials to each individual server. I could
|
||||
run a separate service to do this, but it's super convenient to just have Vault
|
||||
do it, since everything already authenticates to it.
|
||||
|
||||
## Vault configuration
|
||||
|
||||
I use Terranix to manage the Vault configuration - this is because I have all
|
||||
my server's configs in my repo, so I can actually introspect the NixOS
|
||||
configuration to determine how I want to build the Vault config.
|
||||
|
||||
I have a helper script called `terraform` that acts like the normal Terraform
|
||||
binary after having compiled my Vault configuration, so I can run `./terraform
|
||||
apply` and have it just work the way you'd expect. At present it requires GCP
|
||||
credentials to be issued separately, using gcloud, since I store my Terraform
|
||||
state in a GCS bucket, but I'm hoping to instead grant access to this using a
|
||||
Vault-managed GCP service account instead (still with the ability to use my
|
||||
"normal" Google account though, if needed, because obviously I need to be able
|
||||
to use it to fix Vault if Vault is broken...)
|
||||
|
||||
In particular, I generate identities per server defined in the config, and
|
||||
provide myself some useful hooks to make "app" policy configuration more easy.
|
||||
|
||||
### What do I mean by an "app"
|
||||
|
||||
My configuration basically defines an app as a separate Unix user; if you are
|
||||
running as a Linux user named `fooservice` on server `barserver` (and the Vault
|
||||
configuration says that `fooservice` is intended to exist on that server), then
|
||||
`tokend` will issue you a Vault token with the policies `app/fooservice` and
|
||||
`server/barserver/app/fooservice`, if those policies exist.
|
||||
|
||||
This is super useful: for instance, in the main case apps are only deployed on
|
||||
one host, and if I'm moving it around then it makes sense for it to have access
|
||||
to the same secrets. I use [Pomerium](https://pomerium.io) as an authenticating
|
||||
proxy, and so there's an `app/pomerium` policy which grants access to secrets
|
||||
like `kv/apps/pomerium`.
|
||||
|
||||
However sometimes there are users which are deployed on more than one machine -
|
||||
such as `gitlab-runner` - and that user should only get access to secrets on
|
||||
one specific host. I use this concept for granting access to `gitlab-runner` on
|
||||
a server called `clouvider-lon01` to be able to deploy to this blog! It [has
|
||||
access](https://hg.lukegb.com/lukegb/depot/-/blob/branch/default/ops/vault/cfg/lukegbcom-deployer.nix)
|
||||
to get an OAuth token to a specific GCP service account with permission to
|
||||
deploy to Firebase Hosting via the `server/clouvider-lon01/app/gitlab-runner`
|
||||
policy, but the `gitlab-runner` user anywhere else is not permitted to get
|
||||
access to this secret.
|
||||
|
||||
### `server-user`
|
||||
|
||||
There are some secrets which aren't super secret and should be generally
|
||||
accessible by users on servers, even if they don't have their own Vault token.
|
||||
`tokend` checks to see if the user talking to it (via a Unix domain socket) is
|
||||
a normal user rather than a service user, and if so will issue a token with the
|
||||
`server-user` policy instead.
|
||||
|
||||
This token really just has a credential to get access to my Nix binary cache on
|
||||
Google Cloud Storage, so it's not super confidential. There aren't really many
|
||||
instances where this is useful, and in general on "client" devices I expect to
|
||||
authenticate to Vault and get a more fully-fledged token as myself.
|
||||
|
||||
It doesn't have access to, for instance, issue SSH user certificates. That
|
||||
power is restricted to "real" authenticated users who have authenticated
|
||||
directly with Vault.
|
||||
|
||||
### `server/hostname`
|
||||
|
||||
Servers are also permitted to have server-wide secrets. This is mostly just
|
||||
used for `secretsmgr` at the moment - arguably this could be its own app.
|
||||
|
||||
By default, servers [have
|
||||
access](https://hg.lukegb.com/lukegb/depot/-/blob/branch/default/ops/vault/cfg/policies/server.hcl)
|
||||
to `kv/server/$HOSTNAME`, and to issue ACME certificates, and the Nix binary
|
||||
cache credentials. They also have the power to issue subtokens with
|
||||
lesser-power than themselves.
|
||||
|
||||
## Vault App ID credentials
|
||||
|
||||
I use the "App ID" mode in Vault to provision secrets to servers; when setting
|
||||
a machine up (a process I have not yet automated), I run
|
||||
[`reissue-secret-id.sh`](https://hg.lukegb.com/lukegb/depot/-/blob/branch/default/ops/vault/reissue-secret-id.sh)
|
||||
which revokes all existing secret IDs for that host and dumps out a Vault
|
||||
[response wrapped
|
||||
token](https://www.vaultproject.io/docs/concepts/response-wrapping), which can
|
||||
be used one time only to get the secret ID for that host.
|
||||
|
||||
There's a
|
||||
[`provision-secret-id`](https://hg.lukegb.com/lukegb/depot/-/blob/branch/default/ops/vault/default.nix)
|
||||
script installed on every machine which will then install the secret for me.
|
||||
|
||||
Future work in this space for me is binding the secret to the TPM (e.g. using
|
||||
mTLS auth) so I don't have to stick the secret ID on disk... but then again I'm
|
||||
not a multinational corporation, and my secrets aren't worth _that_ much.
|
||||
|
||||
## Vault Agent
|
||||
|
||||
The Vault Agent is a daemon that serves a number of purposes:
|
||||
|
||||
- It can act as a proxy which keeps an internal Vault token and automatically
|
||||
refreshes it, and then attaches it to every request that it proxies to Vault
|
||||
as received on a Unix socket (or a TCP socket... but I don't use it like
|
||||
that)
|
||||
- It can use templates to write secrets to disk (with drawbacks, hence the
|
||||
creation of `secretsmgr` below)
|
||||
- Probably some other functionality I don't really use
|
||||
|
||||
In my setup, only a few things have direct access to the Vault Agent socket,
|
||||
and in future I might get rid of it from my setup entirely. `tokend` and
|
||||
`secretsmgr` have access, and that's pretty much it. This is because the powers
|
||||
that its Vault token gets are a combination of all the policies granted to the
|
||||
machine, including all the apps running on it, so any app with access to its
|
||||
Unix socket effectively gets all the secrets shared to anything on the server.
|
||||
|
||||
The secrets I use it to write to disk are strictly the plain KV type, rather
|
||||
than anything more sophisticated, but I do use some [relatively complicated
|
||||
Polkit
|
||||
rules](https://hg.lukegb.com/lukegb/depot/-/blob/branch/default/ops/nixos/lib/vault-agent-secrets.nix)
|
||||
to allow it to reload/restart services when those secrets change.
|
||||
|
||||
|
||||
## `tokend`
|
||||
|
||||
The user-based authentication I mentioned above (with the app policies and the
|
||||
`server-user`) policy is powered by
|
||||
[`tokend`](https://hg.lukegb.com/lukegb/depot/-/tree/branch/default/go/tokend),
|
||||
which is a daemon that listens on a Unix socket and proxies requests through
|
||||
the local Vault Agent, with a token issued that has a subset of the powers of
|
||||
the original server-wide token.
|
||||
|
||||
The ACLs on talking to `tokend` are much more permissive than those for talking
|
||||
directly to the Vault agent.
|
||||
|
||||
## `secretsmgr`
|
||||
|
||||
`secretsmgr` exists to solve some problems I was having with getting Vault
|
||||
Agent to write secrets that require more complex Vault requests, like the TLS
|
||||
certificates using ACME (which have ratelimits imposed by Let's Encrypt!), and
|
||||
SSH host certificates.
|
||||
|
||||
It's a pretty simple binary which runs using a systemd timer unit, starts up,
|
||||
checks the remaining lifetime of the certificates it's responsible for, and
|
||||
then reissues them if required.
|
||||
|
||||
Similar to the Vault Agent above, I use some [Polkit
|
||||
rules](https://hg.lukegb.com/lukegb/depot/-/blob/branch/default/ops/nixos/lib/secretsmgr.nix)
|
||||
to allow it to restart the ACME certificate consumers (usually nginx or
|
||||
pomerium), and sshd.
|
||||
|
||||
## `access`
|
||||
|
||||
`access` checks to see if there's currently an active Vault token. If not, then
|
||||
it launches the `vault login` flow which in my case asks me to log in with my
|
||||
Google account. If that succeeds, or if I already had a token, then it
|
||||
generates a new ED25519 private key and asks Vault to sign it with a lifetime
|
||||
of about 24 hours, and then inserts it into the SSH agent. This means the key
|
||||
never has to hit disk, since it can just reside in the SSH agent.
|
||||
|
||||
The token that this flow issues is a `user` token (i.e. not a `server-user`
|
||||
token, nor an `admin` token), which has permission to look at some specific
|
||||
secrets related to that user, things which are generally shared -- like those
|
||||
Nix binary cache credentials, but doesn't have general access to administrate
|
||||
Vault. I issue `admin` tokens with a lifetime of 1h when I need them, and at
|
||||
some point will try to scope them down further -- although since I use them to
|
||||
deploy all the policies there is a limit to what I can feasibly do.
|
||||
Loading…
Reference in a new issue