What is Vault?
Vault is a service developed by the fine engineers at Hashicorp, which is designed to safely secure any data you want: passwords, SSL certificates, application tokens, etc. Aptly named, it can be sealed by any authorized user, but can only be unsealed with the use of n keys (3 by default), so in the event of a breach a single bad actor can’t gain access to the system.
Once Vault is unsealed, it still requires an access token to perform CRUD operations, all of which are logged. Permissions can be granted at an incredibly granular level, and it can be deployed in High Availability mode so nodes in the system can go down without disrupting services.
Our Original Approach
Before we used Vault, we had all our configuration in a private Git repo which contained all our secret keys. This is bad because any user who has access to that repo can now get full access to databases without any audit logs to prove who made the access. This is a stage that most startups go through. When your goal is to get a product out the door, there’s no sense worrying about the architecture of security.
When we started getting customers, it was time to get more serious about security. We started off by putting all our keys into Vault so they were in one configured location. Whenever a service was deployed, it’d pull the latest values from Vault and we’d have audit logs of who accessed which database passwords, etc. We were literally using Vault as a secured and audited key-value store. This was an improvement over our previous approach, but came with its own set of issues.
Running Into Issues
After the initial implementation of simply storing key-value pairs we ran into some core issues with our credentials. Firstly, when our SSL certificates changed, or we needed to change important database passwords, we still had to redeploy all the services which used those credentials. Sometimes this resulted in short downtimes as we would roll over to new systems.
Second, since we still only had one username/password combination to access our database, shared by multiple systems, it was very hard for us to track access, and we had no solution to quickly revoke access to a single compromised system.
Another issue came from how we used Vault’s secret backend for AWS. We needed an AWS user who could only access a specific path in an S3 bucket for customer code. But our initial implementation was to generate keys that would expire in a year, thinking no customer server would run for longer than that due to security updates and patches. This meant that if someone gained access to that token, they could wreak havoc on that bucket’s path for a long time without being detected.
Rethinking Vault
As we started getting more and more customers on our system, we started thinking differently about Vault. Instead of using it as a key-value store, we started using it as a token generator which gives you access to short-lived passwords.
We’ve been making all our password leases as short as possible, and implementing code to renew those leases as needed. Each application gets its own Vault token to generate the credentials it needs. When a system needs to connect to MongoDB, it’s given a token which can generate MongoDB usernames and passwords using the MongoDB secret backend. The same pattern matches for Postgres, AWS and Consul. But we don’t just stop there; each one-off script generates short-lived tokens for their duration, meaning credentials live for exactly as long as they need to.
Reaping the Benefits
Short-lived tokens greatly reduce the attack surface of our codebase, and since each application has a unique identifier to access Vault, we can track who requests access for what. Not only that, but the password is validated often, meaning we can revoke access to any suspicious applications easily. Perhaps the best part of this strategy is the ability to easily roll credentials that can’t be automatically generated by Vault, like SSL certificates. Since services can regularly check Vault for new credentials, we can update them manually and put them in Vault, knowing they’ll be picked up soon.
If it’s possible, have Vault generate all your credentials for your services, so you can get unique credentials per application and track the entire access to the originating party. Don’t be shy about creating access tokens; each service and administrator should get their own. And remember to store your personal tokens in a password manager. Vault is not just a key-value store, it’s a way of generating your secrets, and most importantly a way of thinking.