Every production application depends on secrets. Database passwords, API keys, OAuth client secrets, encryption keys, webhook signing tokens. These credentials are the keys to your kingdom, and mismanaging them is one of the fastest ways to end up in a breach notification headline.
We have audited dozens of codebases and production environments for credential hygiene. The patterns we see are remarkably consistent: secrets hardcoded in source files, .env files committed to version control, API keys shared in Slack threads, and credentials that have not been rotated since the company was founded. Each of these is a ticking time bomb.
Here is how to manage secrets properly, from local development through production deployment.
The Mistakes That Get Teams Into Trouble
Hardcoding secrets in source code. This is the most common and most dangerous pattern. A database connection string with the password inline, an API key assigned to a constant, a JWT signing secret defined in a config file that gets committed to Git. Once a secret is in your repository history, it is there forever, even if you delete it in a later commit. Anyone with read access to the repository, including every engineer who ever had access and every tool with repository permissions, can find it.
Committing .env files. Environment variable files are a reasonable way to manage secrets locally, but they must never be committed to version control. We routinely find .env files in repositories because the .gitignore was not set up correctly or because someone committed it before the ignore rule was added. A single committed .env file can expose every credential your application uses.
Sharing credentials in Slack or email. When a new engineer joins the team and needs API keys, someone pastes them in a direct message. That message is now stored in Slack's servers, searchable by anyone in the workspace, and retained according to your Slack plan's retention policy. If any account in your workspace is compromised, every secret shared this way is exposed.
Never rotating credentials. Many teams create API keys during initial setup and never change them. If a key is compromised silently, through a leaked log file, a departed employee, or a third party breach, the attacker has indefinite access. Rotation limits the blast radius of any single compromise.
For a broader look at the security practices that prevent these issues, see our web app security checklist.
Environment Variables vs. Secret Managers
Environment variables are the baseline approach to keeping secrets out of code. Instead of hardcoding a database password, you reference process.env.DATABASE_PASSWORD and inject the value at runtime. This is a significant improvement over hardcoding, but environment variables have real limitations.
Environment variables are stored in plain text on the host. Anyone with SSH access to your server can read them. They show up in process listings, crash dumps, and debug logs. They do not support access controls, audit trails, or automatic rotation.
Secret managers solve these problems. Tools like AWS Secrets Manager, HashiCorp Vault, and Doppler store credentials in encrypted storage with fine grained access controls. They provide audit logs showing who accessed which secret and when. They support automatic rotation, versioning, and integration with your deployment pipeline.
AWS Secrets Manager is the natural choice if you are already on AWS. It integrates directly with RDS for automatic database password rotation, supports cross account access, and charges per secret per month. For teams running significant AWS infrastructure, it is the path of least resistance.
HashiCorp Vault is more powerful and more complex. It supports dynamic secrets (generating short lived credentials on demand), encryption as a service, and pluggable authentication backends. Vault is the right choice for larger organizations with dedicated platform teams, but it is overkill for most startups.
Doppler is a newer entrant that focuses on developer experience. It syncs secrets across environments, integrates with most CI/CD platforms, and provides a clean dashboard for managing credentials across projects. For teams that want managed secrets without operating their own Vault cluster, Doppler is worth evaluating.
For most startups, we recommend starting with your cloud provider's native secret manager and graduating to Vault only when your scale and compliance requirements demand it. This decision fits into the broader cloud and DevOps strategy for your infrastructure.
Rotation Strategies That Actually Work
Knowing you should rotate secrets and actually doing it are different things. Rotation is painful when it requires manual coordination, so the goal is to automate as much as possible.
Database credentials are the easiest to automate. AWS Secrets Manager can rotate RDS passwords on a schedule without any application changes if you fetch the password from Secrets Manager at connection time rather than caching it in an environment variable. The rotation creates a new password, updates the database, and stores the new value, all without downtime.
API keys for third party services are harder because each provider has different rotation procedures. The general pattern is: create a new key, update your application to use it, verify the new key works, then revoke the old key. Automate this with a script that runs on a schedule or a rotation Lambda function. For services that support multiple active keys simultaneously, you can do this with zero downtime.
JWT signing secrets require special care. If you rotate the signing secret, every existing token signed with the old secret becomes invalid. The solution is to support multiple signing keys simultaneously: sign new tokens with the new key, but continue accepting tokens signed with the old key until they expire naturally. Use a JWKS (JSON Web Key Set) endpoint so consuming services always have the current key set.
Encryption keys should be rotated on a schedule, but the old keys must be retained to decrypt data that was encrypted with them. Key management services like AWS KMS handle this automatically with key versioning.
A good target is 90 day rotation for most credentials and 30 day rotation for high privilege credentials like production database passwords and admin API keys. Any rotation is better than none, so start somewhere and tighten the schedule over time.
CI/CD Pipeline Secrets
Your CI/CD pipeline needs credentials to deploy code, run database migrations, push container images, and interact with cloud services. These pipeline secrets are high value targets because they typically have broad access to production systems.
Never store secrets in pipeline configuration files. GitHub Actions workflows, GitLab CI files, and Jenkinsfiles are committed to version control. Use your platform's secret storage instead: GitHub Actions Secrets, GitLab CI/CD Variables (marked as protected and masked), or your CI provider's equivalent.
Scope secrets to the minimum required access. If a deployment pipeline only needs to push to a specific ECR repository and update a specific ECS service, create an IAM role with exactly those permissions. Do not reuse a broad admin credential because it is easier.
Use OIDC federation when possible. Modern CI/CD platforms support OpenID Connect federation with cloud providers. GitHub Actions can assume an AWS IAM role directly without storing any long lived credentials. This eliminates the risk of leaked CI/CD secrets entirely and is the approach we recommend for all new pipelines. Our guide on setting up a CI/CD pipeline covers this pattern in detail.
Mask secrets in pipeline logs. Most CI/CD platforms automatically mask values marked as secrets, but verify this is working. We have seen cases where secrets appear in logs because they were passed through a command that reformatted the value in a way the masking engine did not recognize.
Runtime Injection Patterns
How secrets reach your running application matters as much as where they are stored. The two main patterns are environment injection and runtime fetching.
Environment injection means secrets are set as environment variables when the container or process starts. The deployment system fetches secrets from the secret manager and passes them to the runtime. This is simple and works with any application framework, but it means secrets are static for the lifetime of the process. If a secret is rotated, you need to restart the application.
Runtime fetching means the application queries the secret manager directly when it needs a credential. This supports automatic rotation without restarts and allows the application to handle secret manager failures gracefully. The tradeoff is added complexity and a runtime dependency on the secret manager's availability.
For most applications, environment injection with periodic restarts is the pragmatic choice. Combine it with a deployment pipeline that fetches fresh secrets on every deploy, and you get rotation support with minimal application code changes.
Auditing for Leaked Secrets
Even with good practices in place, secrets can leak through code commits, log files, error reports, or screenshot sharing. Regular auditing catches leaks before attackers do.
Scan your repository history. Tools like truffleHog, git-secrets, and GitHub's built in secret scanning can detect credentials in your commit history. Run these as part of your CI pipeline to catch new leaks, and run a one time full history scan to find old ones.
Scan your logs. Search your application logs, cloud provider logs, and error tracking services for patterns that look like credentials. API keys, connection strings, and tokens have recognizable formats. Set up alerts for these patterns so you catch leaks in real time.
Review access patterns. If your secret manager provides audit logs, review them periodically. Look for unexpected access, access from unusual IP addresses, or access to secrets that should not be needed by the accessing service. These patterns can indicate compromised credentials or misconfigured services.
Check public exposure. Services like GitHub's secret scanning partner program automatically notify you if your credentials appear in public repositories. AWS also monitors for exposed access keys and will email you if your keys appear publicly. Enable these notifications and treat every alert as urgent.
When building your system architecture, secrets management should be a first class concern from day one, not an afterthought bolted on before a compliance audit.
Start Today, Not After the Breach
The best time to implement proper secrets management was when you wrote your first line of code. The second best time is now. Start with the highest impact changes: add .env to your .gitignore, move hardcoded secrets to environment variables, and scan your repository history for leaked credentials. Then graduate to a proper secret manager, set up rotation, and build secret scanning into your CI pipeline.
Every week you operate with secrets in source code or credentials that have never been rotated is a week where a single repository compromise or employee departure could expose your entire production environment. The engineering investment to fix this is measured in days, not months. The cost of getting it wrong is measured in breach notifications and lost customer trust.
If you need help auditing your current credential practices or setting up a secrets management infrastructure that scales with your team, reach out to us. We will assess where you stand, close the gaps, and build a system that keeps your credentials safe as your engineering organization grows.