The Ultimate Guide to Storing Secrets in Bitbucket
Are you wondering what's the best way to manage your Bitbucket secrets?
Let's set the record straight, secrets do not belong in git repositories.
Lucky for us, Bitbucket has a native solution to storing secrets so that there is separation between code and secrets.
This ultimate Bitbucket secrets guide covers:
- High level information about secret management
- Tutorial on storing secrets in Bitbucket Cloud
- A tool that eliminates secrets in code
Developers often use third-party APIs to speed up development and avoid reinventing the wheel. However, many developers hardcode sensitive information, which has become a common way for attackers to exploit vulnerabilities.
Let's dive into secret management in Bitbucket and securing your code from exposing sensitive information.
The Importance of Secure Secret Management
Poorly managed secrets are a problem for small and large businesses.
Whether you are Uber (who was recently hacked and the attacker found credentials in their source code and did more damage) or you are a small business, don't underestimate the importance of keeping secrets out of code.
Even if your repository is private, there is an unnecessary risk of an attacker working their way into the network, then to the codebase, and finding sensitive data to steal additional information.
Use any search tool that searches Bitbucket repositories and you’ll find numerous repositories with hardcoded secrets. Attackers might search specific variables common with infrastructure such as AWS credentials or variables with a specific name (e.g., “password” in the variable name). It’s not uncommon for developers to copy and paste variable names from API documentation, so a quick search for these variable names could result in discovery of numerous enterprise credentials and keys.
Even knowing that keys and credentials shouldn’t be stored in code, an occasional mistake happens. Developers might temporarily store secrets in code for quick troubleshooting or to test a third-party API, but then they forget to remove the hardcoded credentials before committing code changes leading to future security issues.
Whether it’s deliberate or accidental, it’s important to monitor your Bitbucket repository pipelines for any secrets.
The Wrong Way to Secure Secrets
Git repositories solutions such as Bitbucket are relatively new compared to the age of some enterprise strategies. Because secrets management is somewhat a new issue, developers might come up with their own ways to monitor secrets.
They might manually review code to detect secrets in code or delete them after secrets are already uploaded to a repository. With several developers on a team, these strategies become unmanageable.
Scanning for secrets after code is deployed to Bitbucket adds a layer of security, but by then the secrets are already compromised. Archives might still contain secrets, any insecure backups could contain secrets, and attackers have scanners that run continuously to monitor specific repositories when code is committed, meaning secrets are exposed immediately after commits.
Once secrets are committed to a repository, they must be revoked and renewed. It’s possible that an attacker could obtain exposed secrets within minutes of them being committed to a repository, but some teams might think they’ve deleted the secrets quickly enough to avoid a compromise.
The right way to secure secrets is to stop them from being exposed before being committed and integrate automated scanners into the software developer lifecycle.
Integrating tools such as Soteri Bitbucket Security Scanning will eliminate these mistakes and find sensitive data before it's exposed.
Note: The tool is available for Bitbucket Server and Data Center. A Cloud version is coming soon.
The Right Way to Secure Secrets
Organizations and their developers can take a few general steps to secure secrets.
You can train developers to avoid hardcoding secrets, but it shouldn’t be the only security measure. Security requires layers of defenses, and monitoring for secrets in code is necessary to provide a failsafe in case developers accidentally hardcode secrets and forget to remove them before deploying code.
Monitoring for secrets in code as it passes through your Bitbucket pipeline is imperative.
Most DevOps teams have several developers, operations people, and automation tools moving code through the Bitbucket pipeline several times a day. Monitoring for hardcoded secrets avoids human error and catches mistakes even when thousands of lines of code move through a pipeline every day.
Rotate secrets at a set amount of time, such as every 60-90 days. You might be unaware of a compromise of your keys and credentials. Rotating keys limits an attacker’s window of opportunity to use keys for an unlimited time. It also reduces risks if keys are compromised in archives or backups.
Several tips to protect your code from exposing secrets publicly and secure a variable’s value:
Identify and audit all secrets used in code. You don’t know your risks if you haven’t identified all variables that contain sensitive information. Auditing your code for secrets is the first step to building a plan to remediate your security vulnerabilities. This step might take a while to fully finish, but it’s the most critical component in your cybersecurity strategy to better handle secrets stored in your Bitbucket Server or Data Center instance.
Train developers to understand the pitfalls of hardcoded secrets. Developers might hardcode secrets intending to remove them later just to test their software quickly. Whether intentional or unintentional, developers should know the ramifications of hardcoding secrets so that they make it a point of ensuring that their code never exposes secrets. Hardcoding secrets isn’t the only concern. Developers should also avoid hardcoding any sensitive data including personally identifiable information (PII).
Use scanning tools that validate code before it’s committed. Third-party tools are necessary for effective scanning for hardcoded secrets automatically. Some DevOps teams try to handle this step themselves, but it leads to false negatives and security breaches. It’s better to incorporate tools that are built by experts to detect secrets in code without a high level of false negatives.
Add environment variables in Bitbucket. Bitbucket’s dashboard provides a secure way to add environment variables for hardcoded secrets. The secrets are stored on Bitbucket but not actually displayed in the codebase. In addition to being hidden, only users with permission to access the environment variables can view their plaintext values.
Rotate keys at a set interval, preferably every 60-90 days. Good security strategies reduce the window of opportunity for an attacker if an employee is phished or inadvertently and unknowingly divulges their credentials. An attacker might compromise your environment and gain access to keys, but reducing the time that an attacker can use keys limits the amount of damage that can be done. Rotating keys is often done every 60-90 days, and it should be done more frequently with keys that maintain security of sensitive data. If any key is compromised, immediately revoke it and issue a new key.
Audit your access controls. Hardcoding secrets in code is never a good strategy, but you need to take your security strategies a bit further and ensure that the DevOps people with access to secrets should have them. Follow the “least privilege” principle, which says that users should only have access to the data necessary to do their job function. The same rule applies to environment variable access in Bitbucket. Review who has access to secrets and ensure that these permissions are necessary.
Overview of Bitbucket Secret Management (Cloud Only)
Whether you have a public or private repo, it's never okay to store secrets directly in code.
Bitbucket Cloud provides secure ways to store sensitive information in your pipelines so that you can use environment variables that define connections to APIs and other endpoints while still protecting your infrastructure.
Most DevOps teams use three variables in their software development lifecycle:
- Workspace variables: Variables defined for the local agent or local machine directories used to execute code.
- Repository variables: Variables used with settings for your repository for individual projects.
- Deployment variables: Variables defined for the environment executing code. For example, you might have different keys for production and staging environments.
Additionally, there are default variables Bitbucket provides that assist in builds. For example, a script in your build can leverage the default variable BITBUCKET_BUILD_NUMBER to reference the unique ID of the build.
Environment variables in Bitbucket should be called in the bitbucket-pipelines.yml file or any script YAML file used in your Bitbucket pipeline. Here is an example bitbucket-pipelines.yml file that uses environment variables:
Notice that the two variables in the example above do not have hardcoded values. Instead, their values point to variables stored in Bitbucket.
For example, the AWS_ACCESS_KEY variable has the value $DEFINED_ON_BITBUCKET. This value isn’t visible to the public or developers viewing code unless they have permissions to view it in the Bitbucket administration panel.
Note: Bitbucket masks the secure variable's value when displayed in the build logs.
By using this strategy, your YAML files no longer expose secrets to anyone with access to read the plaintext script file. Since the file is in plaintext, any compromise of the DevOps pipeline environment would also expose secrets if they are stored directly in these files. By using environment variables in Bitbucket, you add a layer of protection to your sensitive data. Only users with access to your main Bitbucket dashboard can view or change environment variables.
You can define these variables in your Bitbucket dashboard. Note that Bitbucket has default variables to be aware of, but you can also override them to create your own. To change or create environment variables, you must have the proper permissions in Bitbucket. Here is an overview of where you go to edit variables for each environment variable type.
Workspace Variables
In the Settings section of Bitbucket, click Workspace settings in the left panel. Go to Pipelines > Workspace variables to create your variables.
A few considerations with these variables:
- Repository variables override workspace variables if they are given the same name.
- Variables can be accessed by any developer with write permission on any repository that belongs to the team or account.
- You must have administrator access to the workspace to add or edit variables.
Repository Variables
In the repository, go to Repository settings > Pipelines > Repository variables.
Note that you must be an administrator for the repository to add or edit repository variables.
Deployment Variables
Most DevOps teams work with multiple environments. Development, staging, and production environments are common in the software development lifecycle, and using unique keys for each environment improves security of the entire application. Should an attacker gain access to a development environment variable, the damage might not be as bad as a compromise of a production environment variable.
To set up these variables, go to Repository settings > Pipelines > Deployments.
Or, you can access the variables when editing the pipeline config file.
Secured Variables
DevOps can add a layer of security to their Bitbucket pipelines by creating secured variables. With a secured variable, values are masked in the Bitbucket dashboard, log files, and in any output commands. Any script that uses a secured variable will display the secured variable’s name rather than the value.
Third Party Tools for Bitbucket Secrets
After you create your environment variables, you then refactor your code to remove the hardcoded secrets. Even after you perform refactoring of your secrets, you still must monitor it to ensure that developers don’t inadvertently store them again. For example, if any code changes are rolled back, developers could accidentally make secrets public again or roll back hardcoded secrets that were previously removed.
Soteri scans for added secrets including credentials (e.g., usernames and passwords), API keys, SSH keys, and access passwords that developers might leave in their code.
Note: The tool is available for Bitbucket Server and Data Center. A Cloud version is coming soon.
One benefit of Soteri is that it can be used for automation. The main benefit of DevOps is its ability to automate the software development lifecycle, and Soteri can be integrated into automation procedures to complement it rather than hinder it.
A third-party tool with an API such as Soteri’s lets you incorporate security scans without interfering with your current development lifecycle. Write scripts to scan code as it moves through each environment from development to staging to production.
Some DevOps teams want additional features with a secrets vault that Bitbucket doesn’t offer, so they rely on third-party tools to act as their vault instead of using the Bitbucket dashboard. You should use the native infrastructure vault unless you need a third-party tool with additional features. Integrating third-party secrets management vaults into your development lifecycle creates additional challenges for your DevOps team.
Add Bitbucket Secret Scanning to Your Security Strategy
If you haven’t scanned your code for hardcoded secrets, you should make it a part of your strategy to avoid being the next victim of a compromise. It might seem like a very small risk, but it’s more common than you think.
Let Soteri help reduce your risk and eliminate the vulnerabilities associated with sensitive data hardcoded in your codebase.