What if you were told that you had a misconfigured registry with hundreds of millions of software artifacts containing highly confidential and sensitive proprietary code and secrets exposed in your environment right now? This would be what you’d call a really bad day for security. Recently, the Aqua Nautilus research team found just that in some of the world’s largest organizations, including five Fortune 500 companies.
In some of these cases, anonymous user access allowed a potential attacker to gain sensitive information, such as secrets, keys, and passwords, which could lead to a severe software supply chain attack and poisoning of the software development life cycle (SDLC). In total, we detected thousands of exposed registries and artifact repositories containing over 250 million artifacts and over 65 thousand container images.
Our experience working with the security teams of companies such as IBM, Cisco, Siemens and Alibaba were excellent. We followed established procedures to report our findings through designated channels, and we found that the security teams were eager to learn from our discoveries, take immediate corrective action, and seek out long-term solutions. However, we were surprised that some major corporations ignored our warnings, which put both their businesses and customers at significant risk.
Software development life cycle in the cloud
The SDLC is a set of best practices, processes, and tools that enables organizations to create and deploy software in the cloud efficiently. It aims to reduce the risk of errors and bugs by automating the process of building, testing, and deploying software. The SDLC typically includes writing code and committing it to a version control system, automating the building and testing of the code using Continuous Integration (CI) tools, and deploying the code to a staging environment for testing and validation. Once these steps are successfully completed, the code is then deployed to production or stored in container image or artifact registries for future use.
Registries, repositories, and artifact management systems
Registry, repositories, and artifact management systems are different types of development and operation software used in package management:
- Registry is a central location for storing and managing packages
- Repository is a collection of packages within a registry
- Artifact management system is a tool for managing binary files such as JAR files
The main difference is that a registry is a general term for a place to store and manage packages, a repository is a specific collection within a registry, and an artifact management system is a type of registry for managing binary files.
The scope of our research
The cloud has a vast attack surface, with numerous potential entry points that attackers can use to gain access to a cloud environment. Our research focused on software supply chain attacks, specifically how threat actors exploit registries. We believe that registries are a crucial part of the software supply chain in the cloud and that organizations don’t pay enough attention to them. If attackers gain access to registries, they can propagate and potentially exploit the entire SDLC.
At this point it is imperative to state out that in many cases artifact management systems and container registries are connected to the internet deliberately and by design allowing anonymous users to connect to various areas in the registry or even to the entire registry. This design allows global teams, customers and other stakeholders access to open-source software that is shared across the company or with outside users. In some cases, however, restricted environments are accidentally shared with anonymous users, in other cases teams accidentally publish sensitive information to public areas.
In this blog post, we are shedding light on the potential dangers of misconfigurations in registries and how it can be extremely risky in certain scenarios. We’ll cover misconfigurations such as mistakenly connecting registry to the internet, exposing secrets to public registries, using default passwords, granting high privileges to users, and more.
We’ll also examine anonymous access. In some tools, it’s considered a misconfiguration issue, for example in a private container image registry, while in others anonymous access is a built-in feature intended to make the cloud SLDC easier for organizations.
We discovered that some organizations fail to properly secure these highly critical environments, leaving them exposed to the internet and vulnerable to exploitation, which can lead to serious and damaging attacks.
Summary of the findings
Our research was conducted from an attacker’s perspective, where we examined how an attacker could gain initial access and how far they could move laterally across the cloud development pipeline.
In our research, we discovered multiple container image registries and Quay registries. In addition, we also found Sonatype-Nexus registries and JFrog artifactories that were publicly accessible on the internet. An accessible over the internet registry is one that is connected to the internet and can be accessed at least as far as the login page. An anonymous user access registry is one that can be accessed with anonymous read and/or write privileges.
Here are the key findings:
While we are fully aware that in many cases finding an artifact registry connected to the internet and even one that allows anonymous user access, since in many times these registries are accessible by design, we weren’t expecting to find inside these registries’ secrets. As you can see in the figure below, on 1,400 distinct hosts we found at least one sensitive key (such as keys, secrets, credentials, tokens, etc.) and on 156 hosts we found private sensitive addresses of end points (such as Redis, MongoDB, PostgreSQL, MySQL, etc.).
We found 57 registries with critical vulnerabilities such as default admin passwords, out of which 15 registries allowed admin access with the default password. We detected more than 2,100 artifact registries with upload permissions, which may allow an attacker to poison the registry with malicious code.
We found small, medium, and large organizations from all over the world, including 10 companies from the Fortune 500 list. Only the registries of five Fortune 500 companies contained highly sensitive information and in some cases wasn’t supposed to be exposed or allow anonymous access. Additionally, we found two leading cybersecurity companies had exposed secrets in their registries, and a significant number of smaller companies had similar issues that put them at risk.
The combined annual revenue of all companies in our research is 1.3 trillion USD and, on average, each company has a revenue of 10 billion USD, with a median revenue of 30 million USD.
The risks of exposed and exploitable registries
Here are a few examples of the potential consequences of having an exposed and exploitable registry connected to the internet:
International tech giant: The risks of Shadow IT
During our research, we discovered two misconfigured container image registries that belonged to the development and engineering teams working on a Fortune 100 tech giant’s cloud. One of the container image manifests contained a command to download artifacts from the artifact registry, which included an active API key to retrieve internal binaries as part of building an image.
We found that the artifact registry contained 2,600 repositories with over 240 million artifacts, since its sensitivity we sampled few artifacts and found it including repositories with built artifacts affiliated with the production environment and repositories for managing internal software libraries. Moreover, The API key had ‘can deploy’ privileges, which could potentially allow a bad actor to poison artifacts such as libraries, images, and releases.
We decided to halt our research at this point due to the sensitivity of their environment and disclose our findings to the company’s CIRT team.
The tech giant’s security team was very professional and eager to learn about our findings. They promptly investigated the items in our report and took immediate measures to mitigate the risks. We later learned that this was a case of Shadow IT, where a developer with a side project opened an environment against policy and regulations without proper controls.
The company’s security team shared many lessons to learn from this case. Shadow IT is a serious problem that poses a threat to any organization, big or small. However, several steps could have prevented access to these environments. First, the registry should not be exposed to anonymous users. Second, the manifest should not contain hard-coded secrets. Third, the API key should follow the principle of least privilege and apply key rotation policy, and fourth, network controls should protect the API of the artifact registry.
Here is an illustration of how it could have been done:
[iframe src=”https://1665891.fs1.hubspotusercontent-na1.net/hubfs/1665891/Shadow-IT-Aniamtion-Demo-01-1.mp4″ height=”400″]
An attacker could have gained a further foothold in the company’s cloud environments by poisoning the artifact repositories, which allows him to control the entire software supply chain, from infecting the developers and up to the cloud platform environment
International tech giant #2: The risks of purposely open artifact registries
During our research, we discovered that another tech giant, “company #2” had a deliberately open public artifact registry. In today’s fast-paced software development landscape, it is crucial to ensure that businesses can operate as smoothly as possible. However, these types of environments can be challenging to secure. With hundreds of teams and thousands of professionals working with public environments, it is inevitable that mistakes will happen.
In the case of “Company #2”, we found a project that was nearing its end-of-life that revealed a token. Our disclosure prompted an internal discussion within the organization. Although the company claimed the token was meant to be public, we shared concerns about the potential attack vectors. For instance, if internal names of artifacts such as Python or NPM packages were exposed to the internet, it could create an attack vector for the popular dependency confusion that could have a significant impact on the organization.
The security stakeholders made the decision that all individuals, whether internal or external, must be adequately monitored when it comes to organizational assets. With so many professionals having access, it is only a matter of time before someone makes a mistake. As a result, Company #2 implemented stricter controls over the registries, requiring users to register and issue tokens with expiration dates. This is another example of a security team eager to learn and educate the community.
Healthcare organization: Complete access for an attacker to prey
During our research, we discovered a container image registry that contained plenty of keys and secrets to the organization’s environments, including PGP keys, complete access to websites, database access, staging environments, Stripe (payment application) keys, and their code. This information could have been used by an attacker to poison their codebase and gain unauthorized access to the organization’s environments. Lastly since this was a healthcare organization it could have been targeted by state-sponsored threat actors or financial threat actors who sell private indefinable information over the dark-web which can lead to identity theft of the customers of the healthcare. Unfortunately, it was very hard to find a proper professional to disclose this issue. But after 3 weeks of persevering attempts, we disclosed this matter, and they took immediate action.
Tech startup: Exposing environment variables to anonymous users
We also uncovered a startup that had an anonymous user with access to their artifact registry. It also allowed access to the artifact build section, where the anonymous user was able to view the environment variables. From there, the anonymous user was able to see the admin access credentials to the artifact registry, AWS credentials to production environments, and access to the company’s source code management and CI environments.
We alerted the company of this issue, and the CTO acknowledged it as an instance of Shadow IT with active AWS credentials to highly sensitive environments.
{% video_player “embed_player” overrideable=False, type=’hsvideo2′, hide_playlist=True, viral_sharing=False, embed_button=False, autoplay=True, hidden_controls=True, loop=True, muted=True, full_width=False, width=’1000′, height=’562′, player_id=’112018541926′, style=” %}
Immediate steps to take
If you have a registry instance in your environment, we recommend taking the following actions immediately:
- Check if it is exposed to the internet.
- If the registry is connected to the internet by design, check that the version isn’t critically vulnerable and that the default password is disabled. Verify that the passwords are strong enough.
- In addition, verify that the anonymous user is disabled. If the anonymous user is purposely enabled, verify that public artifacts in your repository do not contain any secrets.
- Rotate any secrets that may have been exposed.
We prepared a handy cheat sheet to help you tackle this with confidence and mitigate the risks:
Aqua’s software supply chain security module automatically checks and alerts on these critical registry misconfigurations. The solution plugs in directly into the CI/CD stack (including registries) and scans for vulnerabilities in the code / image layer, the pipeline process, and tool configuration. If we take this specific example, as soon as you connect your registry to Aqua, you’ll be able to see if you’re using default passwords, have exposed instances / instances which allow for anonymous user access, secret rotation misconfigured, and more. You can see an example from a test environment below.
This is aimed to both continuously audit the environment against compliance best practices and give you the capability to automatically detect and remediate the misconfigurations that are putting you at actual risk, like in the case described in this blog. Users have also expressed that this is extremely handy in cases where they have a wide CI/CD tech stack; instead of having to manually check each of their instances regularly (we found that most companies leverage a variety of different tools and instances for their regular development operations), the tool applies the security standard and validates it across the entire development organization; no matter how wide or complex.
Conclusion and mitigation
At Aqua Nautilus, we often engage with organizations to gain insights into their current cloud security practices. In these discussions, it’s clear that many security professionals, including CISOs, security engineers, DevOps engineers, and application security experts, do not fully grasp the significant and damaging impact that a single misconfiguration can have. Despite claims of hardening their environments, many remain unaware of the risks associated with cloud misconfigurations.
We began our research with the goal of better understanding who is behind exposed and misconfigured registries and the potential impact of a skilled attacker. The findings were both surprising and concerning.
Our research also highlights how important it is for an organization to have a security responsible disclosure program in place. This program is designed to allow security researchers to report in a structured manner any potential vulnerabilities they discovered in company’s products. This way, it enables an organization to quickly address and resolve security issues before they can be exploited by malicious actors.
In our research, we used this program to disclose severe misconfigurations. Nautilus found that organizations with existing responsible disclosure programs were able to fix a misconfiguration in less than a week. However, for organizations without such a program in place, the process was more difficult and time-consuming.
To sum it up, a security responsible disclosure program is a crucial tool that helps organizations protect their systems, data, and reputation from cyber threats. It’s vital that organizations establish or add a dedicated email address and a focal point on their website to facilitate the reporting of potential security issues.
Our main takeaway from this research is that everyone should invest more in detecting and mitigating cloud-native environment risks. To effectively reduce the risks associated with exposed container registries and artifact registries, it’s essential to take the following steps:
- Secure the repositories with network controls such as a VPN or firewall. This can help to protect the repositories from external threats and reduce the risk of unauthorized access.
- Implement strong authentication and authorization measures. This includes using strong passwords, two-factor authentication, SSO and replacing default passwords.
- Regularly rotate keys, credentials, and secrets. This includes periodically changing passwords, access keys, and other forms of authentication and authorization to prevent unauthorized access.
- Implement least privilege access controls and scoping, assigning the appropriate level of access to different roles especially for anonymous access and restricting access to specific repositories and artifacts as needed.
- Regularly scan for sensitive data. This includes scanning artifact and container registries for known vulnerabilities and secrets and conducting regular security assessments in the repositories. It’s important to promptly address and mitigate any vulnerabilities and rotate any exposed secrets in order to prevent exploitation by attackers.
By implementing these best practices, organizations can effectively mitigate the risks of exposing their registries and artifact repositories and protect against potential security threats. Download a full list of recommendations to mitigate risk from exposed registries & artifact repositories here.
Disclosure Timeline
- End of November 2022 – First disclosure of a severe risk
- December 2022- Reaching out to the majority of the companies found at risk
- January 2023 – Finding the Healthcare organization and sealing the breach
- January – April 2023 – 3 months period requested by companies to apply full root cause analysis including mitigation.
| Our research has highlighted the importance of organizations having a security responsible disclosure program in place. This program allows security researchers to report any potential vulnerabilities they have discovered in a structured manner, enabling the organization to quickly address and resolve the issue before it can be exploited by malicious actors. Our research found that, on average, it took only 1.5 days for organizations with responsible disclosure programs to close a misconfiguration once it was reported. However, for organizations without such a program in place, it was a more difficult and time-consuming process. Overall, a security responsible disclosure program is a vital tool that helps organizations to better protect their systems, data, and reputation from cyber threats. It is crucial that organizations establish or add a dedicated email address and a focal point in their website to facilitate the reporting of potential security issues |
