Written by Samuel Bickham and John Cavanaugh
The compromise of SolarWinds is a sobering fact and should resonate with both enterprise stakeholders and software developers alike. An established, successful, widely proliferated product becoming the “weakest link” in the security stack reminds us that vulnerabilities do not discriminate between homegrown code and professionally developed products. In addition, the attack vector we discuss should incentivize increased security vigilance in our lower-level environments as well as our supply chains.
The Element of Surprise
While news of the SolarWinds attack became widely known on December 13, 2020, the first initial disclosure was actually made on December 8, 2020, when cybersecurity firm FireEye revealed that it was hacked by a nation-state APT (Advanced Persistent Threat). APTs are typified by advanced hacking techniques that maintain a low profile, appearing dormant over long periods of time while continuously attempting malicious behavior. As part of this attack, the threat actors stole Red Team assessment tools that FireEye uses to probe its customers’ security. “Red Team” or “White Hats” are the “good guys” performing authorized penetration testing.
It was not known how the hackers gained access to FireEye’s network until Sunday, December 13, 2020, when Microsoft, FireEye, SolarWinds, and the U.S. government issued a coordinated report that SolarWinds had been hacked by state-sponsored threat actors believed to be part of the SVR RF (i.e. Foreign Intelligence Service of the Russian Federation) and all active consumers of the SolarWinds product were now at risk. This prompted immediate mandates for physically quarantining the SolarWinds product from government networks, and recommendations for private sector customers soon followed.
The relatively early discovery was likely happenstance, as one of the victims was FireEye. One can only speculate how much longer this infection would have persisted unnoticed had one of the victims not also been a cybersecurity services provider.
The Attack Vector:
As part of the attack, the threat actors gained access to the SolarWinds Orion build system and added a backdoor to the legitimate “SolarWinds.Orion.Core.BusinessLayer.dll DLL” file. This DLL was then distributed to SolarWinds customers in a supply chain attack via automatic software updates. The diagram below illustrates the attack workflow.
This DLL backdoor is known as Sunburst (as per FireEye) or Solorigate (as per Microsoft) and is executed by the “SolarWinds.BusinessLayerHost.exe” program. Once loaded, the malware will connect back to the remote CNC (command & control) server at a subdomain of “avsvmcloud-dot-com” to receive “jobs,” or tasks, which will execute on the infected computer.
Figure 1 – SolarWinds Supply Chain Attack
While definitive RCA (root cause analysis) for the specific security control failure that led to SolarWinds being compromised is not known to date, the attack workflow description brings a few questions to bear.
- What is the organization’s current security posture in development, test, and/or quality control environment(s)?
- What are the organization’s quantifiable upstream and/or downstream liabilities, and can they be included in security risk analysis?
Many organizations treat their development and/or test environments like large animal safari nature preserves; anything goes as long as it doesn’t leave the fence of the nature preserve – i.e., touch production. This perspective is understandable, given that development is not production, developer agility affects development efficiency, and security controls come with a price tag. However, we must negotiate a balance between maximizing the security posture of our development environments and minimizing our security risks. Just because a server is in the development environment doesn’t exclude it as a security risk. Software-defined networking technologies are continually reducing mean-time-to-deployment for network and security-related changes, and the battle between product release dates and infrastructure configuration changes remains ongoing.
The terms “upstream liability” and “downstream liability” are used in various security and legal contexts, providing multiple definitions and interpretations. One interpretation of these terms can be applied in terms of the logical supply chain. Events operating north of our organization, or before our organization can exert influence, could be considered upstream. Events operating south of our organization, or after our organization has exerted influence, could be considered downstream. When the security risks of both upstream and downstream events are realized, they can be quantified as liabilities.
An example of “upstream liability” would be the failure of a cloud provider’s internal infrastructure or when vendor software updates result in system downtime. An example of a “downstream liability” would be a 3rd party contractor connected to a corporate VPN while being infected with malware or a compromised business partner network connected to our organization via site-to-site VPN.
It is common to see governance in place for management of 3rd party relationships, whether they are legal, regulatory, licensing, or contractual in nature. Criteria for this oversight becomes gray when we consider that software updates from a trusted vendor relationship could pose a security risk, especially when the target of that software update is also a security countermeasure (e.g. a firewall).
The Known Components:
After performing investigations of SolarWinds supply chain victims, researchers have begun to get a better idea of the different malware used in the SolarWinds attack. According to CrowdStrike, a malware named SunSpot was first executed in the SolarWinds network to monitor for and automatically inject the Sunburst backdoor in the SolarWinds development builds. The Sunburst backdoor would then be transferred to victims via automatic updates via the SolarWinds Orion platform. Once executed, Sunburst would routinely connect to a remote command and control server for commands to execute on the infected device. FireEye discovered that the Sunburst backdoor would drop a malware named Teardrop, a post-exploitation tool used to deploy customized beacons via the Cobalt Strike code framework. Finally, Symantec discovered the Raindrop malware, which was also used to deploy beacons via Cobalt Strike onto other hosts in an already compromised network.
SunSpot
1. Sunburst
a. Teardrop
i. Deploys beacons via Cobalt Strike
b. Raindrop
i. Deploys beacons via Cobalt Strike
Hierarchy of SolarWinds Malware
The Conclusion:
Does this cybersecurity incident spell the end of Orion SolarWinds? Not likely. Consumers have surprisingly short memories when it comes to security-induced reputation damage. For example, did consumers and businesses swear off using Equifax, Target, and Sony when they were hacked and consumer data was leaked to the Internet? Hardly. Did the U.S. Government abandon the OPM (Office of Personnel Management) and shut their data systems when they were hacked? No. Executives will take their knocks, and stock prices will take their dip, developers will steam over rectifying the vulnerability, and life will go on. However, each new attack that grows in complexity and damage will add new overhead to development timelines and budgets, as today’s “zero-day” threat becomes tomorrow’s standard protection practice. There is no product or architecture that can prevent all attacks and eliminate all security risks from our organizations. However, with proper care and due diligence, we can reduce our risks, augment our safeguards and have a clear understanding of the potential liabilities we face, both within and without.