We need to regulate these guys – SolarWinds edition

There’s lots of good information coming out about the SolarWinds attacks. All of this, along with information about other attacks that was made public long ago, points to the possible need to start regulating a number of tech companies that I hadn’t thought of as being part of the country’s critical infrastructure – until SolarWinds caused the scales to fall from my eyes.

There’s been a lot of discussion of the SUNBURST malware, which was spread by SolarWinds updates and provided the backdoor that the attackers exploited to penetrate a lot of important networks, especially US government networks. Like most malware, SUNBURST used the “pray and spray” method of cyberattacks: a) attempt to get the malware into as many organizations as possible (which ended up being “only” 18,000); b) wait until the compromised systems start “phoning home”; c) identify the most important organizations that have been compromised; and d) start to find interesting targets to eavesdrop on in those organizatoins.

In the SolarWinds attacks, the benefits the attackers were looking for seem to have been limited to national security espionage. This is unlike for example NotPetya, which had no goal other than bricking as many machines as possible as quickly as possible. It succeeded wildly, not only in the Ukraine (the original target) but even more so in other countries like the US and Denmark, where it almost sank (pun intended) Maersk, the largest ocean shipping company in the world.

Fortunately, these attackers decided not to take the destruction route. Thank God for small favors, as my former boss used to say.

However, SUNBURST was just one of two ingenious pieces of malware used in the SolarWinds attack. There has been very little discussion of the SUNSPOT malware, which was written to attack only one company: SolarWinds. This malware wasn’t designed to do anything like what SUNBURST did: provide a backdoor so the attackers could exfiltrate lots of data from SolarWinds. Instead, it was designed to penetrate SolarWinds’ software build process and plant SUNBURST in software updates, so that SolarWinds would then kindly distribute them to their customers. That attack succeeded brilliantly.

In fact, I nominate SUNSPOT as the second-most-sophisticated piece of malware ever written, after Stuxnet, which was created by the US and Israel to attack Iran’s uranium enrichment program. Stuxnet also succeeded brilliantly.

You can read the full story of SUNSPOT in this article by CrowdStrike, which worked with SolarWinds to analyze the malware and the attack. Here are some highlights, which relate to the subject of this post:

1.     The attackers first penetrated the SolarWinds software development environment in September 2019 – i.e., 15 months before SolarWinds or anyone else knew there was a problem.

2.      Like any good organization about to make a risky investment, the attackers first executed a proof of concept. They wanted to see if they could place a benign piece of software code in SolarWinds Orion during the build process, so it would be included in subsequent updates shipped to SolarWinds customers. The gambit worked perfectly. Just like later on with SUNBURST, SolarWinds had no clue this was happening. In fact, as with SUNBURST, this test code was inserted in multiple releases of the Orion platform, not just one.

3.     With the concept proven, the attackers started to build the final version of SUNSPOT, probably testing it in their own environment. About four months later in February 2020, the attackers compiled and deployed this malware in the SolarWinds software build environment (which was in itself quite a feat, since they had to infiltrate everything they needed to built SUNSPOT into the environment. It would have been much easier to compile SUNSPOT elsewhere and then infiltrate it into the environment, but that would have been a lot easier for SolarWinds to discover).

4.    The most important feature of SUNSPOT was that, just like Stuxnet, it was built assuming there was no possibility for any real-time intervention by the attackers in the software build process. The software had to act completely autonomously; yet it had to quickly adapt to any change in the environment. This was even harder than it sounds since, while SolarWinds wasn’t specifically looking for signs of an intruder inside the build process, the slightest change in the environment – such as a hash value for one of the attackers’ files (which were always named with names that matched files used by SolarWinds in the build process itself) not matching exactly the value for the same file the previous day - would have set off alarms just in the normal build process. This would likely have led to SUNSPOT being discovered and the end of this whole affair (and we’d all have written about how astute SolarWinds was in heading off at the pass what could have been a catastrophic supply chain attack. Instead, what we got was…a catastrophic supply chain attack).

5.     I won’t go into all the brilliant features in SUNSPOT, but I’ll mention one: It was designed to surmount what may have been the biggest problem the attackers faced at SolarWinds: The software build process for the next Orion release went on for months, but neither the attackers nor SolarWinds had any idea exactly how long it would take. Moreover, the process was shut down and restarted every morning, so a new version was produced every day.

There was no way to know in the morning whether today might be the day that everything in the new build worked perfectly. If that happened, the SolarWinds engineers would probably decide to release the code as it stood at the end of that day. This meant that SUNSPOT needed to wipe away all traces of its activity every evening, once it became clear that the build process was about to shut down for the night without having produced the final build, since the slightest discrepancy – for example in the size of a file – would have alerted SolarWinds that something was wrong.

6.     Yet the next morning, SUNSPOT had to recreate everything that had been wiped away the previous evening, to prepare for the possibility that today would be the day the code was declared ready to ship. This whole process is described in great detail in the CrowdStrike article. The important point is that all of this had to be done by SUNSPOT alone, without any prompting or guidance from the attackers - who had no real time visibility into the software build process. Just as the Americans and Israelis who created Stuxnet couldn't control it in any way once it was inside the Natanz uranium enrichment plant, these attackers had to build into SUNSPOT the capability to make all the decisions that would need to be made during the time that the malware was inside the software build environment.

7.      Despite these challenges, when SUNSPOT was deployed in February 2020, it worked perfectly. The SUNBURST malware was planted in about seven releases of the Orion platform, before the attacks decided to remove it from the build environment and cover all of their traces in June 2020. By June, SUNBURST had already infected about 18,000 targets. It’s likely the attackers didn’t have enough good cyber resources available to exploit all of those targets, let alone any new ones that might be added.

What’s the lesson of all this? SolarWinds really dropped the ball. Sure, this was an unprecedented attack that nobody saw coming. But it’s impossible to believe there was no way they could have detected the Russians in their network during the 15 months between when the attackers first entered and when SolarWinds learned about it. Unforunately, SolarWinds only learned they had been penetrated when the rest of the world learned it: after FireEye discovered that they had been compromised by SolarWinds Orion in December 2021

It’s also impossible to believe that there is no way SolarWinds could have learned that their development process was compromised, when there were about 5-6 months during which one or the other of the two pieces of malware (the benign test code and SUNBURST) was being inserted into every Orion update that left their shop.

Do I know how SolarWinds could have detected – and presumably stopped, since that would have been very easy had their presence been known – the attackers? No I don’t. But there’s one thing that I do know (although only with hindsight, I’ll readily admit): SolarWinds, given that they were accepting license fees from huge companies and important government agencies like DoE, DHS and NSA, should have been paying a lot more attention to cybersecurity than they did. They were clearly fat, dumb and happy and on top of the world – until they weren’t.

Companies like SolarWinds are really critical infrastructure organizations (again, I say this with hindsight. I wouldn’t have said it at all two months ago). The entire public has a stake in their safe, successful operation; that stake goes well beyond the license fees that SolarWinds earns. SolarWinds can’t hide behind the “Well, if you don’t like our product, nobody’s forcing you to buy it” rationale. Just as with public utilities, the public has a big stake in SolarWinds’ safety and stability.

The Northeast Blackout of 2003 showed that protection of electric reliability is too important to leave up to the utilities on their own, and the SolarWinds attacks show that protection of network integrity in federal agencies is too important to leave entirely up to providers like SolarWinds. There needs to be regulation to make sure that SolarWinds and their ilk do all they can to protect that network integrity, just like there are regulations to make sure electric utilities do all they can to protect electric reliability (including cybersecurity).

As an economics major at the University of Chicago a fairly long time ago, I was fortunate enough to take two courses on microeconomics with Dr. Milton Friedman. While there was never a doubt that he was in favor of capitalism and free markets, he always made it clear that in some cases, purely market mechanisms can’t adequately protect consumers against common threats. The main case in point at the time was air and water pollution, since the EPA had just been created by Richard Nixon. Friedman pointed out there was no way that individual consumers, or even individual companies, could use market power to protect themselves against most pollution threats; there needed to be regulation.

The same reasoning applies today, in the case of software used in critical government agencies.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.