Browsing Tag

ICS

Fourth Sample of ICS Tailored Malware Uncovered and the Potential Impact

April 25, 2016

I first posted this piece on the SANS ICS blog here.

 

I looked at the S4 Europe agenda which was sent out this morning by Dale Peterson and saw an interesting bullet: “Rob Caldwell of Mandiant will unveil some ICS malware in the wild that is doing some new and smarter things to attack ICS. We are working with Mandiant to provide a bit more info on this in early May without doing the big reveal until S4xEurope.”
For those of you that don’t know, S4 is a conference run by Dale Peterson and this is their European debut (the other versions are in Florida and Japan and are staples of the ICS security conference scene always having hard hitting and top notch presentations). As a trusted conference, S4, and friend, Dale, I give a higher bit of credibility to anything that comes out of there than your typical conference. Add that to the fact that the Mandiant ICS team has a number of extremely credible voices (Rob Caldwell, Dan Scali, Chris Sistrunk, Mark Heard, etc.) this is even more interesting and credible.

Let’s break down what we know and why this is potentially very important.

Background on ICS Tailored Malware
To date there have been exactly three ICS tailored malware families that are publicly known. The first was Stuxnet which contained modules to target the Siemens’ systems at the Natanz facility in Iran and cause physical damage to the P-1 centrifuges. Second, there was the Havex malware used in the Dragonfly campaign (aliases include Energetic Bear and Crouching Yeti) that had a module that specifically searched for ICS specific ports (such as 102, 502, and 44818) and later more importantly an OPC module. Lastly, there was the BlackEnergy 2 malware which contained exploits and versions for GE’s CIMPLICITY and Siemens’ SIMATIC environments.

Why Haven’t We Seen More?
Most of us understand that ICS environments make for great targets especially for nation-state and APT styled actors. The ability for military posturing, political leverage, espionage, and even intellectual property theft make enticing targets. Yet, the numbers simply do not seem to align with the fear that many folks have about these environments being targeted. The question always comes up: why don’t see more ICS intrusions? I do not claim to know for sure but my running hypothesis is that it boils down primarily to three areas:

1. We do not have a lot of visibility into ICS networks. Many of the threats that we are aware of we know about due to vendors releasing reports. These vendors traditionally have end point security solutions and anti-virus in the networks that report back information to them. This allows the vendors to “see” tens of thousands of networks and the threats targeting them. In ICS we do not have these same products in scale and many are disconnected from the vendors (which is ok by the way and sometimes preferable). That combined with a lack of understanding of how to monitor these environments safely and interact with them creates a scenario where we don’t see much. Or in short, we aren’t looking.

2. Most malware families tend to be criminal in nature. APT styled malware is not as common in the larger security field. There simply isn’t as big of a motivation for criminals to make ICS specific malware families when ransomware, botnets, etc. work just as effectively in these environments anyway and they represent a smaller portion of the population. This is similar to the old Mac vs. Windows vs. Linux malware debate. One of the reasons we see more Windows malware is due to pure numbers and not because it’s less secure. There is more motive for criminals to write Windows based malware usually. For the APT styled actors, targeting ICS can be important for military and intelligence purposes but there isn’t as much motive to actually attack or bring down infrastructure outside of conflict scenarios; just to learn and position. I have my suspicions that there are a great number of ICS networks compromised with a large variety of ICS specific malware out there and we just haven’t seen the impacts to begin looking (see point #1).

3. ICS specific knowledge sets are rarer making it more difficult to create well-crafted and tailored ICS modules. The typical “cyber team” for nation-states are pretty good at Windows based environments but down in the lower ICS networks it requires system specific knowledge and engineering skills to truly craft something meaningful. This knowledge set is expanding though meaning we will definitely see more and more of these types of threats in the future.

Why is the Mandiant Discovery Potentially Important?
The claim that Mandiant has found a new ICS tailored piece of malware is important for a few reasons.

First, I have a good amount of respect for the Mandiant ICS team and if they say they’ve found something ICS specific I’ll still require proof when the time comes but I’m more inclined to believe them. Knowing the team members though I’m confident they’ll release things like indicators of compromise (IoCs) and technical knowledge so that the community can independently verify the find. This is great because many times there are claims made, even by trusted companies, without any proof offered. My general stance is that no matter how trusted the company is if there isn’t proof (for example the recent Version claim about the water hack) then it simply does not count. The community has been abused a lot with false claims and proof is required for such important topics.

Second, given that there have only been three ICS tailored malware families to have a fourth is incredibly interesting for the research both into the defense of ICS but also into the threat landscape. Understanding how the intrusions took place, what the targets were, and extracting lessons learned will be very valuable to both the technical and culture challenges in this space. It remains to be seen exactly what Mandiant means by “ICS specific” although I have messaged some trusted contacts and have been told that the agenda point isn’t a misprint; Mandiant claims to have found tailored ICS malware and not just an ICS themed phishing email or something less significant. Although I never wish harm on anyone from a threat and defense research perspective this is an amazing find.

Third, it bodes well for the ICS security industry as a whole to start making some more positive changes. There have been many ICS security companies around for years (security and incident response teams like LoftyPerch, independent consultants and contractors, red teams like Red Tiger Security, etc.) and even some dabbling by larger companies like Kaspersky and Trend Micro (who both have contributed amazing information on the ICS threat landscape). But the Mandiant ICS team in a way represents a first in the community. Mandiant, and its parent company FireEye, is a huge player in the security community. For years the Mandiant team itself has been widely respected for their incident response expertise. To have them come out and make a specific ICS team to focus on incident response was actually a big risk. It is common to see ICS products and services but many of the startups struggle much more than the media and venture capitalists would let on. Mandiant’s ICS play was a hope that the market would respond. To have the team come out with a fourth specific ICS tailored malware family bodes very well for the risk they took and with the appropriate coverage while keeping down hype this could be very important for the industry and market writ large. Of course the customers always get a big vote in this area but it could mean more folks waking up to the fact that yes ICS represent a target and yes the security community can calmly and maturely approach the problem and add value (again, please no hype, wallpapers, and fancy logos though for exploits and malware).

But Aren’t Squirrels More Damaging to the Grid?
I gave an interview to a journalist for a larger piece on squirrels and cyber threats with regards to the power grid and I believe it warrants a discussion in this piece’s context. The common joke in the community is that squirrels have done more damage to power grids than the US, China, Iran, Russia, UK, etc. combined. And it’s true. It is often stated by us in the industry to remind folks that the “OMG Cyber!” needs to calm down a bit and realize that infrastructure operators on a daily basis deal more with squirrels and Conficker than APT styled malware. However, we should not equate the probability of attacks with the importance of them. As an example, let’s consider the recent DHS and FBI report on the risk to the U.S. electrical infrastructure.

I have a lot of love and respect for many of the FBI and ICS-CERT personnel I’ve worked with. I can only describe most of them as extremely passionate and hard working. But, the claim that the risk of a cyber attack against U.S. electrical infrastructure was low was upsetting to me because of how it comes across. On the heels of the cyber attack that impacted the Ukrainian power grid the report seemed to downplay the risk to the U.S. community. It stood in direct contrast to Cyber Command’s Admiral Rogers who stated that “It is only a matter of the when, not the if we’re going to see a nation-state, group, or actor engage in destructive behavior against critical infrastructure in the United States.” He was specifically talking in context of what happened in Ukraine and the importance of it. As the head of both the NSA and the U.S. military arm for cyber it is appropriate for Admiral Rogers to have a good understanding of the foreign intelligence and foreign military threat landscape. For the DHS and FBI to contradict him, even if unintentionally, seems very misplaced in what their expertise and mission is; and this leads back to the squirrel comment.

It is not as important to think of probability with regards to destructive attacks and ICS focused intelligence operations. When a community hears of a “low probability” event they naturally prioritize it under other more high probability events. As an example, prioritizing squirrels over nation-state operations based on probability. The problem with doing that though is that the impact is so much more severe for this “lower probability” scenario that the nation must prioritize it for national security reasons. Telling the infrastructure operators, who really defend the grid not the government, to stay calm and carry on is directly competing with that need although the message should admittedly always avoid hype and alarmism. Mandiant coming out with the fourth variety of ICS tailored malware helps highlight this at a critical point in the debate both among infrastructure operators and policy makers.

Conclusion and What to Do
We won’t know exactly what the ICS tailored malware is, what it’s doing, or technical knowledge of it until Mandiant releases it. It could be a dud or it could be extremely important (knowing the Mandiant team my bet is on extremely important but let’s all remain patient for the details before claiming it to be so). However, infrastructure owners and operators do not need to wait for the technical details to be released. It is important to be doing industry best practices now including things such as network security monitoring internal to the ICS. The other three samples of ICS tailored malware were all incredibly easy to identify by folks who were looking. Students in my SANS ICS515 ICS Active Defense and Incident Response class (shameless plug) all gets hands on with these threats and are often surprised at how easy they are to identify in logs and network traffic. The trick is simply to get access to the ICS and start looking. Or in other words: you too can succeed. Defense is doable. So do not feel you need to wait for the Mandiant report. It is potentially very important and technical details will help hunt the threats but you can look now and maybe you’ll spot it, something else, or at the very least you’ll get familiar with the networks you should be defending so that it’s easier to spot something in the future whether its APT styled malware or just misconfigured devices. Either way – the most important ICS is your ICS and learning it will return huge value to you.

Minimum Viable Products are Dubious in Critical Infrastructure

December 4, 2015

Minimum Viable Products in the critical infrastructure community are increasingly just mislabeled BETA tests; that needs to be communicated correctly.

The concept of a Minimum Viable Product (MVP) is catching on across the startup industry. The idea of the MVP is tied closely to The Lean Startup model created by Eric Ries in 2011 and has very sound principals focused around maximizing the return on investment and feedback from creating new products. Eric defines the MVP as the “version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort.” This enforces the entrepreneurial spirit and need for innovation combined with getting customer feedback about a new technology without having to develop a perfect plan or product first. An MVP is also meant to be sold to customers so that revenue is generated. In short, be open to testing things out publicly earlier, pivot based off of technical and market feedback, and earn some money to raise the valuation of the company and entice investors.

Personally, I believe the lean startup model as a whole is smart. I use some aspects of the model as CEO of Dragos Security. However, I chose not to use the concept of an MVP. Minimum Viable Products are dubious in critical infrastructure. I state this understanding that the notion of getting the product out the door and gaining feedback to guide its development is a great idea. And when I say critical infrastructure I’m focusing heavily on the industrial control system (ICS) portion of the community (energy, water, manufacturing, etc.). The problem I have though is that I have observed a number of startups in the critical infrastructure startup space taking advantage of their customers, albeit unintentionally, when they push out MVPs. This is a bold claim, I won’t point fingers, and I don’t want to come across as arrogant. But I want to make it very clear: the critical infrastructure community deals with human lives; mistakes, resource drains, and misguiding expectations impact the mission.

My observations of the startups abusing the MVP concept:

  • Bold claims are made about the new technologies seemingly out of a need to differentiate against larger and more well established companies
  • Technologies are increasingly deployed earlier in the development cycle because the startups do not want to have to invest in the industry specific hardware or software to test the technology
  • The correct customers that should be taking part in the feedback process are pushed aside in favor of easier to get customers because successes are needed as badly as cash; there is pressure to validate the company’s vision to entice or satisfy Angel or Seed investors
  • The fact that the technology is an MVP, is lightly (if at all) tested, and will very likely change in features or even purpose is not getting communicated to customers in an apparent attempt to get a jump start on the long acquisition cycles in critical infrastructure and bypass discussions on business risk
  • Customers are more heavily relied upon for feedback, or even development, costing them time and resources often due to the startups’ lack of ICS expertise; the startup may have some specific ICS knowledge or general ICS knowledge but rarely does it have depth in all the markets its tackling such as electric, natural gas, oil, water, etc. although it wants to market and sell to those industries

What is the impact of all this? Customers are taking bigger risks in terms of time, untested technologies, changing technologies, and over hyped features than they recognize. If the technology does not succeed, if the startup pivots, or if the customers burn out on the process all that’s been accomplished is significant mistrust between the critical infrastructure stakeholders and their desire to “innovate” with startups anymore. And all of this is occurring on potentially sensitive networks and infrastructure which have the potential to impact safety or the environment.

My recommendations to startups: if you are going to deploy technologies into critical infrastructure early in the development cycle make sure the risks are accurately conveyed and ensure that the customer knows that they are part of a learning process for your technology and company. This begs instant push-back: “If we communicate this as a type of test or a learning process they will likely not trust our company or technology and choose to go with other more established products and companies. We are trying to help. We are innovators.” And to my straw man here, I empathize greatly. Change is needed in this space and innovation is required. We must do better especially with regards to security. But even if we ignore the statistics around the number of failed technologies and startups that would stress why many should never actually touch an ICS environment I could comfortably state that the community is not as rigid as folks think. The critical infrastructure community, especially in ICS, gets cast in a weird light by many outside the community. My experience shows that the critical infrastructure community is just as innovative, and I would argue more so, than any other industry but they are much more careful to try to understand the potential impact and risks…as they should be.

My experience in a new technology startup: when the Dragos Security team was developing our CyberLens software we needed to test it out. Hardware was expensive and we could not afford to build out networks for every type of vendor’s ICS hardware and network communications. Although we have a lot of ICS knowledge on the team we all were keenly aware that we are not experts in every aspect of every ICS industry we wanted to sell to. Customer feedback was (and still is) vital. To add to this we were pressed because we were competing with larger more established companies and technologies but on a very limited budget. So, instead of trying to sell an MVP we simply launched a BETA instead; the BETA lasted over twelve months. How did we accomplish this? We spent $0 on marketing and sales and focused entirely on staying lean and developing and validating our technology. We made contacts in the community, educated them on what we wanted to do, advised where the technology was tested and safe to deploy, and refused to charge our BETA participants for our time or product since they were greatly helping us and keeping our costs down. In turn we offered them discounts for when our product launched and offered some of our time to educate them in matters we did have expertise. This created strong relationships with our BETA participants that carried over when we launched our product to have them join us as customers. We even found new customers when we launched based on referrals from BETA participants vouching for our company. Or more simply stated: we were overly honest and upfront, avoided hype and buzzwords, and brought value so we were seen as fellow team members and not snake oil salesmen. I recommend more startups take this approach even under pressure and when it is difficult to differentiate in the market.

My conclusion: the MVP model in its intended form is not a bad model. In many communities it is an especially smart model. And just because a company is using an MVP route in this space does not mean they are abusing anyone or falling into the pitfalls I listed above. But, as a whole, in the critical infrastructure community it is a process that is more often abused than used correctly and it is damaging in the long term. Customers are not cash cows and guinea pigs – they are investors in your vision and partners. Startups should still push out technologies earlier than trying to wait to create a perfect product without the right feedback, but call these early pushes what they are. It is not a Minimum Viable Product it is a BETA test of core features. Customers should not be asked to spend limited budgets on top of their time and feedback for it nor should they be misled as to what part of the process they are helping in. You will find the community is more likely to help when they know you are being upfront even with understandable shortcomings.

Security Awareness and ICS Cyber Attacks: Telling the Right Story

October 7, 2015

This was first posted on the SANS ICS blog here.

 

A lack of security awareness and the culture that surrounds security is a widely understood problem in the cyber security community. In the ICS community this problem is impactful towards operations and understanding the scope of the threats we face. A recent report by the Chatham House titled “Cyber Security at Civil Nuclear Facilities”shined a light on these issues in the nuclear industry through an 18 month long project.

The report highlights a number of prevailing problems in the nuclear sector that make security more difficult; the findingsdo not represent all nuclear sector entities but take a look at the sector as a whole. Friction between IT and OT personnel, the prevailing myth that the air gap is an effective single security solution, and a lack of understanding the problem are all cited as major findings of the research group.

The group recommends a number of actions which need to be taken and these can be mapped along the Sliding Scale of Cyber Security. A big focus is placed on better designing the systems to have security built into them which can be understood in the Architecture phase of the scale. Another focus was on leveraging whitelisting and intrusion detection systems as well as other Passive Defense mechanisms instead of just an air gap. Lastly, one of the most significant recommendations was towards getting more personnel trained in cybersecurity practices (SANS offers ICS410 and ICS515 to address these types of concerns) and take a proactive approach versus a reactive approach towards finding threats in the environment — this recommendations maps to the Active Defense component of the scale which focuses on empowering analysts and security personnel to hunt for and respond to threats.

One of the more interesting major recommendations put forth by the report was:
“The infrequency of cyber security incident disclosure at nuclear facilities makes it difficult to assess the true extent of the problem and may lead nuclear industry personnel to believe that there are few incidents. Moreover, limited collaboration with other industries or information-sharing means that the nuclear industry tends not to learn from other industries that are more advanced in this field.”

At SANS we have consistently observed this as an issue in the wider community and try to bring the community together with events such as the ICS Summit to help address the concernand promote community sharing. No single event or effort alone though can fix the problem. A lack of information sharing and incident disclosure has led to a false sense of security while also allowing fake or hyped up stories in news media to become the representation of our industry to people in our community and external to it.

This aspect of infrequency of cyber security incident disclosure can be observed in multiple places. As an example, an article from 2014 by Inside Energy compiled incident reporting to the Department of Energy about electric grid outages and over 15 years noted that there were 14 incidents related to a cyber event. The earliest cyber attack was identified in 2003 but then there was a lack of events until 2011-2014 which made up the other 13 cases. It should be noted that the reporting for a cyber attack was any type of unauthorized access to the system including the hardware, software, and data.

We in the industry need to have better data so that we can more fully understand and categorize attacks along models such as the ICS Cyber Kill Chain to extract lessons learned. What is revealing about the Department of Energydata though is the lack of visibility into the ICS networked environment. As an example, in the data set there is a measured understanding of impact for physical attacks, fires, storms, etc. showing great visibility into the ICS as a whole but for every single event regarding cyber the impact was either labeled as zero or unknown; that in combination with no data for 2003-2011 is less representative of the number of events and more representative of missing data. It has become clear over the years that a significant number of ICS organizations do not have personnel that are trained and empowered to look into the network to find threats. This must change and the findings must be shared, anonymously and appropriately, with the community if we are ever to scope the true threat in the community and determine the appropriate resource investments and responses to address the issues.

The ICS community stands a unique opportunity to have our story told by our ICS owners, operators, and security personnel to understand and address the problem ourselves. Valuable compilations of data such as that by Inside Energy using the Department of Energy reports as well as the Chatham House report help reinforce this need. Without involvement from the community, the ICS security story will be told by others who may not have the appropriate experience to make the right conclusions and offer helpful solutions. The need for cyber security will influence change in the ICS community through national level policies, regulations, vendor practices, and culture shifts – it is imperative that the right people with real data are writing the story that will drive those changes.

Three Takeaways from the State of Security in Control Systems Survey

July 7, 2015

This was first posted on the SANS ICS blog here.

 

The State of Security in Control Systems Today was a SANS survey conducted with 314 ICS community members and was released on June 25th. The whitepaper can be found here and the webcast here. A few things stuck out from the survey that I felt it appropriate to highlight in this blog.

  1. Energy/Utilities Represent

Energy/Utilities made up the most of the respondents with 29.3% in total. While the variables impacting this cannot be narrowed down it is likely that pressure from organizations such as NERC, heavy focus on energy protection in the U.S. in national media and politics, and market interest has at least driven security awareness. We also see an energy bias in other metrics on reporting such as the ICS-CERT’s quarterly reports. This is a both a good thing and an area for improvement. It is great to see the energy sector get heavily involved in events such as this survey, in training conferences, and major events like the electric sector’s GridEx. Personally, I’ve interacted with groups such as the ES-ISAC and been extremely impressed. Getting data from this segment of the community helps understand the problem better so that we can all make the appropriate investments in security.

Takeaway: We really need to do more to reach the other communities. Energy tends to be a hot topic item but it is far from the only industry that has security issues. Each portion of the ICS community from water to pharmaceuticals face similar issues. In the upcoming years hopefully reports like this SANS survey will be able to capture more of those audiences. I feel this is likely given the increased awareness in other industries I have seen even in the last few years.

 

  1. IT/OT Convergence Seen as 2nd Most Likely Threat

The number one vector the respondents felt was the most significant threat to their ICS was external threats. This makes sense given the increased understanding in the community regarding external actors and the cyber security of operations. However, interestingly the second top threat identified as the integration of IT into control system networks. I really liked seeing this metric because I too believe it presents one of the largest threat vectors to operations. ICS targeted nation state malware tends to get the most media attention. BlackEnergy2, Stuxnet, and Havex were all very concerning. However, it is far more likely on a day to day basis that not architecting and maintaining the network correctly will lead to decreased or stopped operations. The integration of OT and IT also presents a number of challenges with incidental malware that, while non-targeted, presents a significant risk as has been documented numerous times when important systems halt due to accidental malware infections such as Conficker.

Takeaway: The ICS community needs to be aware of external threats and realize that they pose the most targeted threat to operations. However, it was great seeing that issues revolving around the integration of IT and OT is accurately seen as a concern. Architecting and maintaining the OT network correctly to include safe and segmented integration, structuring such as the Purdue model, and ultimately reducing the risks associated with IT/OT convergence will go a long way for the security of the environment. The type of efforts required to reduce the risk of IT/OT convergence is also the same foundational efforts that help identify, respond, and learn from external threats and threat vectors.

 

  1. Lack of Visibility is Far Reaching

A significant portion of the group, 48.8%, stated that they simple did not have visibility into their environment. This could mean a number of things to include IT and OT not having visibility into each other’s processes and environment, lack of understanding of the networked environment, inability to collect data such as network traffic or logs, and a lack of a plan to pull together all stakeholders when appropriate. Each of these has been observed and continually documented as problems in the ICS community. What is interesting about this single metric though is that it impacts most of the other metrics. For example, respondents who do not have visibility into their environment will not be able to fully identify threats in their environment; 48.8% stated that they were not aware of any infiltration or infection of their control systems. Additionally, when a breach occurs it is difficult to respond correctly without visibility; 34% of the participants who had identified breaches stated that they had been breached multiple times in the last 12 months.

Takeaways: Nearly half of the respondents to the survey indicated that they did not have visibility into the environments. This makes it incredibly difficult to know if they have been impacted by breaches. It also makes it difficult to scope a threat and respond appropriately. I would bet that a significant portion of those participants who indicated they were breached multiple times had links between the breaches that they were unaware of due to a lack of visibility. Re-infections that occur due to not fully cleaning up after a breach are common in the IT and OT communities. ICS community members need to ensure that they are developing plans to increase their visibility. That means including all stakeholders (in both IT and OT), ensuring that at least sampling from the environment can be taken in the form of logs and network traffic, and talking with vendors to plan better visibility into system upgrades and refreshes. For example, a mirrored port on a network switch is a great resource to gain invaluable network traffic data from the OT environment that can help identify threats and reduce time and cost of incident response.

Follow on: To help with the discussion of visibility into the environment I will post two entries to the SANS ICS blog in the upcoming weeks. They will be focused on two of the beginning labs in SANS ICS515 — Active Defense and Incident Response. The first will cover using Mandiant’s free incident response tool: Redline and how to use it in an ICS to gather critical data. The second will cover using some basic features in Wireshark to sample network traffic and identify abnormalities.

Final Thoughts

I was very impressed with the participants of the SANS survey. Their inputs help give a better understanding into the community and its challenges. While the takeaways above focus on areas for improvement it is easy to look at the past few years and realize that security is increasing overall. Security awareness, trained security professionals, and community openness are all increasing. We have a long way to go in the community but we are getting better. However, there are many actions that can and should be taken today to drastically help security. First, we must be more open with data and willing to participate in spot checks, like surveys, on the community. Secondly, wherever there is a lack of a plan forward, such as IT/OT convergence strategies, the appropriate stakeholders need to meet and discuss with the intent to act. Thirdly, incidents are happening whether or not the community is ready for it. Appropriate visibility into the environments we rely on, incident response plans, and identified personnel to involve are all requirements. We can move the bar forward together.