Browsing Tag

critical infrastructure

Threats of Cyber Attacks Against Russia: Rationale on Discussing Operations and the Precedent Set

November 6, 2016

Reports that the U.S. government has military hackers ready to carry out attacks on Russian critical infrastructure has elicited a wide range of responses on social media. After I tweeted the NBC article a number of people responded with how stupid the U.S. was for releasing this information, or what poor OPSEC it was to discuss these operations, and even how this constitutes an act of war. I want to use this blog to put forth some thoughts of mine on those specific claims. However, I want to note in advance this is entirely my opinion. I wouldn’t consider this quality analysis or even insightful commentary but instead just my thoughts on the matter that I felt compelled to share since I work in critical infrastructure cyber security and was at one point a “military hacker.”

The Claim

The claim stems from an NBC article and notes that a senior U.S. intelligence official shared top-secret documents with NBC News. These top-secret documents apparently indicated that the U.S. has “penetrated Russia’s electric grid, telecommunications networks and the Kremlin’s command systems, making them vulnerable to attack by secret American cyber weapons should the U.S. deem it necessary.” I’m going to make the assumption that this was a controlled leak given the way that it was presented. Additionally, I make this assumption because of the senior officials that were interviewed for the wider story including former NATO commander (ret) ADM James G. Stavridis and former CYBERCOM Judge Advocate (ret) COL Gary Brown who likely would not have touched a true “leak” driven story without some sort of blessing to do so. I.e. before anyone adds that this is some sort of mistake this was very likely authorized by the President at the request of senior officials or advisers such as the Director of National Intelligence or the National Security Council. The President is the highest authority for deeming material classified or not and if he decided to release this information it’s an authorized leak. Going off of this assumption let’s consider three claims that I’ve seen recently.

The U.S. is Stupid for Releasing This Information

It is very difficult to know the rationale behind actions we observe. This is especially true in cyber intrusions and attacks. If an adversary happens to deny access to a server, did they intend to or was it accidentally brought down while performing other actions? Did the adversary intend to leave behind references to file paths and identifying information or was it a mistake? These debates around intent and observations is a challenge for many analysts that must be carefully overcome. In this case it is no different.

Given the assumption that this is a controlled leak it was obviously done with the intention of one or more outcomes. In other words, the U.S. government wanted the information out and their rationale is likely as varied as the members involved. While discussing a “government” it’s important to remember that the decision was ultimately in the hands of individuals, likely two dozen at the most. Their recommendations, biases, views on the world, insight, experience, etc. all contribute to what they expect the output of this leak to manifest as. This makes it even more difficult to assess why a government would do something since it’s more important to know the key members in the Administration, the military, and the Intelligence Community and their motivations rather than the historical understanding of government operations and similar decisions. Considering the decision was likely not ‘stupid’ and more for some intended purpose let’s explore what two of those purposes might be:

Deterrence

I’m usually not the biggest fan of deterrence in the digital domain as it has since not been very effective and the qualities to have a proper deterrent (credible threat and an understood threshold) are often lacking. Various governments lament about red lines and actions they might do if those red lines are crossed but what exactly those red lines are and what the response action will be if they are crossed is usually never explored. Here however, the U.S. government has stated a credible threat: the disruption of critical infrastructure in Russia (the U.S. has shown before that they are capable of doing this). They have combined this with a clear threshold of what they do not want their potential adversary to do: do not disrupt the elections. For these reasons my normal skepticism around deterrence is lessened. However, in my own personal opinion this is potentially a side effect and not the primary purpose especially given the form of communication that was chosen.

Voter Confidence

Relations between Russia and the U.S. this election have been tense. Posturing and messaging between the two states has taken a variety of forms both direct and indirect. This release to NBC though is interesting as it would be indirect messaging if positioned to the Russian government but it would be direct messaging if intended for the U.S. voters. My personal opinion (read: speculation) is that it is much more intended for the voters. At one point in the article NBC notes that Administration officials revealed to them that they delivered a “back channel warning to Russia against any attempt to influence next week’s vote”. There’s no reason to reiterate a back channel message in a public article unless the intended audience (in this case the voters) weren’t aware of the back channel warning. The article reads as an effort by the Administration to tell the voters: “don’t worry and go vote, we’ve warned them that any effort to disrupt the elections will be met with tangible attacks instead of strongly worded letters.”

It’s really interesting that this type of messaging to the American public is needed. Cyber security has never been such a mainstream topic before especially not during an election. This may seem odd to those in the information security community who live with these discussions on a day to day basis anyway. But coverage of cyber security has never before been mainstream media worthy for consistent periods of time. CNN, Fox, MSNBC, and the BBC have all been discussing cyber security throughout the recent election season ranging from the DNC hacks to Hillary’s emails. That coverage has gotten fairly dark though with CNN, NBC, Newsweek, and New York Times articles like this one and prime time segments telling voters that the election could be manipulated by Russian spies.

This CNN piece directly calls out the Kremlin for potentially manipulating the elections in a way that combines it with Trump’s claims that the election is rigged. This is a powerful combination. There is a significant portion of Trump’s supporters who will believe his claim of a rigged election and in conjunction with the belief that Russia is messing with the election it’s easy to see how a voter could become disillusioned with the election. Neither the Democrats or Republicans want less voters to turn out and (almost) all of those on both sides want the peaceful transition of power after the election as has always occurred before. Strong messaging from the Administration and others into mainstream news media is important to restore confidence to voters both in the election itself as well as the manner to which people vote.

Unfortunately, it seems that this desire is being accidentally countered by some in the security community. In very odd timing, Cylance decided to release a press release on vulnerabilities in voting machines on the same day, unbeknownst to them, as the NBC article. The press release stated that the intent of the release was to encourage mitigation of the vulnerabilities but with 4 days until the election, as of the article’s release, that simply will not be possible. The move is likely very well intended but unlikely to give voters much confidence in the manner to which they vote. I’ll avoid a tangent here but it’s worth mentioning the impact security companies can play on larger political discussions.

The Leak is Bad OPSEC

I will not spend as much time on this claim as I did the previous but it is worth noting the reaction that releasing this type of information is bad operational security. Operational security is often very important to ensure that government operations can be coordinated effectively without the adversary having the advance warning required to defend against the operation. However, in this case the intention of the leak is likely much more around deterrence or voter confidence and therefore the operation itself is not the point. Keeping the operation secret would not have helped either potential goal. More importantly, compromising information systems is not something that has ever been see as insurmountably difficult. For the U.S. government to reveal that it has compromised Russian systems does not magically make them more secure now. Russian defense personnel do not have anything more to go off of than before in terms of searching for the compromise, they likely already assumed they were compromised, and looking for a threat and cleaning it up across multiple critical infrastructure industries and networks would take more than 4 days even if they had robust technical indicators of compromise and insight (which the leak did not give them). The interesting part of the disclosure is not the OPSEC but in the precedence it sets which I’ll discuss in the next section.

The Compromises are an Act of War

Acts of war are governed under United Nations’ Article 2(4) where it discusses armed conflict. The unofficial rules regarding war in cyberspace are contained in the Tallinn Manual. In neither of these documents is the positioning of capabilities to do future damage considered an act of war. More importantly, in the NBC article it notes that the “cyber weapons” have not been deployed yet: “The cyber weapons would only be deployed in the unlikely event the U.S. was attacked in a significant way, officials say.” Therefore, what is being discussed is cyber operations that have gained access to Russian critical infrastructure networks but not positioned “weapons” to do damage yet. Intrusions into networks have never been seen as an act of war by any of the countries involved in such operations. So what’s interesting about this?  The claim by officials that the U.S. had compromised Russian critical infrastructure networks including the electric grid years ago.

For years U.S. intelligence officials have positioned that Russian, Chinese, Iranian, and at times North Korean government operators have been “probing” U.S. critical infrastructure such as the power grid. The pre-positioning of malware in the power grid has long been rumored and has been a key concern of senior officials. The acknowledgment in a possibly intended leak that the U.S. has been doing the same for years now is significant. It should come as no surprise to anyone in the information security community but as messaging from senior officials it does set a precedent internationally (albeit small given that this is a leak and not a direct statement from the government). Now, if capabilities or intrusions were found in the power grid by the U.S. government in a way that was made public the offending countries could claim they were only doing the same as the U.S. government. In my personal experience, there is credibility to claims that other countries have been compromising the power grid for years so I would argue against the “U.S. started it” claim that is sure to follow.  The assumption is that governments try to compromise the power grid ahead of time so that when needed they can damage it for military or political purposes. But the specific compromises that have occurred have not been communicated publicly by senior officials nor have they been done with attribution towards Russia or China. The only time a similar specific case was discussed with attribution was against Iran for compromising a small dam in New York and the action was heavily criticized by officials and met with a Department of Justice indictment.  Senior officials’ acknowledgment of U.S. cyber operations compromising foreign power grids for the purpose of carrying out attacks if needed is unique and a message likely heard loudly even if later denied. It would be difficult to state that the leak will embolden adversaries to do this type of activity if they weren’t already but it does in some ways make the operations more legitimate. Claiming responsibility for such compromises while indicting countries for doing the same definitely makes the U.S. look hypocritical regardless of how its rationalized.

Parting Thoughts

My overall thought is that this information was a controlled leak designed to help voters feel more confident in terms of both going to cast their ballots and in the overall outcome. Some level of deterrence was likely a side effect that the Administration sought. But no, this was not simply a stupid move nor was it bad OPSEC or an act of war. I also doubt it is simply a bluff. However, there is some precedent set and pre-positioning access to critical infrastructures around the world just became a little more legitimate.

One thing that struck me as new in the article though was the claim that the U.S. military used cyber attacks to turn out the lights temporarily in Baghdad during the 2003 Iraq invasion. When considering the officials interviewed for the story and the nature of the (again, possibly) controlled leak that is a new claim from senior government officials. There was an old rumor that Bush had that option on the table when invading Iraq but the rumor was the attack was cancelled for fear of the collateral damage of taking down a power grid. One can never be sure how long “temporary” might be when damaging such infrastructure. The claim in the article that the attack actually went forward would make that the first cyber attack on a power grid that led to outages – not the Ukrainian attack of 2015 (claims of a Brazilian outage years earlier were never proven and seem false from available information). However, the claim is counter to reports at the time that power outages did not occur during the initial hours of the invasion. Power outages were reported in Iraq but after the ending of active combat operations and looters were blamed. If a cyber attack in Iraq ever made sense militarily it would not have made as much sense after the initial invasion.

I’ve emailed the reporter of the story asking what the source of that claim was and I will update the blog if I get an answer. It is possible the officials stated this to the reporters but misspoke. In my time in the government it was not a rare event for senior officials to confuse details of operations or hear myths outside of the workplace and assume them to be true. Hopefully, I can find out more as that is a historically significant claim. Based on what is known currently I am skeptical that outages following the initial Iraq invasion in 2003 were due to a cyber attack.

Minimum Viable Products are Dubious in Critical Infrastructure

December 4, 2015

Minimum Viable Products in the critical infrastructure community are increasingly just mislabeled BETA tests; that needs to be communicated correctly.

The concept of a Minimum Viable Product (MVP) is catching on across the startup industry. The idea of the MVP is tied closely to The Lean Startup model created by Eric Ries in 2011 and has very sound principals focused around maximizing the return on investment and feedback from creating new products. Eric defines the MVP as the “version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort.” This enforces the entrepreneurial spirit and need for innovation combined with getting customer feedback about a new technology without having to develop a perfect plan or product first. An MVP is also meant to be sold to customers so that revenue is generated. In short, be open to testing things out publicly earlier, pivot based off of technical and market feedback, and earn some money to raise the valuation of the company and entice investors.

Personally, I believe the lean startup model as a whole is smart. I use some aspects of the model as CEO of Dragos Security. However, I chose not to use the concept of an MVP. Minimum Viable Products are dubious in critical infrastructure. I state this understanding that the notion of getting the product out the door and gaining feedback to guide its development is a great idea. And when I say critical infrastructure I’m focusing heavily on the industrial control system (ICS) portion of the community (energy, water, manufacturing, etc.). The problem I have though is that I have observed a number of startups in the critical infrastructure startup space taking advantage of their customers, albeit unintentionally, when they push out MVPs. This is a bold claim, I won’t point fingers, and I don’t want to come across as arrogant. But I want to make it very clear: the critical infrastructure community deals with human lives; mistakes, resource drains, and misguiding expectations impact the mission.

My observations of the startups abusing the MVP concept:

  • Bold claims are made about the new technologies seemingly out of a need to differentiate against larger and more well established companies
  • Technologies are increasingly deployed earlier in the development cycle because the startups do not want to have to invest in the industry specific hardware or software to test the technology
  • The correct customers that should be taking part in the feedback process are pushed aside in favor of easier to get customers because successes are needed as badly as cash; there is pressure to validate the company’s vision to entice or satisfy Angel or Seed investors
  • The fact that the technology is an MVP, is lightly (if at all) tested, and will very likely change in features or even purpose is not getting communicated to customers in an apparent attempt to get a jump start on the long acquisition cycles in critical infrastructure and bypass discussions on business risk
  • Customers are more heavily relied upon for feedback, or even development, costing them time and resources often due to the startups’ lack of ICS expertise; the startup may have some specific ICS knowledge or general ICS knowledge but rarely does it have depth in all the markets its tackling such as electric, natural gas, oil, water, etc. although it wants to market and sell to those industries

What is the impact of all this? Customers are taking bigger risks in terms of time, untested technologies, changing technologies, and over hyped features than they recognize. If the technology does not succeed, if the startup pivots, or if the customers burn out on the process all that’s been accomplished is significant mistrust between the critical infrastructure stakeholders and their desire to “innovate” with startups anymore. And all of this is occurring on potentially sensitive networks and infrastructure which have the potential to impact safety or the environment.

My recommendations to startups: if you are going to deploy technologies into critical infrastructure early in the development cycle make sure the risks are accurately conveyed and ensure that the customer knows that they are part of a learning process for your technology and company. This begs instant push-back: “If we communicate this as a type of test or a learning process they will likely not trust our company or technology and choose to go with other more established products and companies. We are trying to help. We are innovators.” And to my straw man here, I empathize greatly. Change is needed in this space and innovation is required. We must do better especially with regards to security. But even if we ignore the statistics around the number of failed technologies and startups that would stress why many should never actually touch an ICS environment I could comfortably state that the community is not as rigid as folks think. The critical infrastructure community, especially in ICS, gets cast in a weird light by many outside the community. My experience shows that the critical infrastructure community is just as innovative, and I would argue more so, than any other industry but they are much more careful to try to understand the potential impact and risks…as they should be.

My experience in a new technology startup: when the Dragos Security team was developing our CyberLens software we needed to test it out. Hardware was expensive and we could not afford to build out networks for every type of vendor’s ICS hardware and network communications. Although we have a lot of ICS knowledge on the team we all were keenly aware that we are not experts in every aspect of every ICS industry we wanted to sell to. Customer feedback was (and still is) vital. To add to this we were pressed because we were competing with larger more established companies and technologies but on a very limited budget. So, instead of trying to sell an MVP we simply launched a BETA instead; the BETA lasted over twelve months. How did we accomplish this? We spent $0 on marketing and sales and focused entirely on staying lean and developing and validating our technology. We made contacts in the community, educated them on what we wanted to do, advised where the technology was tested and safe to deploy, and refused to charge our BETA participants for our time or product since they were greatly helping us and keeping our costs down. In turn we offered them discounts for when our product launched and offered some of our time to educate them in matters we did have expertise. This created strong relationships with our BETA participants that carried over when we launched our product to have them join us as customers. We even found new customers when we launched based on referrals from BETA participants vouching for our company. Or more simply stated: we were overly honest and upfront, avoided hype and buzzwords, and brought value so we were seen as fellow team members and not snake oil salesmen. I recommend more startups take this approach even under pressure and when it is difficult to differentiate in the market.

My conclusion: the MVP model in its intended form is not a bad model. In many communities it is an especially smart model. And just because a company is using an MVP route in this space does not mean they are abusing anyone or falling into the pitfalls I listed above. But, as a whole, in the critical infrastructure community it is a process that is more often abused than used correctly and it is damaging in the long term. Customers are not cash cows and guinea pigs – they are investors in your vision and partners. Startups should still push out technologies earlier than trying to wait to create a perfect product without the right feedback, but call these early pushes what they are. It is not a Minimum Viable Product it is a BETA test of core features. Customers should not be asked to spend limited budgets on top of their time and feedback for it nor should they be misled as to what part of the process they are helping in. You will find the community is more likely to help when they know you are being upfront even with understandable shortcomings.

Three Takeaways from the State of Security in Control Systems Survey

July 7, 2015

This was first posted on the SANS ICS blog here.

 

The State of Security in Control Systems Today was a SANS survey conducted with 314 ICS community members and was released on June 25th. The whitepaper can be found here and the webcast here. A few things stuck out from the survey that I felt it appropriate to highlight in this blog.

  1. Energy/Utilities Represent

Energy/Utilities made up the most of the respondents with 29.3% in total. While the variables impacting this cannot be narrowed down it is likely that pressure from organizations such as NERC, heavy focus on energy protection in the U.S. in national media and politics, and market interest has at least driven security awareness. We also see an energy bias in other metrics on reporting such as the ICS-CERT’s quarterly reports. This is a both a good thing and an area for improvement. It is great to see the energy sector get heavily involved in events such as this survey, in training conferences, and major events like the electric sector’s GridEx. Personally, I’ve interacted with groups such as the ES-ISAC and been extremely impressed. Getting data from this segment of the community helps understand the problem better so that we can all make the appropriate investments in security.

Takeaway: We really need to do more to reach the other communities. Energy tends to be a hot topic item but it is far from the only industry that has security issues. Each portion of the ICS community from water to pharmaceuticals face similar issues. In the upcoming years hopefully reports like this SANS survey will be able to capture more of those audiences. I feel this is likely given the increased awareness in other industries I have seen even in the last few years.

 

  1. IT/OT Convergence Seen as 2nd Most Likely Threat

The number one vector the respondents felt was the most significant threat to their ICS was external threats. This makes sense given the increased understanding in the community regarding external actors and the cyber security of operations. However, interestingly the second top threat identified as the integration of IT into control system networks. I really liked seeing this metric because I too believe it presents one of the largest threat vectors to operations. ICS targeted nation state malware tends to get the most media attention. BlackEnergy2, Stuxnet, and Havex were all very concerning. However, it is far more likely on a day to day basis that not architecting and maintaining the network correctly will lead to decreased or stopped operations. The integration of OT and IT also presents a number of challenges with incidental malware that, while non-targeted, presents a significant risk as has been documented numerous times when important systems halt due to accidental malware infections such as Conficker.

Takeaway: The ICS community needs to be aware of external threats and realize that they pose the most targeted threat to operations. However, it was great seeing that issues revolving around the integration of IT and OT is accurately seen as a concern. Architecting and maintaining the OT network correctly to include safe and segmented integration, structuring such as the Purdue model, and ultimately reducing the risks associated with IT/OT convergence will go a long way for the security of the environment. The type of efforts required to reduce the risk of IT/OT convergence is also the same foundational efforts that help identify, respond, and learn from external threats and threat vectors.

 

  1. Lack of Visibility is Far Reaching

A significant portion of the group, 48.8%, stated that they simple did not have visibility into their environment. This could mean a number of things to include IT and OT not having visibility into each other’s processes and environment, lack of understanding of the networked environment, inability to collect data such as network traffic or logs, and a lack of a plan to pull together all stakeholders when appropriate. Each of these has been observed and continually documented as problems in the ICS community. What is interesting about this single metric though is that it impacts most of the other metrics. For example, respondents who do not have visibility into their environment will not be able to fully identify threats in their environment; 48.8% stated that they were not aware of any infiltration or infection of their control systems. Additionally, when a breach occurs it is difficult to respond correctly without visibility; 34% of the participants who had identified breaches stated that they had been breached multiple times in the last 12 months.

Takeaways: Nearly half of the respondents to the survey indicated that they did not have visibility into the environments. This makes it incredibly difficult to know if they have been impacted by breaches. It also makes it difficult to scope a threat and respond appropriately. I would bet that a significant portion of those participants who indicated they were breached multiple times had links between the breaches that they were unaware of due to a lack of visibility. Re-infections that occur due to not fully cleaning up after a breach are common in the IT and OT communities. ICS community members need to ensure that they are developing plans to increase their visibility. That means including all stakeholders (in both IT and OT), ensuring that at least sampling from the environment can be taken in the form of logs and network traffic, and talking with vendors to plan better visibility into system upgrades and refreshes. For example, a mirrored port on a network switch is a great resource to gain invaluable network traffic data from the OT environment that can help identify threats and reduce time and cost of incident response.

Follow on: To help with the discussion of visibility into the environment I will post two entries to the SANS ICS blog in the upcoming weeks. They will be focused on two of the beginning labs in SANS ICS515 — Active Defense and Incident Response. The first will cover using Mandiant’s free incident response tool: Redline and how to use it in an ICS to gather critical data. The second will cover using some basic features in Wireshark to sample network traffic and identify abnormalities.

Final Thoughts

I was very impressed with the participants of the SANS survey. Their inputs help give a better understanding into the community and its challenges. While the takeaways above focus on areas for improvement it is easy to look at the past few years and realize that security is increasing overall. Security awareness, trained security professionals, and community openness are all increasing. We have a long way to go in the community but we are getting better. However, there are many actions that can and should be taken today to drastically help security. First, we must be more open with data and willing to participate in spot checks, like surveys, on the community. Secondly, wherever there is a lack of a plan forward, such as IT/OT convergence strategies, the appropriate stakeholders need to meet and discuss with the intent to act. Thirdly, incidents are happening whether or not the community is ready for it. Appropriate visibility into the environments we rely on, incident response plans, and identified personnel to involve are all requirements. We can move the bar forward together.

Barriers to Sharing Cyber Threat Information Within the Critical Infrastructure Community

June 28, 2015

This was first posted on the Council of Foreign Relations’ blog Net Politics here.

 

The sharing of cyber threat data has garnered national level attention, and improved information sharing has been the objective of several pieces of legislation and two executive orders. Threat sharing is an important tool that might help tilt the field away from adversaries who currently take advantage of the fact that an attack on one organization can be effective against thousands of other organizations over extended periods of time. In the absence of information sharing, critical infrastructure operators find themselves fighting off adversaries individually instead of using the knowledge and experience that already exists in their community. Better threat information sharing is an important goal, but two barriers, one cultural and the other technical, continue to plague well intentioned policy efforts. Failing to meaningfully address both barriers can lead to unnecessary hype and the misappropriation of resources. Continue Reading