Goodbye Mike Assante, Thank you For Literally Everything

July 6, 2019

On July 5th, 2019 Mike Assante decided it was his time to leave this world and shuffle off this mortal coil. I say decided because as I would learn from Tim Conway, Mike didn’t want to die on July 4th because it would ruin the festivities of the holiday (he was deeply patriotic) and he didn’t want to die on July 6th which is his wife’s birthday. So essentially Mike chose the 5th. That’s the kind of stuff we make up about people to pretend they’re a badass, but that was just another true story and small feat by Mike. Mike didn’t lose his battle to cancer, he kicked its ass a decade ago, it came back, and he told it “no you’re going to wait your turn.” That’s as much winning as can be had.

A lot has been said about Mike and a lot will continue to be said by many. The mere fact is he impacted an unreal number of people and we’ll all spend years figuring out how much stuff he actually did behind the scenes. I don’t pretend to be able to offer unique insights or thoughts, I only know that this blog is therapeutic for me and hopefully one day useful to his kids. In the military they train you pretty early on that if one of your buddies dies in combat you are suppose to capture your thoughts and write them down. The therapy of this is important but it’s the memories and stories for the kids that are the most important. To see their loved one through someone else’s eyes. I feel some solace in knowing that Mike’s family will have letters and memories to read that could fill books worth of exciting tales most that seem to be something out of fiction.

So here’s my therapy and contribution to the stories.

Mike was everything to me. I never could place what he was exactly. Part father like, part brother, part mentor, part brother in arms, part best friend, and part inspiration. I have never been impacted by death. It sounds odd to say it, I have confided in my wife that I think I’m a heartless person or maybe just ultimately at peace because I’ve lost friends, family, and military brothers before; and not once did I ever cry. It didn’t really impact me. I know it sounds odd, I can’t explain it. But I don’t even go to funerals not because it’s too hard but because I look like the odd one out while everyone else is sad and I’m just moving on. It sounds all sorts of screwed up to say out loud which is why I always avoid the topic altogether. But since Mike got sick again I have spent more nights crying than I can count. This is one of those blogs you cry-type and then realize you have to delete most of it because you made a bunch of mistakes through the tears. Mike was unique. In our last call together he and I talked of the future and what he wanted to see for the community and I sought guidance on my own path and how to honor him (I tried to stay away while Mike was sick, his time with his family was most important, but I felt the need to be selfish at the end and steal some of his time). We talked for awhile until we both started to break down in tears so we abruptly hung up on each other. I cannot put into words how much this hurts. I only know that feeling this way feels selfish when I think about his wife and kids and how they feel now.

Mike had more of an impact on me than any single other person. I remember in 2011 I gave my first public conference talk, it was at Joe Weiss’ ICS Cybersecurity Conference (WeissCon). I was doing some unique things in ICS security at the time not only focusing on finding new threats but at the time also looking at remotely piloted aircraft and the control systems on them, satellites, and some other interesting stuff. At the conference Mike came up to me and we chatted about what things I was seeing. I thought they were unique at least. Mike of course was always about 100 miles ahead on any topic you wanted to talk about “hey Mike did you know there’s $x equipment on $weaponsystem” and he’d listen and brainstorm with you even though he’d not already known but the reason you were working on any given project was likely he mentioned its importance to someone before. (As an aside, I always thought I’d one up him, I’d brief the White House and brag to him “oh did you see John?” he’d casually say about some ridiculously high ranking person. I remember I thought I had him when I testified to the Senate and was invited to some closed planning sessions, as I bragged to him “oh did you see Lisa? Tell her I said hi” he’d say about a sitting US Senator. Or I’d be invited to some highly classified special access program to work on something important only to find out it existed because Mike had told the USG it was something important to do. Mike could *not* be one upped. Ever. He was always ahead of you, period.) We chatted about the problems of the ICS security community. At the time I was in a weird and alone place in the USG. I didn’t have mentors on this topic or others looking at ICS security. I know the Air Force and National Security Agency were looking at ICS but my focus was on finding the threats targeting ICS, not vulnerabilities and hardening the systems and not, at the time, targeting them. No one in my government circles was doing this and we didn’t even have a name for it then. But Mike not only spoke my language, he knew how to encourage me. We opined about the threats together and what could be done as well as what needed to be done. He and I both settled on the need for education and training. I told him one of the hardest challenges I had in the military and intelligence community was having any sort of structured onboarding or training for folks to pick up these skills. Mike talked to me about SANS and the need to do something there. That conversation would then be carried out in emails and conversations here and then until he finally convinced me to do something about it.

In 2013 Mike convinced me to come share some thoughts at the SANS ICS Summit to take place in the following spring (March 2014). I didn’t feel I had a lot to share. Anything good was related to targeted cases of ICS threats and what we were doing at the NSA to counter them; couldn’t exactly put that into slides (I wouldn’t ever disclose I was at the NSA until post Snowden leaks and a Russian group piecing together enough metadata to call me and a couple others out publicly, no use denying it at that point). I had written a book titled SCADA and Me and Mike wanted me to come talk about that and how to talk to people about ICS security. In the Air Force pretty much every flag officer is a pilot; if you can explain it to a pilot you can explain it to anyone. So I made one of the all time horrible presentations on educating your management about SCADA security. For whatever reason it was a hit but I attribute that to the fact that for half my presentation Mike and I just called out to each other, him from the peanut gallery and me on stage, making fun of each other (him for being in the Navy of course, an inferior choice in life). It was there that my life truly changed. Every path I would take from that moment was an offshoot of the one he put me on. He made sure I was properly introduced to Andy Bochman, Ben Miller, Tim Roxey, Tim Conway, Marty Edwards, Bill Lawrence, and others. I want to say we had crossed paths before but not like this. Tim Conway, Mike, and I sat in the corner and talked about that 2011 conversation I had with Mike and what was needed. They asked me my thoughts about ICS410 which I thought was a really useful class to the community but not what I needed for folks going onto my mission. I had by then really fleshed out the ICS threat discovery mission at the NSA and had led it for long enough now I had a decent grasp on the skills needed. It was a mix of threat intelligence, discovery/hunting, malware analysis, and incident response skills. Mike and Tim convinced me I needed to make a class for SANS since it didn’t exist. Of course they failed to mention just how hard that is or how much time it was going to take “oh you’re an active duty officer? Psh you’ll have time good luck man.” I pretty much gave up the next 7 months of my life while I was under the impression that this shouldn’t take long and I was somehow a slacker for not being done yet. As a young officer stationed in Germany I should have been traveling and getting drunk (more); instead I was building ICS ranges in my basement with parts from Ebay.

There were so many things missing from the space though on this topic. Every time I wanted to talk about a topic and looked for foundation material I found it missing. So Mike and I spent so much time together brainstorming so we could come up with models and foundation content to build what 515 needed to be. One of the efforts we’d work on was the ICS Cyber Kill Chain where we discussed what ICS cyber attacks really looked like. There was a sense of optimism in the paper on how ICS ends up being some of the most defensible environments in the planet. But we also captured some of the thoughts to paper we had previously only discussed over too much whiskey; we both would always quietly confide in each other that the community wasn’t ready for what was about to come. As an example, somewhere around the time the CEO of NERC (Gerry) made a comment in a testimony to Congress that a power outage could only really last a few hours, maybe a couple of days, that in his estimates he “couldn’t imagine a cyber attack taking down electric power for more than 2 weeks.” I remember Mike and I texting back and forth during it “Sorry Gerry the adversary is unfortunately not limited by your imagination.” Of course Mike was nicer than I was but it was fun as we proceeded to meet up and talk about how we’d take down the bulk electric system in key locations and how long we could do it for; not simply high level theory but what exactly would we do. We figured we could get to about a month of down time across major portions of the grids. We thought about the same for oil and gas infrastructure and designed, in theory, some attacks against SIS that would facilitate loss of life and damaging of key infrastructure. It was through this that we could think about what we would defend against and how and what it would take; the ICS Cyber Kill Chain paper hinted at the scenarios we discussed very lightly (you don’t want to give an adversary a playbook on how to be evil, most of them aren’t as smart as you think) and a lot of what we put into play March 2014-October 2014 was meant to be forward looking for years without the expectation we’d see these types of attacks for a long time. (Little did we know)

The SANS ICS515 class debuted at GridSecCon in San Antonio, Texas. What I didn’t appreciate at the time was Mike set me on a path where I’d have to make changes in my life. The amount of work SANS ICS515 required as well as the number of times it needed taught was significant. But the real impact was that having that class propelled me into a very hungry community looking for those with experience especially on the “threat” side of ICS security where up and to that point the broad discussions were mostly on vulnerabilities. I spent all my leave (military vacation time) in various companies’ ICS and teaching ICS515. Every week was a conversation with Mike and what needed done. I grew up in a military family. My mother and father are both retired Senior Master Sergeants (E8) in the Air Force. The Air Force is all I ever wanted in life. But Mike gave me a mission that the USG couldn’t compete with not with all the accesses and capabilities and camaraderie in the world. It was a mission to change the community, educate them, empower them, and protect civilization. He set me on a journey that would have me leave the military; it wasn’t his intention but the experiences I was gaining came in direct conflict with what the military wanted me to do after I left the NSA and went “back” to the Air Force doing an offensive mission under CYBERCOMMAND. It put me at odds with the mission and where I saw my contribution to the community. I knew I had to get out.

With the knowledge I was leaving the military I felt comfortable proposing to my girlfriend. We’d get engaged at the SANS ICS Summit in Disney World in March 2015. It was Mike that pep talked me before I proposed. I would leave the military the last day of August 2015. As I teared up as I drove off the base for the last time it was my father and Mike that I called. Now, as a free agent, I was also free and unrestricted when the cyber attack on Ukraine in December 2015 occurred. Because of the class I had the connections and had trained people abroad and was asked to get involved. With Mike, Tim, and a number of other folks around the community we jumped in head first and had a response faster than anyone else with a more measured understanding. There is so much more to the story of Ukraine 2015 than anyone will ever know. This is the case for so many things Mike did behind the scenes that’ll never be public. Only a few of us will ever fully understand or appreciate what Mike did. What he was prepared to lose, the drama that came with it, and the fights that had to be fought behind the scenes. It was the making of a movie from lies to deceit to politics up to the White House itself. And yet I know Mike always was the one to calm me down. I love the USG, don’t get me wrong, but I was ready to go to war with them because of what happened behind the scenes to Mike. But it was Mike that was the cooler mind on it all anyway. In the end he was right, it was all for the greater benefit for us to all play nice’ish in public. It would take me years to calm down; he was cool nearly immediately.

Because of my experience with Mike and Ukraine and my experiences in the ICS515 class he asked me to build, I would find that education alone couldn’t scale fast enough in the face of the threats we were seeing. More was needed. That I reluctantly needed to do a software company. I really hated software companies for the most part, sure there are good ones but so many are snake oil salesmen and quick to take credit for smart users doing cool things that there software happened to be a part of; many just want to get rich which is something that truly doesn’t motivate me in life but just about everyone you meet forever assumes it is when you wear the title of vendor; the last thing I wanted to become was a vendor. But in seeking guidance from Mike I knew it was important. I had previously had “Dragos Security LLC” with my cofounders Jon and Justin but it was really just a legal structure to protect us while we were still in USG when we made the CyberLens tool largely for my students in ICS515. Mike pushed me to do more and to make the leap and try to codify the lessons learned and knowledge into software, real software, that would actually help ICS security. So with his guidance and pushing I founded Dragos, Inc. in May 2016. That December, the Ukraine 2016 cyber attack would occur and now I had a full time team to get involved. And we were involved and stayed involved in every major ICS cyber attack and campaign since then. New insights yielded new conversations for Mike and I; with an intelligence team and incident response team backing us we were able to theorize even more. It accelerated our conversations. We never wrote those things down, we were busy after all, so I comforted myself at least; in our last conversation together before his death we both lamented that we didn’t do even more together. That was Mike for you, he literally changed the community in a way no one else could and literally changed my life forever in ever possible way…and in the end it was all about things left undone and how much more there was to do. Of course, his view was that the mission would carry on without him. My view is that the world will never appreciate how much is lost because he’s not here.

Building a class that would train thousands with critical skills, leaving the military, being in a position to marry my wife, being in a position to have a child (another thing I wouldn’t have done still in the military), starting a company, having a dedication to capturing lessons learned in every form possible for the community, and making life long friends. Everything I am today, Mike played a significant leading role in the story. Any success I will ever have will be a silent homage to him. There has never been someone other than my wife and parents that I’ve been closer to. And my love for Mike was still unique in comparison. I don’t want to write a bunch of stuff about me, but it’s important to understand that anything positive I ever do in this community, it was Mike. My world view and turn from “protect the USG” to “safeguard civilization” was Mike. It’s always been Mike. And Mike’s not here anymore. And I still can’t wrap my head around that. That and at the SANS ICS Summit, Tim and I will continue it on without Mike looking like idiots up on stage knowing that we cannot fill his shoes. He’d punch me by now if he was reading this “oh Rob don’t be like that” or some combination of telling me that I’m doing important things and I give him too much credit, because that was the humble beautiful man he was, but he’s wrong, and I have never given him enough credit, not publicly.

I don’t have real regrets with Mike. We should all be so lucky to get the kind of closure he had in the end. For the last few months when we all know he was going to die he was able to spend time with friends and family on his terms. I keep telling myself and others who ask me how I’m doing “we should all be so lucky to get the kind of closure he had in the end” it’s burned into my thoughts at this point. It’s my hollow echo to people who are kind enough to reach out because they know how much Mike meant to me. But even as I type them they feel like a fake guard I have up to my real thoughts which are just a combination of tears and missing my friend.

I have been able to confide more and more in Tim Conway. Probably one of the best things Mike ever did was introduce me to Tim. In him I’m able to still experience the mission Mike started at SANS and what some of our unfinished plans are. I wouldn’t be strong enough to do it alone. I love SANS don’t get me wrong but it’s difficult to keep teaching and run a company and have a family. Sometimes there’s drama at SANS and my colleagues at SANS know that I’ve…clashed…with some of the choices there and I’ve thought about quitting. And probably in our most sick and sadistic dark moment Tim and I joked through the tears that Mike knew that by dying he was locking Tim and I in forever at SANS. That we’d never be able to leave and we’d always feel this debt in life to complete his vision of a safer world not only through our own individual paths but through the joint path of training at SANS. What a prick. Always knowing what’s best for us even when we don’t. It makes me think of Graham Chapman’s funeral and John Cleese’s speech. That we would all be bereft not to take the opportunity to joke on Mike one last time. To shock the world. And yet if you look at John’s face by the end, it’s hollow and empty and the full of the understanding that his life will never be the same again as the moments he experienced with his friend Graham.

Mike you gave me everything. You made me better. I’ll try hard to live up to your legacy and to strive to be better. I never will get out from under your shadow and I wouldn’t have it any other way, somehow it’s brighter here.

 

Homogeneous Infrastructure and Scalable Attacks

June 14, 2019

Industrial (ICS/IIoT/OT/variations of “not enterprise IT + physics”) infrastructure has benefited greatly in one form or another in having heterogeneous infrastructure. An electric transmission substation in one part of a single company is different than an electric substation elsewhere even in the same company; not only often in vendor choices but configuration, integration, implementation, and physical process requirements. The differences between industries is even more vast.

When attackers want to train for an offensive mission they routinely train and prepare for the environment that they will face. This is universal across domains and is not unique to cyber. As attackers gain experience through repetition the ability to successfully repeat an attack increases. As attackers develop knowledge especially among seasoned professionals on the team they routinely codify that knowledge into software, or malware, to scale their efforts especially to increasingly new members on the teams. We have seen this as a community across numerous attacks, in the industrial community we saw the obvious expediting of knowledge from the Ukraine 2015 attack on the electric grid to the Ukraine 2016 attack when knowledge was codified in the form of the CRASHOVERRIDE malware. Tradecraft to achieve successful attacks, once made public, is no longer bound to the team that created it and creates a blueprint for other teams. The increasing aggression and innovation of adversaries does not out pace defenders but creates an environment that becomes dangerous quickly for teams that do not invest properly in security.

None of these observations are novel. I have in many ways been inspired in my career by many professionals before me. I have learned from so many of those around me. My years in the U.S. Intelligence Community shaped my biases and world view. The peers I made early in my career were true experts and I received a leg up in that regards. But I write this blog as I think of one individual specifically tonight, Michael Assante. Mike has been one of the single biggest influences on my knowledge of ICS cybersecurity. One of the observations I naturally came to from spending time on defense and offense was that the heterogeneous industrial infrastructure we have that lends itself to a sort of natural defense against highly scalable disruptive or destructive attacks is shifting. We are seeing an increasing trend of homogeneous infrastructure as our industrial control and automation vendors acquire one another, settle on common technologies, and otherwise seek common operating platforms and approaches. This isn’t a vendor issue, the pressure comes from customers, and a vicious circle forms. This has been going on for years and is not as easily explainable nor dismissible as simply “IT and OT convergence” although that surely plays a role.

There are numerous professionals that have thought about this challenge. I have hired many like minded individuals to Dragos that have come to similar observations and felt a call to action. But notably on this topic I refer to not only Mike but also Tim Roxey and Andy Bochman. Andy is fortunately evangelizing the work of Idaho National Labs on Cyber-informed Consequence-driven Engineering (CCE). CCE is a larger and more nuanced view of this topic but for the purpose of a short blog I will try to easily express it with the simple question: “should the system controlling critical functions of protection and safety related equipment also be able to run Minesweeper?” Or as Mike would say “it’s not analog systems, it’s next-generation-non-digital-assets.” We need to think critically about not reversing the trend of industry, on how we continue to support business needs, but how we as a community think about the evolution we are achieving and what risks to what specific systems are not worth accepting as society.

I often get asked about what I fear in the world of ICS. I often talk about municipalities and cooperative electric systems, gas compressor stations, and infrastructure sites of under funded but critically valuable infrastructure. I try to do this respectfully understanding the hard mission many have and admirably take on often under resourced to do so. But one of the strategic things I fear most is that the heterogeneous infrastructure we take advantage of now one day crosses the line of homogenous infrastructure to the point that once state-only cyber attacks become scalable enough for non-state actors. At that point, the dynamics shift drastically and irreversibly in a direction that poses significant risk to our world. We live with so much fear and hype around cyber attacks on infrastructure (seriously, the phishing email to the energy company isn’t killing anyone or taking down “the grid”) that I often try not to articulate those things that scare me most. However, as I go introspective tonight thinking deeply of my friend Mike and the personal health battle he is waging right now I cannot help but articulate, written to everyone, that I do fear the path we are on, with all the progress the industry is making (it is impressive to see what many of our infrastructure companies are doing), is met with equally risky decisions that are forming a world where homogeneous infrastructure is met with scalable cyber attacks in a way that is difficult to counter at a pace society can accept.

I will always stress that defense is doable. I hope we as a community can ensure that we understand that statement does not mean “defense is easy” or that “defense is inhereted” but instead that it is an achievable goal we must work towards. My contribution to Mike’s view of the world is the understanding that the threats are becoming increasingly aggressive and numerous. Adding in an intelligence-driven view of the world overlayed onto a consequence-driven view of the world can significantly increase our chance to prioritize risk reduction. As we balance the risks we accept with compensating controls we will continue to win. If we make strategic missteps in our community miscalculating the risk of our changing infrastructure though I fear the inevitable reality of an impactful infrastructure attack that leads to loss of human life.

 

 

 

Attribution is not Transitive – Tribune Publishing Cyber Attack as a Case Study

December 31, 2018

I made a number of tweets on this subject but then the voice of Richard Bejtlich entered my head and told me that all twitter threads should be a blog post, and here I am. This blog looks at the cyber attack on Tribune Publishing and the claims that North Korea is responsible as an opportunity to highlight that attribution is not a transitive property.

 

Shortly after Tribune Publishing lost operations and ability to print papers the press highlighted that there was a cyber attack. The attack was highlighted as a targeted attack by a nation-state. This was all related to one anonymous insider at the company telling the media. Thus, early on I, and many others on social media, called for calm and patience while the details became public. The details are still not public and the company hasn’t officially responded but an insider told media sources that the malware used in the attack was Ryuk which is a family of ransomware (Checkpoint did a great write-up on it here). Checkpoint did some great analysis on the malware and noted that there is commonality in some aspects of the malware and another family of malware called Hermes. They appropriately highlight that while Hermes has been attributed in use to Lazarus Group before, there are alternative explanations including the group who developed Ryuk having access to the source code of the Hermes malware. There are likely alternative hypotheses not explored here as well. However, that link is seemingly being used by others to draw the conclusion that Tribune Publishing was attacked by North Korea.

Here is Forbes making that claim. They are not the only ones though. Fortunately, some journalists took a different approach. Here the New York Times accurately notes that just because (and if) Ryuk was used doesn’t mean it has government ties (thanks David and Nicole!). They also introduce alternative hypotheses including Adam Meyers from CrowdStrike stating that CrowdStrike tracks an eastern european cyber crime group that leverages the malware.

So what is the logic that led to the North Korean claims and what lessons can we extract?

Seemingly, the logic leveraged was that Ryuk has a link to Hermes. Hermes has links to Lazarus Group. Lazarus Group has been attributed to North Korea. Therefore, all uses of Ryuk must be North Korea. That is transitive attribution and is an association fallacy.

The logic seems kind of sound though, so what’s the problem?

There are a few large issues at play for us to explore.

First, we all have a collection bias. I.e. what we analyze is based off what we collect. We cannot know the true extent of collection available, so it is common for analysts to assume their collection is pretty good in comparison to what’s available. In fact though, it’s almost always the opposite where our collection is much worse than we realize. If Ryuk or Hermes malware was leveraged by teams other than Lazarus that would pose a big issue for the attribution claims. The “uniqueness” of malware is directly tied to collection. In perfect collection you could factually state if malware is unique to one team or not. But without perfect collection, and no one has perfect collection, we must understand that malware may appear unique to a specific team but may not be unique to them at all. It just may be unique to them in our collection.

Second, the links Checkpoint drew were not definitive and had other hypotheses identified. Therefore, if an assessment is going to be drawn out for attribution purposes and not the malware analysis purposes done in their blog, we’d need to do a more structured assessment including more data sources such as additional intrusions and cluster those intrusions using some model like the Diamond Model for Activity Groups. Moving the malware usage to a cluster of intrusions would reveal more data points to then start working on a more structured assessment.

Third, I have not looked deeply into Hermes but we’d want to explore the connections Hermes had to Lazarus Group and do the same type of analysis on the links mentioned in the second point for Ryuk.

Fourth, Lazarus Group is a collection of clusters of intrusions from across multiple researchers, teams, and organizations. Whereas Lazarus Group was at one point decently well defined it has come to represent to many a larger clustering of anything North Korean in nature with links to any aspect of known Lazarus Group activity. This is not a put down on any team that tracks the Lazarus Group, it’s simply a realization that the analyst bias that goes into selecting intrusions and putting them into the Lazarus Group is done differently across different teams and thus a super group is not likely to be granular in its accuracy. (I talk about the problem with this type of threat tracking here). This may not matter at all if you want to attribute the principal group responsible for Lazarus Group as North Korea. But attribution, especially when you want to attribute all parties and not just the one chiefly responsible, is not binary. Not every single intrusion that goes into the clustering of “Lazarus Group” is going to be accurate. Not every single intrusion is going to be North Korea developed malware used by North Korea operators. There are alliances (North Korea allies in other states or organizations), there are supply chains (where they source exploits, code, etc.), there are operators vs. developers, there are different operations teams, there are different customers of the operation’s intelligence requirements, etc. to consider. All this means that if you want to do attribution to North Korea off of Lazarus Group you can get to a pretty good confidence level (likely Moderate Confidence if you’re just basing it off of intrusion analysis). If you’re wanting to reverse engineer individual aspects of that grouping though the attribution wouldn’t necessarily hold. I.e. individual families of malware, intrusions, aspects of malware such as encoding routines, etc. could all be an important puzzle piece in multiple puzzles, not just Lazarus Group.

The fourth point is the biggest hindrance in attribution being transitive. All the puzzle pieces that go into doing an assessment can be important. But by themselves they are likely not. I’ve seen so many people ask for the “smoking gun” when talking about intelligence analysis. The FBI’s attribution of North Korea to the Sony Attack comes to mind (which I wrote about here in Wired) where the FBI’s assessment was sound but the infosec community wanted them to “prove it” so they released some technical pieces of evidence, which to the FBI probably seemed pretty good in hindsight but to the public were not conclusive. This is a common analyst mistake. When you do analysis there are pieces of evidence that become really important to you, but only in context of all the analysis you did. I.e. it’s really important to you now with all the knowledge you have about the case. But you needed a lot of other data and context to have it be important. So releasing just “the important stuff” externally will not likely resonate with others who cannot come to the same conclusions you did on just partial data. Even with identical data sets two analysts will likely come to different conclusions anyway. To address this you never get in the habit of arguing about evidence, you position and argue your assessment. The totality of the data and your analysis, not just pieces of data.

All this is a round about way of saying that if you take a piece of data from an assessment (such as links to Hermes malware) and take it away from all the other data, then you cannot take the assessment with that piece of data. You cannot just simply look for Hermes malware to pop up and go “yup that’s Lazarus Group”. Further, links of Hermes to other malware families like Ryuk and thus attacks where Ryuk show up further complicate the issue. The more analytical leaps you make the less likely your assessment is going to be sound.

This doesn’t mean that the attack wasn’t done by North Korea. If it was knowing their intention would be an entirely different and especially difficult assessment to make. But at this point, no actual assessments have been done. The only thing being highlighted in certain media outlets is transitive attribution because of links observed in different malware families. This is sloppy and will lead to numerous inaccuracies. Additionally, there can be political issues if high profile targets like the New York Times and Wall Street Journal (luckily they haven’t) come out and attribute the attack to North Korea. That puts pressure on the US government as well as the North Korea government. There are real impacts to attribution claims between states.

In summary, as an analyst you should be aware that assessments do not often have a transitive property. Understand your collection biases and what goes into the assessments you make. From there, if you need to make a new assessment, then you need to go through the process of collecting data and analyzing and producing an assessment, short cuts such as transitive analysis will not be better than a low confidence assessment. Do not strive for perfection where you have analysis paralysis (sometimes it’s ok to make gut calls as an analyst) but understand when something is a guess, a hypothesis that’s missing plenty of data sources and other hypotheses are also equally possible (low confidence), or when you’ve done structured analysis across multiple data sources to achieve a higher level of confidence (moderate or high).

Threat Hunting, TTPs, Indicators, and MITRE ATT&CK – Bingo

November 25, 2018

This blog is a continuation of a fantastic discussion with Richard Bejtlich. He responded to a question online, I blogged about it here and then he responded here and in response to another question I posed here. In this blog I’ll reply to his replies. The main point of this blog to me though is not to debate but to document my own experiences and look briefly at the evolution of terms and schools of thought as it applies to our cybersecurity field (the term cybersecurity vs. infosec is actually a great example of a term that has evolved yet to many still have two very different meanings and to some are interchangeable).

TL;DR Version: Richard and I seem to disagree on what an indicator is (reasonably so as his definition of one is perfectly supportable I just have a different view based on different experiences) which I didn’t previously realize was impacting how both of us viewed threat hunting. In doing so it seems that we’ve both come up under different “schools of thought” and therefore “hunting” can be leveraged in many different ways but you should strive to document and put structure to it if possible to really get the most value out of it.

Historical Context Matters But We Shouldn’t be Bound To It

In Richard’s blog he masterfully brings in some historical context of hunting’s origins (which few people really bring in history in a way Richard does but more should), or at least what was first documented (much which was done by him), in reference to the U.S. Air Force and National Security Agency. He also masterfully gives credit (another thing we should all strive more to do) to folks like Tony Sager and where his journey took him from the Air Force CERT to GE-CIRT where his use of the term was solidified even more with the wonderful analysts he cites.

What hunting was in the U.S. Air Force in the mid 2000’s and what it was at GE-CIRT shortly after doesn’t have to define the term for the rest of the field. It’s an amazing contribution, but not definitive. I think that’s, on face value, a frustrating topic. If I was at the GE-CIRT as an example and I really fleshed out a term my natural reaction would be to push against people using it differently (Richard is not doing that, I’m making a different point) but in reality terms, use-cases, and our field’s knowledge in general is constantly growing. There’s actually a logical fallacy called Appeal to Tradition which essentially states that something is what it is because it’s always been that way. But what’s interesting here is “hunting” was something numerous groups did. None documented or arguably had as much impact as the folks like Tony Sager, Richard, NSA VAO, AF-CERT, and the GE-CIRT but the experience is not less valid.

As an example, in my own experience in the National Security Agency one of my jobs was the Discovery Lead for one of the NSA Threat Operations Centers (NTOC) (would have been close proximity to Tony’s group but operating independently). Prior to this I had no knowledge of the term “hunting” and was not exposed to the schools of thought that were being developed at GE-CIRT or, interestingly enough, the use of the term and practice that Tony Sager and VAO were pursuing. I was part of a small team that established a new NTOC office at a field site and we were tasked with finding the “unknown unknowns”. This was admittedly very vague guidance but it was explained to us that the role of our NTOC office was explicitly finding the cyber threats that weren’t already documented and being tracked. Find what’s evading our insight to reveal collection gaps. We called this “Discovery” which, on a day-to-day basis, became known as “hunting.” The line between hunting and cyber threat intelligence though were very blurred for us because of our requirements; I would note that hunting was one way we went about satisfying our cyber threat intelligence requirements by identifying previously unknown intrusions (hunting) that we would then analyze (CTI). What we effectively were doing was taking an intelligence requirement, collecting against it through hunting, analyzing the intrusions we observed, producing insights, and distributing it to others as blogs, defensive mitigations and new detections, and reports. We used models such as the newly developed Diamond Model (previous to the paper’s publication) under the tutelage of Sergio Caltagirone and if I remember correctly we were the first team or one of the first to do so outside of his where he, Andrew Pendergast, and Chris Betz created the model. The interesting thing to the discussion here is that “hunting” was something that developed for my team without, to my knowledge, external prompt although I imagine there was cross-pollination from Sergio, Andrew, and Chris with the various teams they interacted with. For us, Discovery and our use of the term “hunting” was always about “finding new threats” but the main value in doing that was in identifying new defensive recommendations and gaps in collection even though we also found plenty of new threats to track. (For anyone curious I ended up choosing industrial control systems as the focus for my Discovery team since the collection would be so different and thus maybe the threats would be to; it turns out they were. This was a defining moment for me in ICS cyber security and a lot of what I had to develop in knowledge there at the NTOC helped in my journey and greatly inspired my work today at Dragos). Interestingly though one of the ways we found new threats was in the application of adversary tactics, techniques, and procedures as analytics/patterns instead of specific indicators. This aspect seems to distance Richard and I further which I’ll cover in the next section. But to close out the topic on the value you get out of hunting…

Richard acknowledges that you can identify visibility and resistance gaps as part of hunting, but it’s not the reason to go hunting.

 

In response to that I would say it may not be the reason that he and others hunt but it was always the reason my team hunted early in my career and it shaped how I view hunting now. His school of thought is simply different. I would also wonder aloud if a term needs to change when the actions are the exact same but the value propositions you’re aiming for are numbered differently. If you do exactly the same things and use exactly the same approach but you hope to find threats instead of collection gaps should you rename your effort? I’m not sure on that yet but my initial thoughts would say no. The “school of thought” for hunting and how to achieve it was very specific for Richard and me; I assume there are others in other organizations and parts of the world who have similar experiences that make their schools of thought much different. Richard and I are obviously both heavily influenced by a U.S. Department of Defense flavoring. I’d be interested if anyone else had their own journeys to document around the same time period.

We Were Bound To Disagree

It is clear that Richard and I disagree on the term hunting and how it’s used but the important part is why we disagree. I actually have no issues whatsoever with how Richard is defining the term and its background for his uses; his experiences are unique and many including myself have benefited from them. I don’t think there is a right or wrong to this discussion, just a friendly exploration to flesh out the topic for ourselves (or at least mine) and others. If we go back to the original points though it was clear Richard and I were bound to disagree and I didn’t see it initially.

I referenced but then moved past a point early in my other blog that Richard referred to all threat detection falling into two categories of either “matching” or “hunting”. I thought it was an interesting discussion which I slightly disagreed with but hurried past to get to the core debate. I noted that everything is “Matching” or “Unknowns” but not that hunting is contained in one or the other; it can be across both. If you are matching indicators you are not hunting in my opinion which I think Richard would agree with. If you are matching adversary behaviors though you might be depending on how you’re doing it (if the behavior is already a defined detection then no, but if it’s a pattern you’re searching out to find new activity that isn’t defined, then yes). When in fact, that was my error and in reality the disagreement is rooted there. A subtle but important point because I didn’t realize (although to his credit he’s definitely written on it before) that Richard considers adversary TTPs to be a sub-class of indicators of compromise (IOCs). I asked him that question after his first blog and he was kind enough to answer it here.

Again, Richard is great to identify and credit influential folks such as David Bianco and Chris Sanders, both who I consider friends and have written things that have definitely helped me further my thoughts too. He references David’s Pyramid of Pain (an extremely useful model by the way) where it very clearly calls out TTPs as a type of Indicator. I’m going to disagree again, surprise surprise, but this is where everything “came together” for me in that the disagreement is in the evolution of terms and schools of thought and nothing more. David’s use of the term indicator is mostly in keeping with another foundational work by Eric Hutchins, Mike Cloppert, and Rohan Amin titled Intelligence-Driven Computer Network Defense Informed by Analysis of Adversary Campaigns and Intrusion Kill ChainsYes the kill chain paper. Here they define out indicators to be atomic, computer, or behavioral. Behavioral indicators here are defined as those indicators that are made up of atomic and computer indicators to describe adversary activity, the example given in the paper is “the intruder would initially used a backdoor which generated network traffic matching [regular expression] at the rate of [some frequency] to [some IP address] and then replace it with one matching the MD5 hash [value] once access was established.” Where David’s use of the term seems to be in disagreement is “behavioral” which on its face would speak to TTPs but in reality TTPs can be described and leveraged now without any atomic or computed indicators to accompany them. My second point in the next paragraph will dive deeper into that.

Why is this such an important point to this discussion? For two main reasons.

  • First, terms and schools of thought evolve. “Indicator” today is almost exclusively associated with indicator feeds and the atomic and computed form of indicators. Some still use behavioral indicators and talk about them quite a bit to great value. But for the majority of the industry if you say “indicator” they’ve come to learn about them, use them, and see value propositions defined by atomic and computed values. Security professionals using indicator that way or not wrong. We even see another “school of thought” coming up which is based around MITRE’s ATT&CK; quite a few of their use-cases would fit nicely into mine by using ATT&CK as one framework for guiding threat hunts. In one of their papers they specifically note the focus on tactics and techniques (TT in the TTP) for them is important to go beyond “traditional indicators of compromise.” Here it would seem they are not using TTPs as a form of IOC either.
  • Second, what you can to do today for detection was not necessarily possible for most teams as little as a decade ago. The cybersecurity community has evolved quickly made many more advancements than we often credit ourselves. One such advancement is in the form of analytics. Analytics are effectively questions you ask of the data. Analytics have been around a long time in many forms but with the advancement of the community and of computing power we can now use analytics in more large scale ways and in a distributed fashion at scale (run the analytics at the edge where the data exists instead of pulling all the data back and then running analytics across it). One type of analytic, that I wrote about and referenced in the last blog when I mentioned the four types of detection paper, are threat analytics. Threat analytics effectively are adversary behaviors, i.e. TTPs or tradecraft (different things by the way). But they are not behavioral indicators in the way Hutchins, Cloppert, and Amin identified them. They don’t include any atomic or computed indicators; post detection there will be indicators but you don’t define the indicators ahead of the detection. The entire analytic might say “alert any time a file is dropped to a system, opens up a network port, generates network traffic to an external IP address, and then downloads an additional file once communications are established”. This analytic would get to the example given in the kill chain paper but without knowing the hash or IP address of the backdoor or anything about the adversary leveraging the behavior. This is done through an analytics engine now which have been around for awhile but are more common and accessible than ever before. When the analytic is defined it is “matching” to Richard’s point. But when I’m leveraging the TTP or tradecraft outside of the analytic to go search for and find new threats (numerous threats can leverage identical TTPs, so you’re searching for “unknown” threats using “known” tradecraft) I’m not matching any indicators and instead am using an intelligence-driven hypothesis to identify new threats. That’s EXACTLY what we did at the NTOC site I was at and we called that hunting.

As an aside, I think my second point is even how Richard is doing  his hunting to some degree because in his blog he gives an example that you could tell an analyst to go look go HTTP user agents as they are being leveraged by the adversary for malicious network communications but not tell them what user agents to look for, that at a high level is a tactic of the adversary which would classify as a component of a TTP and not be an indicator. It is not some anomaly that is occurring to filter through but a hypothesis generated from some insight the defender had such as seeing adversaries do it before (intelligence-driven) or based on their own expertise in how the protocol should work (domain expertise). I don’t want to argue points for Richard though so maybe I’m not interpreting it correctly but I think if we spent more time (preferably over beer) on this we’d probably root out more commonalities in approaches.

All of that long winded way of saying: Richard and I fundamentally seem to disagree on what an indicator is which is having a flow down effect I didn’t previously realize to how we view and define threat hunting. We would still end up debating back and forth on the ordering of the value propositions but both agree that the value propositions all exist (find threats, identify gaps, develop new detections, etc.) which is another reason everyone should be hunting regardless of how you order the value you get out of it.

This has been fantastic. I have thoroughly enjoyed this back and forth, and thank you again Richard for being such a gracious person to explore it all with me while challenging me to document my experiences in this blog.

Hunting vs. Incident Response vs. Just Doing Your Job

November 22, 2018

One of the things that motivates me to write is either being really pissed off or finding someone I really respect and having a different opinion. This blog is about the latter. Richard Bejtlich tweeted (oh the horror) that to him, hunting is just another form of detection. Now his real argument is of course more nuanced and well thought out but for the purpose of my blog and putting forth my view I’ll try to make his as simplistic as possible.

 

Now in reality, debating with Richard about topics related to detection, network security monitoring, and consequently threat hunting is probably similar to debating Plato about societies in the Republic. But I’ve always found myself learning more from debating smart people about small disagreements than debating the ones I outright disagree with. So enough compliments, let’s get into it.

The Argument

The heart of the argument, as I’m positioning it, is about what hunting is and is not and thus how folks use it. The initial prompt was from Twitter user @hellor00t about where “hunt teams” should be placed in the CIRT and whether or not hunt is just a new term for IR analyst. The comment was “seems like title semantics to me.” Which is totally fair. Richard, as previously stated, noted that hunt is just a form of detection. That incident response teams detect intruders using two major modes: matching and hunting. There’s a lot to unravel here but I’ll bypass the modes of detection argument slightly to say I actually agree but wouldn’t call one mode hunting. You’re either matching knowns (behaviors of adversaries, indicators, behaviors of users, etc.) or exploring unknowns (anomalies). You should cover your knowns first and then explore the unknowns but your strategy for detection should be more well thought out. I covered that topic with Sergio Caltagirone in a paper on the four types of threat detection which actually lines up nicely with Richard’s point (except the term hunting) here.

My Position

Honestly you can do whatever you want that gets you excited about security and proves to be an efficient and effective way to do your job for the organization. But I do think hunting goes beyond Richard’s point. I’ve written about threat hunting (not nearly as much as Richard) previously a few times before but here are two papers that help phrase my “school of thought” a bit that I wrote with David Bianco on generating hypotheses and another about what hunting is and isn’t with the “other” Rob Lee at SANS. My main thought process is that threat hunting is a proactive and iterative approach to detecting threats. It seems to align directly with what Richard wrote but I’d add that it really isn’t, to me, about detecting threats. I know that’s super confusing so I’ll go into depth a bit more.

Hunting is a hypothesis-led approach to testing your environment for threats. The purpose, to me, is not in finding threats but in determining what gaps you have in your ability to detect and respond to them. As an example, how do you know whether or not your organization would ever be able to detect and respond to the latest tradecraft observed by $ActivityGroup? You should generate a hypothesis about how that tradecraft would look in your environment played out against what you have to protect. You could determine the “route” the intrusion might take based on what you’ve observed previously and see if you have the people, process, and technology required at each point to successfully detect the tactics and procedures of the threat as well as the ability to respond to them. That’s only one way to generate hypotheses but hopefully serves as a tangible example. You’re looking to achieve a layered defense while figuring out your gaps. As an output of hunting, you should have gaps to address as well as detections you can now create as a result of your hunting (i.e. threat hunting cannot be fully automated but the output of your hunting should be more automation in your environment against the threats you hunted for). You could also create playbooks, or step-by-step guidance, on what an analyst should do to validate malicious activity and how to respond based on what you learned. Additionally, you could update your collection management framework (a topic I’ve talked about before and will have a paper out later this month on) to discuss what collection and data sources you do and don’t have in your own environment.

In short, hunting, to me, is a way to assess your security (people, process, and technology) against threats while extending your automation footprint to better be prepared in the future. Or simply stated, it’s incident response without the incident that’s done with a purpose and contributes something. Ad-hoc hunting to me isn’t hunting, it’s just proactively searching for threats in your environment (although admittedly saying you’re “hunting for threats” rolls off the tongue a lot better than “log correlating and searching”). Responding to alerts isn’t hunting either, it’s just triaging and investigating alerts. However, there are many “schools of thought” and you’re welcome to do whatever works for you in your environments.

Threat hunting is also an extremely useful way to measure the effectiveness and breadth of your threat intelligence efforts. Dan Gunter wrote an excellent Masters paper at SANS here on the topic.

Back to the original question of “where do you put your hunt team?” I really wouldn’t have a dedicated threat hunting team. I’d treat threat hunting as a multiple times a year or at least once a year assessment against the scenarios that are most concerning to you and your organization. Bringing together people from across the security organization and even outside that organization can be really useful to thinking outside the box about your hypotheses generation efforts. Having a dedicated team can make the process stale and cause too much group think. I’d rather see people “take turns” in that role if there’s a need to have a dedicated function.

In Conclusion

In conclusion (for now I assume) I will note I probably don’t disagree with Richard much but it was a useful prompt to have me think about threat hunting and put some thoughts in a blog to hopefully help others flesh out their views. My view is that hunting is not a form of detection, or isn’t just a form of detection. If you’re measuring the value of threat hunting on how many threats you find you’re likely not to be able to justify it at all compared to just doing proactive security work. Threat hunting, in my opinion, should be a much more structured test against hypotheses that is pushing your organization forward and ensuring you’re prepared against “styles” of threats. I do completely agree with Richard’s view though that Response is too narrowly defined as just incident response and after the fact to folks. I talk about “detection and response” largely because the terms can be useful in communication and when defining skill sets of folks. However, I do agree that detection and response both should be tightly coupled in people, process, and technology and that response is, in its best form, proactive.

 

What It’s Like to Raise Venture Capital (Just My Experience)

November 22, 2018

Last week it was announced that Dragos, Inc. raised $37M in our Series B financing event. That brings the total amount of money raised for the company to $48.2M since the company was founded in 2016 and marks my third time raising (Seed Round of $1.2M, A Round of $10M, and now the B Round). There’s a lot written explaining venture capital so I won’t attempt to explain it here in a short blog; but the purpose of this blog is to share some insights into things I’ve learned as a security practitioner that were new and surprising to me or just general observations that were interesting enough to document. Over the past two years I’ve taken more than 300 calls with more than 150 venture capitalists, officially “pitched” to more than 50, and had term sheets from about a dozen. I’ll start off with a comment about the team and company before diving into some of the observations I’ve had across those experiences.

“If you want to go fast, go alone; but if you want to go far, go together”

The proverb/quote above has some controversy around it regarding its origins but I like the quote nevertheless. It explains a lot about startups (to be fair so does the TV show Silicon Valley). When Jon Lavender, Justin Cavinee, and I started Dragos as co-founders we had previously built CyberLens together back in 2012/2013 time frame under the name “Dragos Security.” The tool was simply an assessment tool to process out packet captures and draw a map including some deep packet inspection of industrial control system (ICS) protocols. Getting it together as three people was really easy. It was a good year into the life of Dragos, Inc. when we were still adding things into the Dragos Platform that were already accomplished in CyberLens. The difference of course was the scale and stability of the Dragos Platform, our technology at Dragos, was significantly more vast than CyberLens. As we started adding a “feature” into the product it would break into numerous features and requirements. In theory what our technology does is simple: passively map out industrial environments, utilize threat analytics to identify the M.O./behavior/pattern of adversaries instead of just indicator or anomaly based detection, and offer up playbooks in an analyst workbench that act as step-by-step guidance to validate the alerts and scope the investigation. Not too bad right? Over the last two years we’ve seen the need to scale to 10 network appliances back to a server, to 40, to 300, and beyond. All on-prem without being able to take advantage of the cloud (in most cases) because of the sensitivity of industrial environments. At Dragos we’re about to be 100 people strong. You move so quickly you don’t really stop to think about the numbers and scale and all the things you’re doing. But one of the reasons startups tend to outpace larger companies is the ability to move those folks in a single direction and throw the entire organization behind it without as much communication challenges as large companies; flexibility is key. It’s amazing to see that occur and yet moving forward always seems like it takes place at a snail’s pace. You look back over 1 quarter and you realize that’s not true; but going far definitely requires the team and creating a scalable, deployable, stable, and usable technology often requires more than just a few people. I’m proud of the team we have today and the growth we’re achieving. Every now and then I sit back, pour a stiff drink, and just have to feel the exhaustion overcome me all at once. But then you sit forward and keep going again. You see the difference you’re making for your customers, the lives of the people on your team changing and growing, and you just feel so fortunate. Luckily, my eight month old son and wonderful wife provide more centering for me than exhaustion making the journey a bit more doable.

Venture Capital is Neither Good nor Bad

As a security practitioner I often found myself opining about venture capital. Bankers, private equity folks, finance majors, and “business people” were what I always assumed made up all the ranks of Venture Capital and to be honest I couldn’t stand the idea of them. I didn’t despise anyone, everyone takes their own path in life. But non-security practitioners trying to fund and decide “market winners” without an appreciation of the technology always bothered me. For anyone that’s seen my writings before it’s probably pretty apparent that I don’t think too highly of “management” positions leading security teams or technology teams without an understanding of security or technology. So it extended beyond venture capital. I also saw venture capitalists as just interested in making a buck (that is their job but the amount of “let’s make the world a better place” I’ve seen even in the hard times has been stunning coming from them) and largely figured it was better to bootstrap a business than take capital. While a lot of bad venture capitalists exist I learned that the bad apples were really giving a bad name to a much larger and more diverse field. Here’s some areas that I have evolved my own thinking:

Watch the Burn, Spend the Money, Watch the Burn

One of the reasons I hated the idea of venture capital is that I figured and had heard they would try to get you to spend so much money that you would have to raise more money later. They’d “get you over a barrel” and you’d be too deep in debt to do anything but go forward and raise more. That impression I had was entirely wrong, maybe I’ve just seen a different group of venture capitalists but there’s a whole science and art to throttling the business correctly. I’ve heard “watch the burn” from my board members (made up of our investors) to the point that it’s annoying not because it’s not accurate but because it’s been stated so much it couldn’t possibly have left my mind. The burn is how fast you’re burning through capital. Venture capitalists don’t want you to spend too much money, they want you to spend the right amount of money to invest in the business’ future. The logic is actually pretty simple when you think about it. In an enterprise software styled company where you are investing heavily in R&D you’re going to need to raise venture money to support it (sales can’t support the R&D costs especially before you have a product). But once you have the product (though it’s never done) you still have to invest heavily in sales and marketing. If I put $1 into sales and marketing today I likely won’t see any return on that for 9+ months. It takes time to generate the lead, make the contact, get the meeting, make the pitch, fit into this or next year’s budget, etc. The sales cycle can easily be 9-18 months long. But if I can invest $1 into sales and marketing today and get a return of $2 in 12 months that’s a 100% growth. Throttling the business to be “profitable” especially early on is robbing the future of the business. If I can lose $5M this year and make $10M in sales next year, that’s a great thing for the business. Thus venture capital is needed so that you can spend what you need to, but you don’t just want to spend money you want to understand what the return on investment is and monitor your burn as closely as possible to make sure you’re not over investing or under investing. This can cause a “foot on the gas foot off the gas foot on the gas” back and forth in the company on an almost weekly basis. There’s risk into taking this approach and it’s not the only one but previous-me would have stated “well just grow as you can support it and don’t try to lose money in hopes of earning more business next year.” Today I understand that the speed of the market, demand from the customers, growth of competitors, marketing from incumbents, and more all goes into deciding that spending large sums of money to grow the business is hyper critical especially in a software company where deal sizes and sales cycles are both large.

Lots of Different Types of Venture Capitalists 

There are a lot of different types of venture capitalists, and I’m talking about the people not the firms yet, I’ll approach those next. Some venture capitalists are the bankers, private equity folks, business grads, and financial analysts that I had previously thought about. But many are also “operators” that have built and led companies before. In reality there are good and bad in both forms. I’ve found that as a founder I really resonate with operators. Having a few of them on my board early on gave me good guidance and made a collaborative environment. They were also more empathetic to the needs of the business instead of spreadsheets and trying to math the company to death. As we grew it was also important to add those who had more experience as the “typical” type of finance people who helped provide a bit of devil’s advocacy to the board on various decisions and help make sure that the “emotion” of running the business and the “experience” of having done it before wasn’t completely out of touch with what the numbers could support. It’s obvious of course that it takes all types, but to me seeing the different value propositions of people and their experience and how they influenced the company was interesting. As a gaming nerd there’s so much that role playing games have taught me about business, more so than any of the free online classes I’ve taken on business (seriously you should watch those free videos like the free MIT and Harvard classes and you should throw in some good RPGs as homework). Class skills and development of different professions while understanding people’s skill tree maximums and strengths/weaknesses is much easier when you just imagine everyone you meet has a playing card or a stats page that you can see when you hover over them.

Lots of Different Types of Venture Capital Firms

There are many different types of VC firms. Some have experience in your field, some do not. VC Firms vs. Corporate VCs are a major difference in how they view the world as well. There is even the informal ranking system of “Tiers” that is very loosely made but can help you think about the VCs to determine which are right for you. From the very beginning the idea at Dragos was to IPO the company. Making it to the stock exchange with a public listing of your stock sounds like a good thing from a monetary perspective but it provides something more important to my team. We want to keep Dragos around and really make the world a more secure place for our most important and critical infrastructures. The industrial world is huge and deserves protection. Having an “exit” by getting bought by another company generally doesn’t allow that. Acquisitions of technology companies are not always bad but especially early in the company’s life when they are just really getting their culture and tech and value propositions right, getting adopted into another company can be a death sentence. It’s going to take years to put a real dent into industrial cybersecurity. When you raise venture capital though you have to be able to give your VC’s an exit. They need to be able to exit the company at some point and recoup their investment plus how much their investment has grown. As it turns out, the appetite for how this occurs differs from firm to firm. There is not simply 1 “persona” at any firm but their track record, size of fund, and “tier” can help inform what they’re looking for in how you exit.

As an example, if your VC firm has a small “fund” they’re investing out of ($100M-$200M) and they haven’t had a lot of IPOs before then it should be pretty obvious they are looking for an acquisition as an exit. If they invest $5M in your company early on and you can exit for $200-$300M they get enough return to put a real “dent” in their overall fund size to show returns to their investors (which surprisingly enough are not just rich people but more often college endowments, retirement funds, etc.). But if your investor has done IPOs before and is investing out of a large fund ($500M+ and often times $700M+) then the check sizes they write are going to be larger (they’re going to look to invest more like $15M instead of $5M so they can put their capital to work) and they’re going to need the exit to be larger (which is more likely to be an IPO than an acquisition) to get an exit that actually puts a dent in their overall fund. There’s way more to evaluating VC firms including their networks, who the board member is that will actually sit on your board, and how they enable their investments but figuring out what they need in terms of an exit is a really important part of the equation to make sure it aligns with the company you want to build. An IPO is an “exit” for many of the investors but it’s a growth stage for your company. The founding team and its members shouldn’t be thinking about an exit but should be aware that taking venture capital is more of an investment/loan than it is anything else.

Raise What You Need Not What You Can

If you’re outside of a company you rarely get deep insight into what’s going on in that company. One of the ways we measure companies though, at least when consider venture capital, is how much they raised and at what valuation. At one point I would have thought raising the maximum amount and then not raising again for a long time is a good strategy; turns out that’s very silly. The simple version is easy to understand, if I raise $10M at a $15M pre-money valuation you give up a lot of the company in terms of dilution. If you raise $5M at $15M and then raise another $5M in a year at a $30M valuation you’ve just raised money without giving up as much of your company. It’s never that simple though and you have to figure out how much you can raise and how much you can put to use to get you to the next milestone for your company. If the venture community is hot, then investments might come easy, if it’s not then no matter how good you do the money will “dry up” to some extent. Additionally, you always want to make sure you can hit the goals to get to the next milestone.

As an example, did you raise $30M at $100M pre-money valuation? Congrats that’s awesome. Did you raise $60M at $150M valuation? Looking a bit dicey by comparison. But why? Let’s assume those are C series numbers since they are really big amounts (although in the industrial community a lot of the Series B raises are more like Series C, our $37M raise at a B is more like a C round for comparable IT security companies). If you raise $30M on $100M pre-money valuation then your post money valuation is $130M. To raise a D round on this trajectory you really want to show investors that you can 2-3x their investment, this entices the new investors and shows good health for the company. I.e. at the D round you’d really want a $260-$390M pre-valuation. Ideally you could get there in a 2-3 year period showing rapid growth. But if you raise $60M at a $150M valuation for your D round now you have a $210M post money valuation. Which means you’d ideally want to hit $420M-630M pre-money valuation to show the same level of growth. But the larger the numbers the slower the growth might be and there is a limit in how much you can invest in your company each year to hit a return especially if you’re being compared to the growth of the company that took $30M. I.e. pouring $50M onto a company instead of $30M may not actually give you any extra value depending on market size, customer adoption, sales cycles, and more. So if you raise more than you need just because it’s available you are potentially taking more dilution in the company than you need, you are possibly not doing anything more for your growth rate, and you may make it incredibly difficult or even impossible to raise your next round which means you have to get acquired as you’ll be burning too much to go cash flow positive or raise more. There’s no perfect formula nor only one school of thought, but this was all interesting to me as I learned more about raising capital and watched our own experience. If you’re wondering, we raised the $37M specifically to get us to the next round which we have a target valuation and date for already (I found it useful to always be thinking about your next round and walking backwards into it for the current round).

Terms, Terms, Terms

It would also seem obvious that a company that raises $30M at $100M valuation is a better company than the one that raises $30M at $90M valuation (pre-money on both). However, what most people never get a chance to see are the terms of the investment. The term sheet is the “offer” that an investor makes a company for the terms of the investment that then get ironed out in a much larger package of closing documents. Those aren’t made public but in my opinion are far more important than the valuation of the round. A seemingly bigger and better investment may have horrible terms. As an example, one term that could included in the investment is “participating preferred” or even have a multiple on it. A multiple is idiocy in my opinion and would only make sense to a company in distress and participating preferred in general is a term I refused to ever accept. There are numerous terms you have to watch out for that can flavor the entire trajectory of the company but let’s dive into participating preferred which I think of as one of the worst terms.

Let’s imagine an investor puts in $10M into your company and you add equity for them which gives them 10% of the company. And then your company eventually exits for $200M. In a simple world where that was all the investment you had then the $200M would be split between all the preferred stock holders (the investor) and the common stock (the founders and employees). So $100M would be split according to how much equity everyone has. The investor’s $10M turns into $20M and that’s what they leave with. The rest of the $180 is split between the founders and employees depending on how much they own each. But if the investment was participating preferred at a 1x then the investor would get back their $10M and then get to participate in 10% of the remaining $190M. That means they’d walk away with $29M and the company would split $171M. In reality the investors would actually get more because of interest accrued on the initial $10M. This is very simplistic but even in the simplistic view you can see why this would be problematic. Even if your goal isn’t everyone splitting money and instead is just growing the company, you could potentially have lots of extra money flowing out of the company at an IPO. Simply put, no matter the situation you really don’t want money leaving your company that you could be investing in your company or splitting between the people who were putting in the day-to-day work. In my experience, my goal was a simple term sheet. An adviser of mine told me to demand that the terms of the deal be so simple that they could be written on a napkin. We aimed for that and succeeded in doing it but it is way harder than it sounds.

Boards Are Equally Important and Not Important

This is a weird statement to write because I get a lot of value out of my board as advisers and a sounding board. But previously I would have thought a board had a lot of control over a company. I don’t know why but I just always assumed a lot of decisions get made at the board level. Turns out that’s not the case. The person running the company day-to-day is the CEO. The CEO knows so much more about the company than the board members could possibly know. In relation to venture capital the board wants to know that the company is growing, monitor their investment, and provide input and connections that could benefit the company to help it grow. As an example, one of the ways I used my board is to help me with interviewing key hires especially VP’s because they’ve seen many people and have experience I don’t have on to act as a sounding board. They provide good connections throughout the various industries we work in and also provide some guidance to make sure that I’m growing the business in the right way at the right speed and am building a company that others will want to invest in which will help our team build the type of company that can really impact the industry. But they don’t make a lot of decisions. I assume a lot of people think they do because I even have people outside my company that want to do business with us or want something from us hit up my board members directly assuming they have some significant influence over me; very little makes me more likely to ignore someone.

It turns out my view of how involved boards were was entirely off. They provide help that I ask for, but they don’t make a lot of choices. Salary and compensation packages get approved at the board level (which I find to be useful actually), the annual budget gets approved there, and the rest is just providing them updates and getting advice. You could get bad terms that dictate a board has more say but I’d be concerned about that for a lot of reasons. Opening a new office, choices around the product, hiring and firing, strategy for the company, go-to-market, etc. that’s all on my team internal to the company. The board is interested and active but entirely disconnected from the “decisions” that get made around that. The approval of an annual budget has a high level of impact but it’s something my team and I put together and put in front of them, their input is “that looks normal” “that looks like it’ll get you what you want” “have you considered upping the investment in X?” “I think you’ve over invested in Y”. It’s useful, it’s their approval, but it’s still the work from inside the company and whatever I put in front of them is going to be within the range of what gets approved.

I imagine this is all super obvious to most other people but I was surprised to find how little boards actually do and yet how they remain vitally important. One of the things I’ve really found value in my board is simply having a well experienced team of folks as a sounding board. It really can’t be overstated. As the CEO whether or not its true you constantly feel you’re the person who’s suppose to have all the answers and lead the ship. Sure everyone else actually does all the work, and the decisions are super collaborative in the company, but that feeling remains. At the board level though it feels far more like a discussion with less stress. I’ve found my board to be my “safe space” to just think about stuff out loud without repercussion and have folks more experienced than myself in areas that are different but important provide thoughts or validation. At the end of the day it’s not lost on me that so many in the community, so many customers, and all the families of all my employees have a very vested interest in me not screwing up my part. The act of “yea Rob that makes total sense” or “I really think you should dig into that more” provides far more comfort than I can put into words.

Every Company is Different and Your Journey Will Be Filled with Friends and at Times Lonely

These are my thoughts. These are some of my observations. Yours are bound to be different but I hope it’s useful to see one person’s thoughts on all this. I’ve never felt more excited and more proud than leading this team at Dragos and I’ve also never felt so very alone in my career. That sounds weird but let me clarify. Being the CEO and founder of an industrial cybersecurity company that has received venture capital, now above $48M, and is forging into a market that’s largely been undefined…is a lonely and exciting endeavor. Industry analysts and financial folks have tons of opinions, but none of them have actually been down this journey. I’m empathetic to the various industry analysts trying to define the market or provide hot takes on where the market will or won’t go; and it’s useful to the community because buyers and community members cannot just listen to the obviously biased companies leading the way, but those same industry analysts and financial folks have far less experience in this topic. The number of peers I have for the very specific thing I’m doing is tiny. And to be honest it’s not like the founders of competitive companies really get together and have open discussions other than “hey thanks for helping also build the market, good luck to you, I hope I utterly destroy you on the battlefield, but I respect you” lol. I often look at the industrial cybersecurity community and industry practitioners that I respect pontificating on the market, who will be winners, what that means, what products will be needed, etc. and I struggle not to roll my eyes a bit as they have never built companies or products in that space. I have to force myself to realize that what they’re doing is still an important part of the equation and very useful to folks. But I do also understand that they are also extremely biased, operating with completely different experience, and sometimes are advising on things that they have far less experience in than people realize especially compared to the companies that live it day to day and are on that unique journey. I’ve had heated arguments with people I respect and call friends simply because their experience guides them to view things so differently than the experience I have. And yet, that’s ok. Actually it’s really useful for everyone. And taking a step back to just keep that in mind is vital to not becoming arrogant or being misguided yourself. To solve a problem as complex as any that guides you down such a unique path is going to require thought process and input from everyone, welcoming it is important especially when it seems counter to your own experiences.

Why say all this? To stress the point that in this community, this industry, this time, etc. things are so specific that any lessons learned I have may not really apply to you in your journey. And it’s so different that even the founders that have built a company before may not have experiences that help or relate to you; I found the “entrepreneur” networks to be largely disappointing with experiences that didn’t translate to the challenges my team is facing. I found the folks in my company and their insights as well as the experiences and insights of our customers to be far more useful to me. But maybe those networks work for you. I don’t think I’m special, but I think this journey is unique. Yours is too. Hopefully my thoughts are useful but don’t take them as the only thoughts nor any of my takes as the only way to view those topics (hell maybe I’m wrong on plenty and will have different views in a few years). Instead, I’d focus more on how my opinions have changed and be open minded to have yours change as well.

What It’s Like to Testify to the U.S. Senate

July 21, 2018

One of my SANS students challenged me recently that I haven’t posted on my blog in awhile; I realized I hadn’t and checked today and found that it’s been almost a year since my last post. So first of all, apologies on the delay. I’ve been busy (Dragos, Inc. has been expanding rapidly and we’re about to reach 60 employees, my wife and I welcomed our first born into the world, and industrial cyber attacks haven’t exactly slowed down) however, no excuses, I should post and share my experiences more and appreciate the folks who take time to read. This post comes as a request to some attendees at one of my SANS @Night talks where they asked me what the experience was like and how it happened. I feel that it’s really not transparent how someone gets to testify in front of the Senate and what all goes into it so I hope this post is useful to illuminate the experience a bit. The recorded testimony can be found here.

How I Got Invited

I imagine most folks’ path to testifying in front of the Senate is different. I also assume for most they are seen so widely outside their own community as an expert that it’s a natural thing. My experience was a bit different. One of the things that motivates me a lot is educating people especially on topics of intelligence and industrial security. The intersection of those fields with policy is pretty clear and unavoidable. I’ve also always been a bit outspoken about individuals speaking on and influencing technical discussions without having technical experience. This has led me to getting involved in educating policy folks on topics of industrial cybersecurity, especially critical infrastructure. An ex-Senate staffer I ran across asked if I’d be interested in offering some sessions for Senate and House staffers on the topic of electric grid cybersecurity. Over the course of about a year I would routinely go to DC and spend a few hours talking about how the power grid works, what cyber threats actually do, and demystifying a lot of the hype (yes our infrastructure operators have made fairly resilient infrastructure, no cyber threats aren’t magic or all powerful, and no Ted Koppel’s book is not accurate in the real risk). In addition, I got invited to speak on a panel on cyber threats to the grid at the Siebel Scholars’ conference with a panel made up of Richard Clarke, Kevin Mandia, and Liam O’Murchu (which was moderated by Ted Koppel…super nice guy, he just didn’t write a technically accurate book). While at the dinner another Senate staffer liked what I had to say and offered to keep in touch, I exchanged cards and that was that.

Eventually it was the staffer from the Siebel event that emailed me out of the blue one day asking if I’d be willing to testify in front of the Senate’s Committee on Energy and Natural Resources on cyber threats to U.S. infrastructure. Because I had spent time with other staffers in the Senate the recommendation went over well as a few of the other staffers knew me and agreed I’d be good to have on the panel. What I also really enjoyed was finding out that both Republican and Democrat staffers had recommended me; I joked with them that they all thought I was on their “side” and that they couldn’t figure out where my politics actually fell (to be honest I’m not political, just opinionated) but in truth they were all just on the same page of wanting to protect infrastructure and didn’t care about the politics around it.

Preparing for the Testimony

In preparation for the testimony I used YouTube to watch other testimony to the committee on the subject of cybersecurity and determine what type of questions the Senators had previously. I also pulled the written testimony of Thomas Rid and Kevin Mandia, two individuals I respect in the field that have also testified, and looked at their style of writing and what they chose to highlight. I then prepared my written testimony on the key points I wanted to get across. For my testimony the three key points were essentially:

  • that industrial cyber threat landscape is largely unknown so we cannot just adopt IT cybersecurity frameworks/regulations/best-practices into industrial networks because they were built off of risk observed against cyber threats targeting the Enterprise networks and how to handle them
  • that regulation has served a purpose (such as NERC CIP for the power grid) and helped the industry but that we have exhausted reasonable regulations and the industry is struggling to innovate or do real defense in the face of new regulations that come out every few years
  • that the Department of Energy’s Office of Cybersecurity, Energy Security, and Emergency Response (CESER) that was being formed at the time has a unique opportunity to help work with private sector and form partnerships not only with asset owners and operators but also the private sector security community; many of the government’s actions right now in reaction to fear of cyber threats are quickly pushing them into being competitive with private sector firms that are actually better equipped to deal with certain situations and we should all attempt to play to our strengths instead of competing

It was interesting to try to distill everything I ever wanted to say down to effectively seven pages (found here) and to do so in a non-technical audience of policy makers. Some of my points, such as highlighting that technologies such as artificial intelligence aren’t a silver bullet for security, were put in to directly counter some of the things the Senators had been told in previous testimony that I felt mislead them. I then sent out my testimony to people I knew and trusted around the community who helped me steer clear of any wording or language that might come off wrong. As an example, I was mentioning “NERC” “DOE” and “energy companies” so I sent my written testimony to people I trusted at NERC, DOE, and numerous energy companies to make sure I was representing them the best I could. I.e. don’t get in front of the Senate on TV broadcast across the country and talk out of turn about matters that impact others.

What I didn’t expect was that the list of people testifying goes public well before the hearing so people reach out to you and the Senators about you. I had various interests groups email me trying to get their points included in my testimony; almost exclusively it was groups I’d include in “the crazies” category. The two most surprising was a group that wanted me to bash NERC and tell the Senators how the grid was so fragile and needed new oversight and something about EMPs (please stop with the EMP stuff) and then a member of the industrial community emailed some Senators directly and, from what was explained to me later by a staffer, essentially petitioned them not to listen to whatever was going to be said about cyber threats and instead to realize that level 0 sensor cybersecurity is the only relevant topic. I consider this person a friend and long-time community member and do not think he was being malicious but to be honest I found that to pretty rude because it, intentionally or not, was a distraction and could have subverted the points of the people that were presenting on the topic. (I’ve avoided the level 0 debate in the community because it becomes almost fanatical but in essence my position is that all the risk we’re seeing due to cyber threats does not start at a sensor level and if you do not learn what the adversaries are doing, how they’re doing it, and counter them before it ever gets to that level you will lose out). I only include this discussion of unexpected components to testifying to note to everyone that Senators and staffers get all sorts of things sent to them all the time. You really need to be active in the process to counterbalance the viewpoints but also ensure you do so in a manner that is conducive to the overall narrative.

Delivering the Testimony

For the verbal testimony itself you cannot read from your seven page written testimony. The Senators have a very limited amount of time to hear testimony and ask questions and they want every second to answer as many questions as possible. They are extremely busy and this is genuinely the time they’ve dedicated to focus on this specific topic. So you get five minutes to summarize your points, they’ve read your written testimony in advance, and they get a few minutes to ask you questions back. Who they ask questions to is entirely dependent on the Senators and who they want to talk about and on what subjects.

I have to admit I was actually pretty nervous delivering my written testimony. This was a recorded event and not even live. I’ve been on live news before on channels such as CNN and Fox in front of millions of people and felt more confident. I speak at a lot of venues. I’ve keynoted numerous conferences. And none of it compared. This was nerve wracking and I had to steady my hand by holding the paper in front of me. I also made a deliberate choice to speak fast and get through as much material as possible in the five minutes, I knew the Senators had already read the testimony (or at least their staffers did) and that they had prepared questions, so my verbal testimony was intended more for those that watched the hearing after the fact. When it got time for the QA session I was back in my element and felt confident. But that verbal testimony was the most nervous I’ve been in awhile.

I found the most important part of the verbal testimony was to try to answer the Senator’s questions as quickly and coherently as possible. I also found that all of their questions were extremely reasonable. Even those that appeared to have an agenda actually only were getting clarification on points they had previously been told by others. And to be honest, more lobbyists and self-interested people try to speak to Senators than folks doing the mission. What that meant to me is a clear understanding that some of the Senator’s views in topics such as just disconnecting the grid from the internet were based off of other people and not some agenda of their own. To be honest I found that the Senators all cared deeply about the topic and were highly professional. The country definitely feels divided these days but in the hearing I was in, at least, everyone there seemed to sincerely care about infrastructure security with no partisan political games tied to it.

At the end of the testimony I also felt that I won some major kudos points at home. My son was being born that day and my wife and I both felt that I should still go and testify; she is an immigrant from Holland and I served my country in the military so we had this extra sense that this was something patriotic that had to be done; so she gave me the go-ahead (luckily I didn’t miss my son’s birth though). The Senate was apparently informed about my son’s imminent birth and just how awesome my wife was being by letting me go…so on official record, at around 2:06:17…Senator Murkowski and the committee congratulated my wife and thanked her, and I got on official record “she’s awesome” so in essence…I win.

Final Thoughts

It’s really difficult these days to talk about politics with how decisive it gets especially in the media. We have things that make us very passionate and strike at the core of who we are and what we believe in. I’m not political at all and even I’ve been extremely bothered by some of the things I’ve seen such as the consistent attack on the U.S. Intelligence Community. However, testifying to the U.S. Senate was the most lifting political experience I’ve ever had. I swelled with pride in a way not easy to describe and felt the joy in this grand experiment we call democracy in a way I found unexpected and amazing. I would encourage the technical practitioner community to engage your elected officials and their staffers. Seek to educate others. And if you get the chance to testify to take the opportunity and try your best to represent the community well in doing so; it was an amazing experience that I highly recommend.

 

 

 

 

 

Russian Election Meddling, GRIZZLYSTEPPE, and Bananas

August 17, 2017

It’s been awhile since I’ve been able to post to my blog (as it turns out doing a Series A raise for my company Dragos has been time consuming so I apologize for the absence in writing).  But it is fitting that my first blog post in awhile has something to do with the GRIZZLYSTEPPE report. I almost got sucked back into writing when I saw the Defense Intelligence Agency (DIA) tweet out the Norse cyber attack map.

Matt jumped on it pretty quickly though which was great.

I tried to attempt to fill the person in running the account just in case they didn’t understand why folks were less than excited about their presentation.

But in their responses to me it seemed they didn’t fully understand. They articulated that they use unclassified data for the conference but use classified data at work. Of course the problem wasn’t the data (even though it’s not just unclassified but completely bad/fake data) it’s the idea that a cyber attack map aka “pew pew map” is not a good way to communicate to any audience as its simply a marketing ploy. However, it’s not worth a full blog post so I’ll just instead request everyone to do their homework (should only be a quick Google search) on why pew pew maps are stupid and everyone serious in the discussion should stop using them.

On To the Main Discussion

But on to the main topic. What does Russian election meddling, the GRIZZLYSTEPPE report, and bananas all have in common? Absolutely nothing. Each are individually completely unrelated to each other and people should stop putting any of them together as it ultimately just makes people look silly (to be fair no one’s associated bananas with the election interference yet but it might be a better correlation than the GRIZZLYSTEPPE report).

This discussion was all spawned by an article that the New York Times released on August 16th, 2017 titled “In Ukraine, a Malware Expert Who Could Blow the Whistle on Russian Hacking“. Spoiler alert: he can’t. I went on a bit of a Twitter rant to explain why the article wasn’t good, it can be found here, but I felt it was a complex and an important enough topic to cover in a blog.

The NYT piece posits that a hacker known by his alias “Profexer” was responsible for writing the P.A.S. tool and is now a witness for the FBI after coming forward to Ukrainian police. The P.A.S. tool, the article puts forward, was leveraged by Russia’s intelligence services without his knowledge (not sure how he can be a “witness” then but I digress). The authors of the article previously explicitly stated P.A.S. was used in the break-in of the Democratic National Committee  (DNC) but they had to issue a correction to that (to their credit, folks from NYT reached out to me after I critiqued it on Twitter to try to get the story correct after it was published; I asked for the correction as I’m sure others did but in reading the updated article the correction doesn’t actually address the larger issues so I wanted to cover them here in the blog).

 

Figure 1: Correction Related to P.A.S. and the DNC

Where did they get this assertion that P.A.S. was used in the DNC breach? By tying the GRIZZLYSTEPPE report (which does note that P.A.S. has been used by Russian security service members before) to the DNC breach. The GRIZZLYSTEPPE report has nothing to do with the DNC breach though and was a collection of technical indicators the government compiled from multiple agencies all working different Russian related threat groups. The threat group that compromised the DNC was Russian but not all Russian groups broke into the DNC. The GRIZZLYSTEPPE report was also highly criticized for its lack of accuracy and lack of a clear message and purpose. I covered it here on my blog but that was also picked up by numerous journalists and covered elsewhere. In other words, there’s no excuse for not knowing how widely criticized the GRIZZLYSTEPPE report was before citing it as good evidence in a NYT piece. Interestingly, the journalists didn’t even link to the “Enhanced Analysis” version of the GRIZZLYSTEPPE report which was published afterwards (and is actually much better) as a response to the critiques of the first one.

A major issue exists though with the correction to the NYT article. It changes the entire point of the story. If Profexer isn’t actually a “witness” to the case because P.A.S. wasn’t used in DNC then what’s the message the journalists are trying to get across? Someone who wasn’t working with the Russians, developed a tool that the Russians didn’t use in the DNC case, and didn’t have any insight into any of the Russian threat groups or campaigns cannot be a good witness.

Even after the correction though the journalists draw the readers attention to the breach early and often to continue to reinforce that this gives new insight into that case.

Figure 2: Snippet from NYT Article Referencing DNC Breach and Profexer

And again the journalists explicitly state that Profexer is somehow a witness to what occurred and reference him back again to the election hacking.

Figure 3: Snippet from NYT Article Claiming Profexer is a Witness

The article goes on to note how this changes our thoughts on the Russian groups (APT28 / APT29 or COZYBEAR / FANCYBEAR) and how they operate; the journalists state that using publicly available tools or outsourcing tool development to cyber criminals is against the modus operandi (MO) of the Russian security services. I do not know where the journalists get this claim but they do not source it; I disagree with the claim but I’ll note the burden of proof here is on them with regards to showing where they’re claiming the previous MO and I’ll simply state that there have been numerous publications and reports showcasing Russian threat groups including the security services using other groups and people’s tools and exploits. This isn’t new information and it’s fairly common for many threat groups to operate in this way.

The attribution on APT28 and APT29 is some of the most solid attribution the community has ever done. Numerous cybersecurity firms have covered this group including FireEye, CrowdStrike, Kaspersky, TrendMicro, and F-Secure but we’ve also had government attribution before by the German intelligence services on a breach into their government that pre-dates the DNC breach. A cursory look will reveal that organizations have been tracking this Russian threat group for about a decade now. Yet none of the people who’ve actually covered these groups were cited in the NYT article. Instead the journalists chose to cite Jeffrey Carr and his quote is confusing to most readers because he is trying to detract from the attribution where he states: “there is not now and never has been a single piece of technical evidence produced that connects the malware used in the D.N.C. attack to the G.R.U., F.S.B. or any agency of the Russian government.” It’s almost as the journalists just wanted a contrarian view to look balanced but what an odd selection if not just set up their witness to be even more important.

I want to be very clear on my next critique: I actually don’t think Jeffrey Carr is a bad person. I know he ruffles the feathers of a lot of folks in the community (mine included at times) but on the two occasions I’ve met him in person he’s been an absolutely nice person to me and was civil and well articulated. That being said, he is not an expert on attribution, not an expert on these groups, nor has any reason to be cited in conjunction with them. He’s often widely criticized in the community when he tries to do attribution and it’s often painfully full of 101 intelligence analysis failures. The NYT didn’t do him any favors by including him in this article and seriously detracted from the idea that they understood enough about this topic to cover it. Simply stated: “cyber” is not an expertise, if you are covering a niche topic like attribution or a further niche topic like Russian group attribution you need to use folks who have experience in that subject matter.

Please Stop Arguing About Attribution Without Expertise In It

This is a bit of a big request but it’d be very useful if people stop taking a stance on why attribution is difficult or not and whether or not attribution is right or not if they have never had experience in doing attribution. This is important because the journalists in this article seem to want to help bolster the case against the Russian intelligence services yet make it more confusing. At one point they try to set up their witness as some new smoking gun to be added to the case as a push back to people like President Trump.

Figure 4: Snippet from NYT Article Setting Up the Importance of the “Witness”

Attribution is not about having a smoking gun. Attribution is a good example of doing true intelligence analysis; there are no certainties and you only can come to an assessment such as low, moderate, or high confidence. Almost every single piece of data put forward in that assessment can and should have counters to it. Very reasonable counters as well. It’s why when anyone arguing for attribution argues a single piece of evidence they almost always lose the argument or look silly. It’s simply very rarely about one piece of evidence and is instead the analysis over the total data set. The attribution levied towards Russia for meddling in the U.S. elections is solid. The reason President Trump and others don’t want to accept that has nothing to do with the fact that there hasn’t been a witness or a “single piece of technical evidence produced that connects the malware used in the D.N.C. attack to the G.R.U.” it is because they do not want to accept the conclusion or the reality it presents. There’s nothing that’s going to change this. I’m convinced that if President Putin came out and said “yea it was us” we’d have critics coming forward saying how it’s a false flag operation and it’s actually not true.

But what’s the problem with people arguing these points? It detracts from the already solid assessment. It’s similar to when the FBI wanted to release IP addresses and some technical indicators during the Sony hack to talk about how they knew it was North Korea. I critiqued that approach when it happened here. The basis of my argument was that the FBI’s attribution to North Korea was likely correct but their presentation of evidence as proof was highly misleading. Obviously the FBI didn’t just use those technical indicators to do the attribution, so how could anyone be expected to look at those and be convinced?  And rightfully so people came out and argued against those technical indicators noting they could easily be wrong and that adversaries of any origin could have leveraged the IP addresses for their operations. And the critiques were correct. The technical evidence in isolation was not good. The totality of the data set though was very sound and the analysis on top of it though were very sound.

I often think of this like climate change arguments. You can have 100 scientists with a career in climate studies posit forth an assessment and then two people with absolutely no experience argue on the subject. One of the people arguing for the climate scientists’ position could grab out a single data point to argue and now the person arguing against that first person is arguing against an uninformed opinion on a single data point instead of the combined analysis and work of the scientists. The two people arguing both leave understandably feeling like they won the argument: the original assessment by the scientists was likely right but the person arguing against the data point was also probably right about their argument against that data point. The only people who lost in this debate were the scientists who weren’t involved in the argument and who’s research wasn’t properly presented.

Closing Thoughts

I never like to just rant about things, I try to use these opportunities as things to learn from. All of this is actually extremely relevant to my SANS FOR578 – Cyber Threat Intelligence course so a lot of times I write these blog posts and reference them in class. So with that theme in mind here’s the things I want you to extract from this blog as learning moments (to my students, to the journalists, and to whomever else finds it valuable).

  • If you are doing research/writing on niche topics please find people with expertise in that niche topic (Jeffrey Carr is not an expert on attribution)
  • If you are going to posit that the entire public understanding of a nation-state group’s MO has changed because a single piece of evidence you’re likely wrong (do more homework)
  • If you are going to posit that there is a witness that can change the narrative about a case please talk to people familiar on the case (determine if that type of evidence is even important)
  • If you are going to write on a topic that is highly controversial research the previous controversy first (GRIZZLYSTEPPE was entirely unrelated to the DNC case)
  • Attribution is not done with single pieces of evidence or a smoking gun it is done as analysis on complex data sets most of which is not even technical (attribution is hard but doable)
  • The most interesting data for attribution isn’t highly classified but instead just hard work/analysis on complex scenarios (classification standards don’t imply accuracy or relevancy)
  • Just because someone’s code was used by an adversary does not imply the author knows anything about how it was used or by whom (the threat is the human not the malware)
  • Stop using pew pew maps (seriously just stop; it makes you look like an idiot)