Search results for

career in scada

A Collection of Resources for Getting Started in ICS/SCADA Cybersecurity

August 28, 2016

*Last Updated Jan 2023*

I commonly get asked by folks what approach they should take to get started in industrial control system (ICS) or Operational Technology (OT) cybersecurity. Sometimes these individuals have backgrounds in control systems, sometimes they have backgrounds in security, and sometimes they are completely new to both. I have made this blog for the purpose of documenting my thoughts on some good resources out there to pass off to people interested. Do not attempt to do everything at once but it’s a good collection to refer back to in an effort to polish up skills or learn a new industry. There are also many skills that may not immediately be relevant to your job but I believe these topics all work together (ranging from analysis of threats to understanding the physical process of a gas turbine).  Rest assured, no matter how ill prepared you might feel in getting started realize that by having the passion to ask the question and start down the path you are already steps ahead of most. We need passionate people in the industry; everything else can be taught.

General Thoughts:

IT and OT/ICS cybersecurity can be very different. There’s definitely transferable skills between both fields though. Often times folks look at ICS cybersecurity and think it’s different because there are legacy systems, different network protocols, and purpose built systems like programmable logic controllers (PLCs). While those are all true in reality the biggest difference is the mission function of the systems. There are unique purposes of the systems, unique impacts in failure, unique risks, and unique threats – so applying the same cybersecurity practices meant for a different environment, with different impacts, against different risks and threats seems counter intuitive. A broad generalization that can help understand this is that in IT cybersecurity there is a large focus on the system and data. We put a lot of protection and focus on the system (patching, EDR, passwords, application whitelisting, etc.) because if an adversary gets on a system, escalates privileges, etc. it’s a bad day. We also put a lot of focus in IT cybersecurity on data (encryption in transit, encryption at rest, data loss prevention, etc.) especially with the need to protect data, people’s personal information, credit cards and financial transactions, etc. But in ICS cybersecurity it’s more systems of systems and physics. Sure we care about some data and some systems. But in reality it’s more about an adversary’s ability to take System 1 and manipulate System 2 to cause a physical manifestation in System 3. As an example, an adversary that knows how to take access an Engineering Workstation to reprogram the logic on a PLC to cause an over pressurization event in a pipeline is going to be very dangerous whether or not they use vulnerabilities, exploits, and malware to do it or just native functionality and expertise. And physics is what we care about for what is technically possible or not possible on those systems in the first place with a large focus on ensuring safety and reliability of both people, the environment, and the operations.

I would advise any new person starting in the field to spend time really focusing on the “mission” first. I.e. what is it that the plant or site is trying to accomplish. What are they in business or production for? Then apply the cybersecurity that makes sense against the risks that actually impact the mission. Coming at the problem with what “right security” looks like before understanding the business and the mission purpose will lead you astray quickly. But if you understand the point of what the operations folks are trying to accomplish it’ll allow you to be a valuable partner.

Optional Pre-Reqs

It’s always good to pick up a few skills regarding the fundamentals of computers, networks, and systems in general. I would recommend trying to pick up a scripting language as well; even if you don’t find yourself scripting a lot understanding how scripting works will add a lot of value to your skill set.

  • Learn Python the Hard Way
    • Learn Python the Hard Way is a great free online resource to teach you, step-by-step, the Python scripting languages. There’s a lot of different opinions about different scripting language. In truth, most of them have value in different situations so I’ll leave it to you to pick your own language (and I won’t tell you that you’re wrong for not learning Python, even though you are). Another good programming resource is Code Academy.
  • MIT Introduction to Computer Programming
    • MIT’s open courseware is a treasure for the community. It shocks me how many people do not take advantage of free college classes from top universities. This is the Introduction to Computer Science and Programming course. It should be taken at a slow pace but it’ll give you a lot of fundamental skills.
  • MIT Introduction to Electrical Engineering and Computer Science
    • Another MIT open course but this time focused on electrical engineering. This is a skill that will help you understand numerous types of control systems better as well as have a better grasp on how computers work.
  • Microsoft Virtual Academy
    • Microsoft Virtual Academy can be found at various locations on YouTube. I have linked to the first one; I would recommend browsing through the topic list for everything from fundamentals of networking, to fundamentals of computers, to how the Internet works.

Intro to Control Systems

Control systems run the world around us. Escalators, elevators, types of medical equipment, steering in our cars, and building automation systems are types of control systems you interact with daily. Industrial control systems (ICS) are industrial versions of control systems found in locations such as oil drilling, gas pipelines, power grids, water utilities, petrochemical facilities, and more. This section will go over some useful resources and videos to learn more about industrial control systems and ultimately “the mission” of some of the sites. If you know how a waste water treatment facility process works as an example you’re then more capable to understand the instrumentation and automation around it and the cybersecurity that would be relevant to that site.

  • The PLC Professor
    • PLC Professor and his website plcprofessor.com contains a lot of great resources for learning what programmable logic controllers (PLCs) and other types of control systems and their logic are and how they work. Some resources are free while others are paid. At some point, getting a physical kit as a trainer to learn on is going to be a requirement.
  • Control System Basics
    • This is a great video explaining control system basics including the type of logic these systems use to sense and create physical changes to take action upon.
  • What is SCADA?
    • You’ve no doubt heard the term SCADA, if you haven’t you will. It stands for Supervisory Control and Data Acquisition and is a type of ICS. This video is a nice basic approach to explaining SCADA.
  • Department of Energy – Energy 101
    • The Department of Energy has a series of Energy 101 videos to explain basic concepts of different types of energy generation, sources, etc. It’s a fantastic series that should excite you about the field while explaining key terms and concepts.
  • Wastewater Treatment Explanation Video
    • We all need wastewater treatment facilities and learning about them helps you understand how control systems work and just how complex simple tasks in life can be (if we didn’t have control systems). These types of videos are important for you to watch and learn so that you get exposed to different industries. ICS is not really a community, it’s a collection of communities.
  • Waste Water – Flush to Finish
    • Another good wastewater explanation video.
  • Refinery Crude Oil Process
    • This is a video explaining a refinery crude oil process. If these types of videos don’t excite you to some extent you may be in the wrong career field. The world around us is magnificent and learning different industries will start to help you ask the right questions which will lead to your education on the subject.
  • Natural Gas Processing
    • This is an older video (the industry has definitely become more advanced than represented here) but extremely interesting on how natural gas is harvested, processed, and transferred. Think about all the control systems that have to go into this seeminly simple process.
  • How a Compressor Station Works
    • One particularly interesting (and historically difficult to secure) portion of the ICS community is the natural gas pipeline. This video talks about natural gas to some extent but really focused on compressor stations. Compressor stations as remote sites offer numerous opportunities and challenges to defenders. In short – they’re pretty cool.
  • Chemical Engineering YouTube Channel
    • A great series of videos explaining and showing different components of chemical processing.
  • Steel from Start to Finish
    • This is an example of how steel is made. The video, like the others in this section shows an important process that can help you understand all that goes into control system security. It’s important to know the real world impacts and applications of the processes we are trying to defend to fully understand how important safety and reliability are as the main component of industrial automation.
  • How It’s Made: Uranium Part 1 and Part 2
    • Uranium mining is especially important for the nuclear power industry. There’s a lot of misconceptions around uranium and its mining; many aspects of this type of mining are similar to other types of mining but the purification, transportation, manufacturing, and utilization of uranium (highlighted in part 2 of the videos above) are particularly interesting and unique. There’s an amazing amount of industrial control systems involved in these processes.
  • Uranium Mining
    • There are multiple ways to perform uranium mining, here is an alternative way with a video by the Nuclear Energy Institute.
  • Nuclear Reactor Explained
    • This is a simplistic but extremely easy to digest explanation and animation of a nuclear reactor. Nuclear energy has a bad rap due to pop culture but is a highly clean and safe form of energy. It’s really useful to understand this process and how these systems are designed and, ideally, isolated.
  • Nuclear Power Station
    • Building from the last video, here’s another video diving deeper into nuclear power. What you should focus on here is the design and engineering that go into the safety systems. Safety systems can be bypassed, there are no ‘unhackable’ things, but this helps you to understand just how these systems are designed to be safe by default even if not build with security in mind. The Fukushima event can be observed as a worst case and extremely unlikely scenario. Learning from it will be important; here you’ll find a good video on it.
  • Thermal Power Plant
    • There are many ways to generate power; this video explains thermal power and the complexity of the environment.
  • SCADA Utility 101
    • Rusty Williams has just the right type of southern speaking which makes an audience want to learn more. The guy is awesome, the video explains SCADA from an electric utility perspective, and this is a much watch.
  • Electric Generation and Transmission
    • Didn’t get enough of Rusty? Here’s another video of him explaining the generation and transmission of electricity.
  • Copper Mining
    • There are many differences in mining depending on what you are mining, but much of the fundamentals of exploration, extraction, and processing is similar across numerous industries. This video on copper mining, skip to about 1:30 to get past the specific mine’s financials and marketing, gives a nice quick high level view of some of the process and equipment you’d find in the mining industry.
  • Gold Mining
    • Whereas the initial mining fundamentals can be the same, as noted there are many differences including how you achieve prospecting and how you process the extracted minerals. Gold mining has a number of interested aspects worth learning about.
  • Cyanidation for Extraction Processes (Animated Video and a Real Life Example)
    • Cyanide is mostly known for its form as hydrogen cyanide but in other forms (such as sodium, potassium, or calcium cyanide) it is useful in extracting precious minerals from ore and often used in gold processing. The videos above are quick animated and real life examples of the cyanidation process. The Wikipedia article here is also very useful.
  • Fundamentals of Manufacturing Processes
    • Manufacturing makes the world around us. The manufacturing industry is broad from auto, to food and beverage, to chemical, to pharmaceutical, and more. This is an MIT course that’s hosted online for free. It’s a 10 week course but it is fantastic and going through a wide variety of types of manufacturing.
  • Chemical Industry Process Equipment
    • This video is unlike the others in that it does not really show the full engineering process. However, the video talks through a wide variety of equipment that you would find in the chemical industry. I find this video useful to learn about a variety of equipment, much of which you could find in numerous industries. I would recommend taking terms you’re unfamiliar with and looking up Wikipedia articles for each after the video.
  • Beverage Manufacturing (Coca-Cola)
    • Here’s a great example of a manufacturing video focused on beverages, in this case Coca-Cola. The food and beverage industry and its manufacturing processes are wonderful forms of batch processing. This video is obviously a bit of a promotion as well but there’s great explanations throughout the video including how to make bottles (800 bottles a minute!), how to make cans, how to clean cans with sulfuric acid, and of course how to fill them with coke (1,700 cans per minute!).
  • Control Lectures
    • This is a fantastic series by Brian Douglas which covers a wide range of lectures on control systems in a very easy to process way.
  • Safety Systems
    • It’s good to get familiar with safety systems as well. Safety systems can either be active or passive. As an over simplification think of these as systems that take control of the system when an unsafe event occurs and helps to regulate it or shut it down safely. It can also be the product of good engineering instead of a dedicated system. Either way, there is a trend in the community to have integrated safety systems into one device; where the control device is also the safety device. This has cost savings but horrendous cyber security consequences and thus horrible safety consequences.
  • Safety Valves
    • Building on your understanding now of safety systems here’s an example of a safety valve in a process and how it can work to keep the operations, and more importantly the people around it, safe.
  • Industrial Disaster Explanation Videos
    • The U.S. Chemical Safety and Hazard Investigation Board has a number of videos explaining industrial disasters. This is an important resource to understand what can go wrong in industrial automation regardless of the cause (these are not cyber related but are important to understand as things that cyber could potentially cause if we are not careful). In IT, if things go wrong people do not generally die – in ICS death, injury, and environmental harm is a very real concern.

Intro to Computer and Network Security

There’s a lot of resources in the form of papers below (especially the SANS Reading Room) which are all great. However, you really need to get hands on so many of the resources are focused on tools and data sets. Try to read up as much as possible and then deeply dive into hands on learning.

  • The Sliding Scale of Cyber Security
    • I wrote this paper specifically to address the nebulous nature of “cyber security.” When people say they specialize in cyber security, what exactly does that mean? I put forth that there are 5 categories of investment that can be made. The prioritization for the value towards security should be towards the left hand side of the scale. It is ok to invest in multiple categories at once but understand the true return on investment you’re getting versus the cost.
  • VMWare
    • You’ll want to be able to set up Virtual machines (VMs) to get hands on with files and various security tools. VMWare is a great choice as is VirtualBox. VMWare has a free version you’ll want to use (Player). Don’t worry about getting Workstation or Player Pro until later when you are more experienced and want to save snapshots (copies of your VM to revert back to). Below you’ll find a sample video on VMs, feel free to Google around for better understanding.
  • Security Onion
    • You’re going to want to get hands on with the files presented in this guide; Security Onion is an amazing collection of free tools to do just that with a focus on network security monitoring and traffic analysis.
  • SANS’ SIFT
    • If you’re super cool you’ll want to get into forensics at some point; the SIFT VM from SANS is a collection of tools you’ll need to get started.
  • REMnux
    • Before you try out reverse engineering malware (REM) you’ll want to have a safe working environment to do so. This is not a beginner topic but at some point you’ll likely want to examine malware, Lenny’s REMnux VM is the safe place to do that.
  • Malware Traffic Analysis
    • Brad’s blog on malware traffic analysis is one of the best resources in the community. It combines sample files with his walk throughs of what they are and how to deal with them. You can learn a lot this way very quickly.
  • Open Security Training
    • This website is dedicated to open (free) security training. There are a number of qualified professionals who have dedicated time to teach things from the basics of security to advanced reverse engineering concept. You could spend quite a time on this website’s courses and all of them would make you more capable in this field. There are often full virtual machines (VMs), slides, and videos for the courses.
  • Sample PCAPs from NETRESEC
    • These packet capture samples are invaluable to learning how our systems interact on the network. Take a tool like Wireshark and analyze these files to get familiar with them and the practice (Wireshark will continually be your friend in any field you specialize in).
  • DEFCON Capture the Flag Files
    • DEFCON has made available their files (and often times walkthroughs) for their capture the flag contests. These range from beginner to advanced concepts in offensive security practices such as red teaming. Learning how to break into systems and how they fail is great for defense. It’s not required but it can be helpful.
  • Iron Geek
    • This is an invaluable collection of videos from conferences around the community. If you’re looking for a specific topic it’s a good idea to search these conference videos. Felt like you missed out on the last decade of security? Don’t worry most of its captured here.
  • SANS Reading Room
    • The SANS Institute is the largest and most trusted source of cyber security training. Their Reading Room is a free collection of papers written by students and instructors covering almost every topic in security.
  • Honeynet Project
    • Consider this a capstone exercise. Read up on honeypots and learn to deploy a honeypot such as Conpot. The idea is that to run a honeypot correctly you’ll have to learn about safeguarding your own infrastructure, setting up proxies and secure tunnels, managing cloud based infrastructure such as an EC2 server, performing traffic analysis on activity in the honeypot, malware analysis on discovered capabilities, and eventually incident response and digital forensics off of the data provided to explore the impact to the system. Working up to this point and then running a successful honeypot for any decent length of time really helps develop and test out a wide range of skills in the Architecture, Passive Defense, Active Defense, and (potentially in the form of Threat Intel) Intelligence categories of the Sliding Scale of Cyber Security.

Intro to Control System Cyber Security

Cybersecurity is not a new topic but in ICS it is mostly unexplored. The hardest part for most folks is learning who to listen to and what resources to read. There are a lot of “experts” out there who will quickly lead you astray; look at people’s resumes to see if they had the opportunity to do what they are speaking to you about. Because they don’t have experience doesn’t mean they are necessarily wrong but it’s an easy check. As an example, if someone calls themselves a “SCADA Security Guru” or something like a “thought leader” but they’ve only ever been a Chief Marketing Officer of an IT company, that should be a red flag. It is important to be very critical of information in this space but continually push forward to try to make the community better. Below are some trusted resources to help you on your journey.

  • S4’s ICS Onramp Series
    • Fantastic collection of quick hit videos from some of the best known folks in the industry walking through the things you need to know as an “onramp” experience for new folks to our community
  • An Abbreviated History of Automation and ICS Cybersecurity
    • This is a great SANS paper looking at the background on ICS cybersecurity. Well worth the read to make sure you understand many of the events that have occurred over the past twenty years and how they’ve inspired security in ICS today.
  • SANS ICS Library
    • This is the SANS ICS library which contains a number of posters and papers to get you started. Reference the blog as well for good explorations of topics. I write the Defense Use Case series as well which explores real and hyped up ICS attacks and lessons learned from them.
  • SCADAHacker Library
    • Joel has a fantastic collection of papers on ICS security, standards, protocols, systems, etc. Lots of valuable content in this collection.
  • The ICS Cyber Kill Chain
    • The attacks we are concerned most with on ICS take a different approach than traditional IT. This is a paper I wrote with Michael Assante exploring this and detailing the steps an adversary needs to take to accomplish their goals.
  • The Five ICS Cybersecurity Controls
    • There are a lot of cybersecurity controls that can be applied and many good standards and frameworks; however it can be overwhelming and unrealistic to try to do everything. Tim Conway and I wrote this paper after analyzing all the known ICS cyber threat groups and attacks with a focus on what would be the best strategy and approach for the “basics” of what every organization can do to be efficient and well prepared.
  • Analyzing Stuxnet (Windows Portion)
    • This is Bruce Dang’s talk at the 27th CCC in Germany on his exploration of analyzing Stuxnet. He was at Microsoft and was one of the first researchers to analyze it. This is a good understanding of the Windows portion of analysis. I show this video even though it’s a bit more advanced to highlight that there are often an IT and (operations technology) OT side of analysis.
  • Analyzing Stuxnet (ICS Portion)
    • Ralph Langer was responsible for deep diving into Stuxnet on it’s ICS payload portion. This talk gives a good understanding of the OT side of the analysis.
  • To Kill a Centrifuge – Stuxnet Analysis
    • This is Ralph Langer’s excellent paper exploring the technical details on the Stuxnet malware and most importantly the ICS specific payload and impact. It is a good idea to read through the paper and Google the terms in the paper you do not understand.
  • SANS ICS Defense Use Case #5 – Ukraine Power Grid Attack
    • This is a paper I wrote with Michael Assante, and Tim Conway released through the E-ISAC on our analysis of the Ukraine power grid attack in 2015. There are also recommendations for defense at each level of the ICS kill chain (applying 1 control is never enough to stop attacks).
  • CRASHOVERRIDE – Analysis of the Threat to Electric Grid Operations
    • Following the attack on Ukraine’s grid in 2015, there was an effort by the adversary to make their efforts more scalable with the added automation of malicious software. The malware leveraged in the Ukraine 2016 cyber attack (second ever cyber attack to cause loss of load in an electric system) was called CRASHOVERRIDE.
  • Anatomy of an Attack: Detecting and Defeating CRASHOVERRIDE
    • Much of the information around CRASHOVERRIDE wasn’t made immediately available to the sensitivity of what happened and the desire to not have the tradecraft proliferate. Once more information was being made known though Joe Slowik, an intelligence analyst at Dragos, published the findings behind the adversary ELECTRUM which was responsible for CRASHOVERRIDE.
  • TRISIS Malware: Analysis of Safety System Targeted Malware
    • In 2017 there was an attack on a Saudi Arabian petrochemical company. Dragos and FireEye completed analyses of the malware (FireEye called it TRITON and Dragos called it TRISIS; they did not coordinate with each other and did not know each firm was working on the malware analysis until a week or so before publication). Despite initial reporting in media by parties not involved in the analysis, Saudi Aramco was not the victim of the attack. Saudi Aramco was actually the incident response team that went and helped out the facility.
  • PIPEDREAM – The Most Flexible & Capable ICS Malware To Date
    • In 2022 PIPEDREAM was discovered as the first cross-industry scalable and repeatable ICS malware. In this talk I go through the capability and what we are allowed to say about it while focusing people on defensive strategies against it. PIPEDREAM was truly a game changer for the industry.
  • CHERNOVITE’s PIPEDREAM
    • This is the whitepaper by the Dragos team on their analysis of PIPEDREAM.
  • The Industrial Cyber Threat Landscape
    • This is the testimony I gave in 2018 to the Committee on Energy and Natural Resources of the United States Senate. It contains recommendations for the community and a discussion of the cyber threat landscape.
  • ICS Threat Intelligence: Moving from the Unknowns to a Defended Landscape
    • This is a talk I did at the SANS ICS Summit that gets into why our threat landscape is largely unknown, what we can do about it, and how we can really move the community forward by incorporating intelligence instead of theoretical best practices.
  • Perfect ICS Storm
    • Glenn wrote a great paper looking at the interconnectivity of ICS and the networks around them with considerations on how it impacts monitoring and viewing the control systems.
  • Network Security Monitoring in ICS 101
    • Here is a great intro talk on network security monitoring in an ICS by Chris Sistrunk at DEFCON 23. Network security monitoring is exceptionally useful in ICS because it can be done with minimal data sets and passively which works inside the confines of the safety and reliability requirements of an ICS network.
  • Dragos Webinars and Blogs
    • The Dragos webinars and blogs are highly informative on performing threat analysis, defense, and response as it pertains to ICS cyber threats. They are very rarely marketing or promotional and far more content-driven.
  • S4 Videos
    • The S4 conference run by Dale Peterson is a great community resource. He has posted a number of the conference presentations which will give you a great look at the ICS security community especially from the researcher perspective.
  • Defense Will Win
    • Dale Peterson’s excellent S4 talk that has an upbeat attitude of “defense will win.” This is something I completely agree with and for a few years now I have been championing the phrase “Defense is Doable” to help folks not get down when it comes to ICS cyber security. It may seem like the hardest challenge out there but it’s worthwhile and these are the most defensible environments on the planet; maybe not the most defended – but we will get there.
  • Dragos Year in Review 2017 and the following years
    • Each year Dragos puts out a year in review that covers threats, vulnerabilities, and lessons learned across incident response and assessments. They were made to provide ground-truth base metrics and stats to the community about what is going on around the community. They are light on marketing language and focused on sharing insights useful to the community.
  • The Industrial Cyberthreat Landscape
    • My keynote at RSA detailing to a broad audience what is unique about OT/ICS and the threats we face with the latest insights from the frontlines.

Recommended ICS Cybersecurity Books

  • Rise of the Machines: A Cybernetic History
    • It seems a bit odd to put a non-technical book as my first recommendation but I assure you it is with reason. Dr. Thomas Rid wrote this book to attempt to fully understand the history, implications, and usages of the word “cyber”. Delightfully, control systems have a major role throughout the book. It was control systems that got us started with “cybernetics” which is eventually where we would have the “cyber” word that fills our daily lives.
  • Handbook of SCADA/Control Systems Security
    • Robert (Bob) Radvanovsky and Jacob (Jake) Brodsky put together this wonderful collection of articles from people throughout the community. It covers a wide variety of topics from a wide variety of personalities and professionals.
  • Protecting Industrial Control Systems from Electronic Threats
    • Joe Weiss is a polarizing individual in the community but only because of how passionately he cares about the industry and how long he’s been in the community. Many of us here today in the community owe much to Joe and this book offered an early look at control system cybersecurity.
  • Industrial Network Security
    • Eric Knapp and Joel Langill wrote this book looking specifically at the network security side of ICS. It’s a fantastic resource exploring different technologies and protocols by two professionals I’m glad to call peers and friends.
  • Hacking Exposed: Industrial Control Systems
    • This book takes a penetration testing focus on ICS and talks about how to test and assess these systems from the cybersecurity angle while doing it safely and within bounds of acceptable use inside of an ICS. It’s written by Clint Bodungen, Bryan Singer, Aaron Shbeeb, Kyle Wilhoit, and Stephen Hilt who all are trusted professionals in the industry.
  • Santa and Me: The SCADA Before Christmas
    • The third book I’ve written which is a lighthearted twist on the classic “twas the night before Christmas” poem. You’re not going to learn a ton but for those of you with children it’s a great way to expose them to our industrial world.
  • Countdown to Zero Day: Stuxnet and the Launch of the World’s First Digital Weapon
    • Kim Zetter wrote this masterpiece on Stuxnet and much of the geopolitical and historical context around it as well as the investigation into it.
  • Sandworm: A New Era of Cyberwar and the Hunt for the Kremlin’s Most Dangerous Hackers
    • Andy Greenberg wrote a fantastic book looking at the Sandworm threat group which was responsible for the 2015 cyber attack on Ukraine’s electric system; this was the first time ever a cyber attack caused electric outages.

Recommended Professional Training

You in no way need certifications or professional training to become great in this field. However, sometimes both can help either for job opportunities, getting a raise, or polishing up some skills you’ve developed. I highly encourage you to learn as much as you can before getting into a professional class (the more you know going in the more you’ll take away) and I encourage you to try to find an employer to pay your way (they aren’t cheap). If your employer doesn’t have a training policy it’s a good time to try and find a new employer. Here are two professional classes I like for ICS cyber security training (I’m biased because I teach at SANS but I teach there because I believe in what they provide).

  • Department of Homeland Security and Department of Energy Training
    • The ICS-CERT and Idaho National Labs provide a variety of online and in person training. One of the most well known is the ICS 301 class which is a 5-day introduction to ICS hosted in Idaho Falls, Idaho. It is a free course and highly recommended.
  • SANS ICS 410 – ICS/SCADA Essentials
    • This class is designed to be a bridge course; if you are an ICS person who wants to learn security, or a security person who wants to learn ICS, this course offers the bridge between those two career fields and offers you an introduction into ICS cyber security. Over the years this course has become of staple of people entering our community.
  • SANS ICS 515 – ICS/SCADA Active Defense and Incident Response
    • This is the class I authored at SANS teaching folks about targeted threats (such as state adversaries or well funded crime groups) that impact ICS and how to hunt them in your environment and respond to incidents. More than just focusing on the threats though this class helps you understand the risks our community faces and how to develop strategies against them with hands on practitioner focused labs and training.
  • SANS ICS 612 – ICS Cybersecurity In Depth
    • An absolute gem of a class that teaches a ton across foundational ICS security, architecture, passive defense, etc. topics. I say foundational not because it’s entry level but because it should be required for anyone joining the field. It’s a hands on class with a full control system setup and is most students’ best opportunity to get hands on with real industrial equipment and processes.
  • Assessing and Exploiting Control Systems
    • Justin Searle is the author of SANS ICS410 and he also made Assessing and Exploiting Control Systems. This course is an introduction to vulnerability and penetration testing of these systems with a focus on everything from PLCs to RF. A lot of the focus tends to be on smart grid and electric but there are elements for everyone. The same class is also hosted at SANS from time to time, but it is significantly cheaper to find it at BlackHat if you can grab a spot. The class moves around so the link above is for an old class but Google the name and where it’s being hosted to find it.
  • Dragos 5 Day Training
    • Dragos hosts a five day training that covers an introduction to ICS, assessing ICS, threat hunting, and security monitoring. Uniquely, it provides access to industrial ranges and is hosted in Houston, Texas, Hanover, Maryland, Dubai, UAE, and Melbourne, Australia. The industrial ranges and physical equipment make for an exciting educational experience. However, the class is only open to those in the asset owner and operator community (e.g. working at an energy, manufacturing, auto, etc. company) such as Dragos customers and partners. Most of the other training in the market tries to avoid vendor tools and practices to be vendor-neutral. I love this and engage this way even in my own SANS class. However, the reality is in your day-to-day work you’re going to be working with vendor tools and want to learn from their best practices too. This is a unique opportunity to train with those operating on the front lines of the community and understand their specific approaches.

Recommended Conferences

No matter how much time you spend reading or practicing eventually you need to become part of the community. Contributions in the form of research, writing, and tools are always appreciated. Contributions in the form of conference presentations are especially helpful as they introduce you to other interested folks. The ICS cybersecurity community is an important one on many levels. It’s one of the best communities out there with hard working and passionate people who care about making the world a safer place. Below are what I consider the big 5. These conferences are the ones that are general ICS cyber security (not a specific industry such as API for oil and gas or GridSecCon for electric sector) although those are valuable as well.

  • SANS ICS Security Summit
    • For over fifteen years the SANS ICS Security Summit has been a leading conference on bringing together researchers, industry professionals, and government audiences. The page above links to the various SANS ICS events but look for the one that says “ICS Security Summit” each year. It is usually held at Disney World in Orlando Florida. Its strong suit is the educational and training aspects not only because of the classes but also because of the strong industry focus.
  • DigitalBond’s S4
    • The S4 conference is a powerhouse of leading ICS security research. Dale puts on a fantastic conference every year (now with a European and Japanese venue as well each year) that brings together some of the most cutting edge research and ideas. S4 in the US is often held in January in Florida.
  • The ICS Cyber Security Conference (WeissCon)
    • Affectionately known as WeissCon after its founder Joe Weiss, the conference is now owned and operated by SecurityWeek and usually runs in October at different locations each year in the US (Georgia is usually a central location for the conference though). The conference brings together a portion of the community not often found at the other locations and has a strong buy-in from the government community as well as the vendor community.
  • The ICS Joint Working Group (ICSJWG)
    • The ICSJWG is a free conference held twice a year by the Department of Homeland Security. I often encourage people to go to the ICSJWG conference first as a type of intro into the community, to then go to the SANS ICS Security Summit for more view into the asset owner community and to get training, then go to S4 for the latest research, to go to WeissCon to see some of the portions of the community and vendor audience not represented elsewhere, and finally to CS3Sthlm to get an international view. It is perfectly ok to go to all five of the big conferences a year (I do) but if you need a general path that is the one I would follow initially.
  • CS3Sthlm
    • CS3Sthlm used to be known as 4SICS and is held every year in Stockholm, Sweden. It is one of the leading ICS security conferences in the world (I consider it one of the “big five”) and it is in my opinion the best ICS security conference in Europe. The founders Erik and Robert are some of the friendliest people in the ICS community and have a wealth of experience to share with folks from decades defending infrastructure.
  • Dragos Industrial Security Conference (DISC)
    • DISC is the Dragos annual conference however it is unique in that it is entirely dedicated to research and insights into the ICS cyber threats and responding to them. The conference is 100% free and open to those in the industrial asset owner and operator community. It happens every year on November 5th in Maryland, USA.

This is just a small collection of a lot of the fantastic resources out there. Always fight to be part of the community and interact – that is where the real value in learning is. Never wait to have someone show you though, even the “experts” are usually only expert in a few things. It is up to you to teach yourself and involve yourself. We as a community are waiting open armed.

 

What a Record Setting Investment into the ICS/OT Cybersecurity Market Means to Me

December 8, 2020

“ICS cybersecurity? What’s that? Is it worth doing? Can it be done? No it cannot, I heard…Even if you did the market isn’t large enough to support it long term”

Dragos, Inc. announced today its C-Series financing which is the largest investment ever into an industrial control system (ICS) / operational technology (OT) cybersecurity company. The investment is $110M for a total raised of more than $158M over the four years the company has been around. As the co-founder and CEO it fills me with great pride because of the team Dragos has assembled and our amazing customers who have truly partnered with us on our collective journey. Seeing them leverage our technology, services, and intelligence to make their companies more secure and further their maturity is something amazing to behold. Most citizens never understand or gain insight into how hard their infrastructure companies work to provide safe and reliable services and goods; I can tell you first hand this community works amazingly hard. There’s a lot of unknown passionate professionals running proof of concepts, implementing projects, advocating internal to their org, getting trained, working long hours, etc. all to allow companies like Dragos to exist to serve this community. Thank you.

I’ve written before on what it’s like to raise venture capital, you can view that blog here. In this post I want to walk through some of the challenges I’ve faced for Dragos from an investment perspective and the path along the way explicitly to help explain what I think this investment means for the broader OT/ICS cybersecurity market and community. I’ll speak a lot about our journey so far but the point isn’t about Dragos’ financing but instead the amazing realization that OT cybersecurity is worth doing, a large enough market to do it in, and that it can be done.

I will say without any intent to hype it up that I do believe this is a watershed moment and I hope to share that perspective with you.

No alt text provided for this image

I started Dragos a little over four years ago with my co-founders Jon Lavender and Justin Cavinee who had worked with me at the National Security Agency on our mission of identifying and responding to threats to ICS worldwide. We started the company not out of the desire to create a company or technology. To be candid we all abhorred the idea of becoming a software vendor after a career of being practitioners and community members in this space. But we did so out of a stark realization that the industry was changing and the threats were becoming more numerous and aggressive. What we were seeing as “answers” were a copy/pasting of IT security best practices into the ICS networks with little regard for the unique mission and threats those systems faced. I had authored the SANS ICS515 class on ICS incident response and network monitoring to help educate and train the workforce but realized that the only way to scale human knowledge fast enough in the face of what we were seeing was to also ensure those practitioners had ICS specific cybersecurity technology as well. We needed to do this in a company that would refuse to get acquired and be a long term player to put a dent in the problem. It’s my view though that to make the best technology you need the best people and you need to be hyper informed on the changing risk landscape if you’re going to counter it. So we built Dragos focused on our visibility, monitoring, and response technology but also with a professional services team of ICS cybersecurity experts to do everything from threat hunting and pentesting to incident response and architecture reviews while being trusted advisors to our customers. To inform everything we did and to help educate our community we built an intelligence team to identify and track threats specifically focused on ICS. To date we track 14 state adversaries explicitly doing so. I say all that in context of this fund raising to say – most investors hated our approach.

We got off the ground with a Seed investment from DataTribe. The only reason they invested was they had a background in the intelligence community and military and understand that we were mission focused. I’m sure they didn’t know much about what we wanted to do but they knew the problem was important and we were the team that would stop at nothing to satisfy the mission. When I went to raise the Series A round of $10M to finance our operations I met with and pitched well over 100 investors. Many of them sought us out to learn more about ICS/OT cybersecurity. The broader OT security market which encompasses ICS and the industrial internet of things (IIoT) (not to be confused with IoT, Alexa and a Gas Turbine have little in common) was very interesting to investors but none of them seemed to believe it was worth focus. I received pushback from the investors that fell into three distinct camps; these camps were challenges I heard from plenty of non-investors as well that I had encountered over the years:

  • Companies have tried to do ICS security before and failed. It’s not doable. People don’t care past regulation or fear. These companies won’t change. OT specific cybersecurity will never be successful.
  • The market is too small. If you’re interested in getting quickly acquired we’ll invest but if you’re interested in going the distance, we’re not along for the ride. The OT market is so niche.
  • IT and OT are converging. There won’t be an OT network in the years to come. IT? OT? It’s all just T. Enterprise cybersecurity will be rolled into the plants there’s no need for OT specific cybersecurity. OT specific security isn’t worth doing.

Finding people that didn’t agree with the three points above in the broader market was hard. In reality, very few of the practitioners in our small ICS security community believed such things. I think many people in our community have wondered if it’ll take some giant cyber attack on ICS to get people to take it seriously, but my view was “we’ve had all the attacks we need.” Every industry has representative attacks and stories. That wasn’t the issue or need and no one should hope for it. The reality was there wasn’t a large investable market which means for the business there was no obvious need to address this risk. I viewed building a technology company and staying around long term as necessary to getting these companies resources for workforce development, training, etc. as much as anything else.

On the three points here’s where I disagree in order:

  • Just because a few companies have failed on this path doesn’t mean that it won’t be successful. But more importantly the efforts I’ve seen failed before were largely re-skinned IT security efforts with some ICS marketing. It was obvious they were going to fail. This community does care about its infrastructure but we are a community of people who understand what does and doesn’t work. Our infrastructure members will invest beyond regulation and fear but not in things they don’t believe will work. Also undeniably over the last decade there has been a larger and more proactive community advocating cross company cross vendor cross conference etc. on what does and doesn’t work.
  • The OT market is huge. It’s hard to put a real number on it; some orgs claim 20-30B, but however you size it, it’s huge. Most people associate it with electric utilities and oil and gas. But manufacturing, rail, water, mining, transportation, etc. should come to mind. And the physical systems in the data center. And building automation systems. And airports. And and and. It’s actually harder to find companies that don’t have OT than those that do. These businesses aren’t in the business of selling emails. They produce goods, interact with the physical world, and provide services all powered by OT. The major risk is in the OT and when executives are aware of that and have an answer to address it they will in a way a majority of investors I’ve met have misunderstood.
  • IT and OT convergence happened a decade ago. I’m near tired of hearing how it’s “coming.” We have had Windows in ICS/SCADA/DCS/OT/etc. networks for more than a decade. The convergence is actually the digital transformation of these organizations coming at the same time of ICS specific adversaries. But no matter what the underlining operating system is that’s not the point. The point of OT cybersecurity is that the mission is different. The threats are different. The risks are different. The culture to get the job done is different. The challenges are different to succeed. Therefore the way you secure it will be different. I’m not saying all IT security is useless in the plants. There’s plenty we can learn from and adopt. What I’m saying is the unique and most critical part of these businesses deserves a specific focus that understands and accounts for the people, culture, process, technology, mission, risks, threats, etc. of that side of the business. To not accept that is naïve.

When I went to investors saying we wanted to focus exclusively on OT cybersecurity and we wanted to partner with our customers not just in providing technology but also having smart people and actual insights to provide it didn’t go so well with most of them. You cannot describe all the VCs in one broad stroke just like you cannot describe any group; and I’ve met and really enjoyed getting to know plenty of VCs, but to say the vast majority didn’t understand this market is an understatement.

Not only were the pushbacks from above tangible but also “and you want to hire experts to do professional services? Won’t that lower the margins on the software sales? I don’t think that’s a good idea.” But even extremely mature companies are relatively immature in their OT cybersecurity journey and need a partner not just a technology. That’s also how we get better. So it was non-negotiable. Our team’s people are and were our secret weapon. For all the words like innovation and disruption that get flaunted in Silicon Valley it was interesting how many investors we scared away by simply being different than what they had seen before. The reason we were successful in our A round was largely due to Energy Impact Partners and AllegisCyber. AllegisCyber is a VC built by former operators (ran companies before) which helped them see what we were doing beyond a spreadsheet. Energy Impact Partners though deserved the lion’s share of the credit as they are a VC built by the electric companies. Southern Company, National Grid, Xcel, Oklahoma Gas and Electric, etc. and those companies knew first hand how important OT was and the necessity of a full solution.

No alt text provided for this image

By the time the B round came about, a $37M investment, a lot of the naysayers of OT cybersecurity in the context that it couldn’t be done fell to the side. We were flooded with investors who wanted to invest. But, most of them were talking about and thinking about acquisitions. My view was and is that the OT cybersecurity market is so large that it can not only support one company IPO’ing or being of that size but multiple. This was not a widely shared view to say the least. Most of the 70+ investors calling who were interested in us because of the importance of ICS quickly had the wind taken out of their sails and the conversations would noticeably shift when I mentioned our vision was to be a long term company and not to build to be an acquisition target. To them it was clear now that OT cybersecurity could be done. They agreed it should be done. They did not believe it was a large market. Luckily, this time around we had Canaan which is a well respected Silicon Valley VC to add that type of credibility to our name in those circles but believed in the mission and in the market size. They saw what many at the time didn’t and I think that a big reason for that is how involved they had been with pharmaceutical companies and others realizing that maybe there was something to this OT market. Vision is an easy word to say and hard in practice. (Hear their perspective in this blog here.) We were fortunate to also be joined by a direct strategic investment from National Grid, Emerson, and Schweitzer Engineering Labs. Obviously those three understood OT and have continued to be great partners.

No alt text provided for this image

To claim that the C round is some sort of finish line is obviously silly. It’s really just the starting point. But to have a record setting $110M investment isn’t about Dragos. It’s about our OT cybersecurity community and the broader market. It’s a massive signal to everyone that not only is OT cybersecurity important (most everyone gets that), and is doable (people starting to realize that), but that the market is large enough to make it a worthy investment (new to most). This time, instead of taking calls from all the interested investors, we focused on letting the industry tell the story. The only thing more powerful to me than a large investment is having the asset owners and operators themselves tell their story. Thus, for the C-round we had the venture arms of National Grid and Koch Industries lead the round with investments from Saudi Aramco and HPE as well. One of the largest electric and natural gas companies in the world, with the largest manufacturer in the world, with the largest oil and gas company in the world, with one of the largest manufacturers in the supply chain in the world. That’s a powerful story. That’s a signal to everyone including the investors that the OT cybersecurity market is large, worthy of investment, and will be around for a long time. These are industry leaders saying not only do we believe in the technology we’re seeing but this market and category is important to our businesses at a strategic level. That’s a powerful signal to the other companies in their space and broader. That’s the new piece here. That’s the story. That’s what I think serves as a watershed moment. The community itself standing up and saying “we’ll get this done ourselves, it’s of strategic value.”

There are plenty of savvy investors and VCs that I’ve had the privilege to get to know. But across the broad swath of them the conversations have changed as they learned about our C round. And it’s not just investors. I’ve run into the naysayers every month and sometimes every week of my entire career. It gets tiring. And don’t even get me started on “you’re technical? Are you sure you can be the CEO? Shouldn’t you bring in someone else?” discussions. That’s a less polite blog I’ll write some time. But I know many of you in our community run into the same conversations about our ICS community. To all of you I will tell you now that I can say with great confidence the folks telling you that it “can’t be done” “shouldn’t be done” or “cannot be done long term” are on the wrong side of the argument. We have a lot of work to be done. But this is a community milestone.

It’s not a Dragos only story. The work by so many firms, so many passionate professionals, students, practitioners, leaders, government agencies, and even competitors have been a part of getting here. And here we stand on a larger platform than ever before, as an OT/ICS cybersecurity community, to tell our story.

If you’re in our community we at Dragos hope this provides some ammo for you to propel your ICS security journey forward. If you’re not in the ICS security community and you want to join, we hope this is a good signal to you that you can have a wonderful career here and its worth your time. Your local power company, water utility, oil and gas, manufacturing, rail, data center, mining, etc. companies are hiring. Go check them out. Their mission is worth investing in.

Common Analyst Mistakes and Claims of Energy Company Targeting Malware

July 13, 2016

A new blog post by SentinelOne made an interesting claim recently regarding a “sophisticated malware campaign specifically targeting at least one European energy company.”  More extraordinary though was the claim by the company that this find might indicate something much more serious: “which could either work to extract data or insert the malware to potentially shut down an energy grid.” While that is a major analytical leap, we’ll come back to this, the next thing to occur was fairly predictable – media firms spinning up about a potential nation-state cyber attack on power grids.

I have often critiqued news organizations in their coverage of ICS/SCADA security when there was a lack of understanding of the infrastructure and its threats but this sample of hype originated from SentinelOne’s bold claims and not the media organizations. (Although I would have liked to see the journalists validate their stories more). News headlines included “Researchers Found a Hacking Tool that Targets Energy Grids on the Dark Web” to EWeek’s “Furtim’s Parent, Stuxnet-like Malware, Aimed at Energy Firms.” It’s always interesting to see how long it takes for an organization to compare malware to Stuxnet. This one seems to have won the race in terms of “time-to-Stuxnet”, but the worst headline was probably The Register’s with “SCADA malware caught infecting European energy company: Nation-state fingered”. No this is not SCADA malware and no nation-states have been fingered (phrasing?).

The malware is actually not new though and had been detected before the company’s blog post. The specific sample SentinelOne linked to, that they claim to have found, was first submitted to VirusTotal by an organization in Canada on April 21st, 2016. Later, a similar sample was identified and posted on the forum KernelMode.info on April 25th, 2016 (credit to John Franolich for bringing it to my attention). On May 23rd, 2016 a KernelMode forum user posted on their blog some great analysis of the malware. The KernelMode users and blogger identified that one of the malware author’s command and control servers was misconfigured and revealed a distinct naming convention in the directories that very clearly seemed to correlate to infected targets. In total there were over 15,000 infected hosts around the world that had communicated to this command and control server. This puts a completely different perspective on the malware that SentinelOne claimed was specifically targeting an energy company and it’s obvious it is most certainly not ICS/SCADA or energy company specific. It’s possible energy companies are a target, but so far there’s no proof of that provided.

I do not have access to the dataset that SentinelOne has so I cannot and will not critique them on all of their claims. However, I do find a lot of the details they have presented odd and I also do not understand their claims that they “validated this malware campaign against SentinelOne [their product] and confirmed the steps outlined below [the malware analysis they showed in their blog] were detected by our Dynamic Behavior Tracking (DBT) engine.” I’m all for vendors showcasing where their products add value but I’m not sure how their product fits into something that was submitted to VirusTotal and a user forum months before their blog post. Either way, let’s focus on the learning opportunities here to help educate folks on potential mistakes to avoid.

Common Analyst Mistake: Malware Uniqueness

A common analyst mistake is to look at a dataset and believe that malware that is unique in their dataset is actually unique. In this scenario, it is entirely possible that with no ill-intention whatsoever SentinelOne identified a sample of the malware independent from the VirusTotal and user forum submission. Looking at this sample and not having seen it before the analysts at the company may have made the assumption that the malware was unique and thus warranted their statement that this campaign was specifically targeting an energy company. The problem is, as analysts we always work off of incomplete datasets. All intelligence analysis operates from the assumption that there is some data missing or some unknowns that may change a hypothesis later on. This is one reason you will often find intelligence professionals give assessments (high, medium, or low confidence assessments usually) rather than making definitive statements. It is important to try to realize the limits of our datasets and information by looking to open source datasets (such as searching on Google to find the previous KernelMode forum post in this scenario) or establishing trust relationships with peers and organizations to share threat information. In this scenario the malware was not unique and determining that there were at least 15,000 victims in this campaign would add doubt that a specific energy company was the target of the campaign. Simply put, more data and information was needed.

Common Analyst Mistake: Assuming Adversary Intent

As analysts we often get familiar with adversary campaigns and capabilities to an almost intimate level knowing details ranging from behavioral TTPs to the way that adversaries run their operations. But one thing we as analysts must be careful of is assuming an adversary’s intent. Code, indicators, TTPs, capabilities, etc. can reveal a lot. They can reveal what an adversary may be capable of doing and they should reveal the potential impact to a targeted organization. It is far more difficult though to determine what an adversary wishes to do. If an adversary crashes a server an analyst may believe the malicious actor wanted to deny service to it whereas the actor just messed up. In this scenario the SentinelOne post stopped short of claiming to know what the actors were trying to do (I’ll get to the power grid claims in a following section) but the claim that the adversary specifically targeted the European energy company is not supported anywhere in their analysis. They do a great job of showing malware analysis but do not offer any details around the target nor how the malware was delivered. Sometimes, malware infects networks that are not even the adversary’s target. Assuming the intent of the adversary to be inside specific networks or to take specific actions is a risky move and even worse with little to no evidence.

Common Analyst Mistake: Assuming “Advanced” Means “Nation-State”

It is natural to look at something we have not seen before in terms of tradecraft and tools and assume it is “advanced.” It’s a perspective issue based on what the analyst has seen before. It can lead to analysts assuming that something particularly cool must be so advanced that it’s a nation-state espionage operation. In this scenario, the SentinelOne blog authors make that claim. Confusingly though, they do not seem to have even found the malware on the energy company’s network they referenced. Instead, the SentinelOne blog authors claimed to have found the malware on the “dark web”. This means that there would not have been accompanying incident response data or security operations data to support a full understanding of this intrusion against the target, if we assume the company was a target. There are non-nation-states that run operations against organizations. HackingTeam was a perfect example of a hackers-for-hire organization that ran very well-funded operations. SentinelOne presents some interesting data and along with other data sets this could reveal a larger campaign or even potentially a nation-state operation – but nothing presented so far supports that conclusion right now. A single intrusion does not make a campaign and espionage type activity with “advanced” capabilities does not guarantee the actors work for a nation-state.

Common Analyst Mistake: Extending Expertise

When analysts become experts on their team in a given area it is common for folks to look to them as experts in a number of other areas as well. As analysts it’s useful to not only continually develop our professional skills but to challenge ourselves to learn the limits of our expertise. This can be very difficult when others look to us for advice on any given subject. But being the smartest person in the room on a given subject does not mean that we are experts on it or even have a clue of what we’re talking about. In this scenario, I have no doubt that the SentinelOne blog authors are very qualified in malware analysis. I do however severely question if they have any experience at all with industrial and energy networks. The claim that the malware could be used to “shut down an energy grid” shows a complete lack of understanding of energy infrastructure as well as a major analytical leap based on a very limited data set that is quite frankly inexcusable. I do not mean to be harsh, but this is hype at its finest. At the end of their blog the authors note that if anyone in the energy sector would like to learn more that they can contact the blog authors directly. If anyone decides to take them up on the offer, please do not assume any expertise in that area, be critical in your questions, and realize that this blog post reads like a marketing pitch.

Closing Thoughts

My goal in this blog post was not to critique SentinelOne’s analysis too much, although to be honest I am a bit stunned by the opening statement regarding energy grids. Instead, it was to take an opportunity to identify some common analyst mistakes that we all can make. It is always useful to identify reports like these and without malice to tear apart the analysis presented to identify knowledge gaps, assumptions, biases, and analyst mistakes. Going through this process can help make you a better analyst. In fairness though, the only reason I know a lot about common analyst mistakes is because I’ve made a lot of rookie mistakes at one point or another in my career. We all do. The trick is usually to try not to make a public spectacle out of it.