Walking the tightrope between best practice and negligence – things to consider when responding to targeted or advanced persistent threats

Over the course of the last few months I’ve been asked several times how incident responders should react to notification that their company has been breached by targeted or advanced persistent threats (T/APT).  In every case I offer the same, simple insights:

1. People count. A trained, analytically curious team will have a far greater chance of detecting and responding than an untrained, mechanical team with great tools. Teams must be allowed (encouraged) to talk to their peers in other companies (their detection becomes your prevention!) and your organization must be aligned from top to bottom to allow your trained, analytically curious team the ability to take the necessary steps to deter attackers in real time –without waiting for management to give permission.
2. Pick a standard for your defense in depth and implement it. Models might include those published by NIST, ISO, or SANS. Pick one, prioritize your spend, and implement. Even if it takes five years, you’ll be better off tomorrow that you are today.
3. Good process is a must! Your security team must know how to respond to information without having to think. The military calls this ‘command by negation’ –a commander can act within a specific framework before having to ask permission to deviate from the predetermined plan. When an incident occurs, no member of the security team should have to wonder about what to do next. Preplanned, practiced processes allow teams the ability to focus on the incident, not how to communicate about the incident, how it should be coordinated, or who does what when.

Diving a little deeper, there is one area where I grow increasingly concerned:

When responding to a targeted or advanced persistent event, should the incident response team leave compromised systems online to collect valuable intelligence, or should they follow long-established guidelines of pulling the affected computer(s) offline to stop the attack?

While other options exist (I’m a real fan of Kill Chain matrices!), from a technician’s perspective I struggle with the dichotomy of my roots as a systems administrator battling my more mature intelligence and analytic perspectives –Ops versus Intel.

I posted my draft to the Red Sky Alliance for comment. Almost immediately one of the members jumped in offering  “I fully subscribe to the notion of not playing whack-a-mole with the adversary by reacting before fully assessing (as best is possible) the nature of the and depth of entrenchment.  If you have an entrenched APT presence the chances of multiple variants and CC vectors are high.  We have isolated “one off” “sleepers” with dormancy timers as long as 12 months.    We have also observed highly disciplined adversaries that systematically establish multiple front and back door methods of access and only expose them as required to initially validate capability. ”[1]

Many today would argue that leaving compromised computers online is a required activity. In fact, I reviewed a paper recently where this was issued as the only option and three companies in the last two weeks have told me that they were counseled to leave compromised systems online. From a tactical perspective, it’s true. One way to really obtain a good picture of the problem is to allow them to carry out objectives and build a picture. From that data, profiles can be built allowing distillation of all information into (hopefully) a few pieces of information that can be used to thwart many attempts.

The other side of my brain says this: If I’m the CEO of a company and my CISO comes to my office and tells me that he wants to leave systems online post-compromise, the first thing I’m going to want to know is how he/she intends to protect the intellectual property, PII, credit cards, etc., being targeted. We’ve probably spent millions (more?) of dollars developing information and business that might be walking out the door on the back of rogue connections. If the CISO has a less than mature organization, and can’t show me how his smart people, tested process, and good tools are going to allow the activity but prevent the flow of information outbound, I’m probably going to shut the effort down and err on protecting company data.

If you were a hockey coach, would you take your goalie out of the game for a quarter to watch how the other team shoots on goal? Probably not. The better option might be to watch the other team fire shots on a defended goal to see how they handle the defenses posed by your team. Leave the goalie in, tape the game. Analyze it as many times as needed later.  Be ready to block all known outbound accesses including C2 and, if necessary, be prepared to disconnect from the Internet (companies can and do lose of control of their domain controllers, credentials and network!). When machines are compromised and data is leaving the company for a distant overseas location, a CISO with the most mature, highly skilled team had better scratch his head twice before leaving those machines online for any length of time. Even the most skilled team with great tools risks losing valuable intellectual property. Obviously this is a risk management decision for the CISO, but lets consider this. If you’re wrong and you lose company data, are you willing to be held accountable?  At what point does analytics turn to negligence? What is your defining moment –the tollgate; one that will tell you you’re about to cross the line from CISO to cowboy? How much longer will CIOs, CEOs and boards be accepting of practitioners who let valuable data leave the environment to learn how to stop it?

In every case, when asked if a system should be left online, or taken offline, my first question is “How skilled is your team?” If the team is trained, can operate on the fly to protect the company, has deep curiosity, likes watching packets at O-dark-thirty, not afraid to go deep into packet TiVO, can focus on analytics and not process, lives on Mountain Dew and have the ability to actually stop data from leaving, my response would probably be much more favorable than not. If your team is on the other end of the skill spectrum, hired because they were cheap labor, has just enough capability to keep you out of compliance trouble, and go home at 5, you’re probably going to be in a heck of a lot of hot water when CD-sized compressed encrypted files start flying out of your networks during the early morning hours.

Thoughts in summary:

• First, see my opening. Basic blocking and tackling counts. Do this now.
• Next, companies who choose to leave infected computers online to determine scope and penetration of an APT infection should only do so under the guidance of skilled, full time personnel who will monitor and stop intellectual property losses. If you don’t have personnel skilled in handling targeted or advanced threats, don’t do it. Don’t experiment unless you’re willing to lose (several times and possibly all of your intellectual property) before you win.
• If you’d like to operate in ‘intelligence gathering’ mode, hire a consultant with documented successes and experience in the APT space. There are several who are brilliant in this space. I’d be happy to offer a selection if asked.
• Companies (all companies) should have some ability to record all network activity. While easier said than done, this is an activity that allows the coach to replay the tape. You won’t stop intellectual property losses, but it may help next time. Can’t afford the cost? Hire a service. This is a must.
• Companies should be prepared to disconnect from the Internet –especially those with little or no experience fighting targeted or advanced attackers. In fact, any organization that can operate ‘off-Internet’, but does not have a plan for isolating itself for an extended period of time, while maintaining business critical processes is, in my opinion, negligent.

In today’s environment, a serious cyber incident is the single most likely disaster any organization will encounter. Directly observing your Intellectual Property go out the door is never a viable option.

Who would you rather be when your company shows up in the Wall Street Journal? The smart CISO with a small curious team, solid tested process and the right tools, or the CISO who ends up caught in a civil suit for malpractice facing the end of litigation support in his professional liability policy. How would you feel if you ordered computers to be left online in the face of massive data loss? This is not a ‘don’t do it’. It’s a warning to be smart. Be prepared.