On the 4th of October we released Dissect1, our incident response framework, as open-source software. To give you an idea of how Dissect is used by our FoxCERT team in the real world, we want to take you along for a high-level overview of the first 72 hours of an incident response engagement. It is based on a real incident and gives a good overview of our average IR engagement. Of course, the details have been altered to preserve the anonymity of our client.
Friday 18.00, H (Hour) + 0
The incident handler on call receives a heads-up from the account management team that a potentially large incident is incoming. 15 minutes later, after rushing home, the incident handler receives a call from the client on the emergency response hotline. The client was notified by law enforcement that they had a security issue. After performing an initial triage by phone, it was confirmed we were dealing with a significant incident. We agree with the client on an initial budget and establish an investigation team. Simultaneously, the client uploads the first pieces of investigation data to the Fox-IT forensic lab.
Friday 21:00. H + 3
A kickoff with our team (an incident handler and two analysts) and the team of the client takes place. Our team provides initial feedback on the uploaded data and presents a strategy on how to approach the incident. The approach is to get EDR (Enterprise Detection & Response) coverage on all systems for live monitoring, and to perform data acquisition on several systems in order to analyze historical events. We use Carbon Black as our preferred EDR platform, and for data acquisition we use the in-house developed Acquire which is part of the Dissect framework. Within the space of incident response there is a debate about the pros and cons of taking containment measures immediately versus waiting first to get a better view of the environment. We tend to first try to get a better view of the environment if we’re unsure what kind of threat actor we’re dealing with.
Technical interlude
Our data acquisition technique boils down to “give us everything”. We prioritize the acquisition of critical systems (both in importance and likelihood of compromise), but beyond that we want to get as much host data as possible. This allows us to do analysis in bulk, instead of having a back-and-forth with the client each time we want to collect a new system. “Everything” is usually still a multi-step process though, and this time was no exception!
Friday 23:00. H + 5
First Carbon Black EDR agent online.
Saturday 01:30. H + 7.5
Our team goes to sleep, after ensuring that more Carbon Black EDR agents show up online and that the client team knows how to perform data acquisition and upload that data. We agree to provide the client with the next update on Saturday at 14.00 CET, which is early morning for the client. We agree to resume work around 10:00 CET in the morning to have some findings ready by then.
Saturday 09:30. H + 15.5
The investigation team starts a little bit earlier and, using Carbon Black EDR insights, discovers concerning live activity of the adversary and decides to wake up the client (in the middle of the night, their time) to prepare for a meeting in 45 minutes. While it generally feels positive when our approach works, telling a client that they likely need to take important and high impact decisions such as shutting down internet connections and thus operations, can feel uncomfortable. Nonetheless it is important we act based on facts and the evidence currently available to us.
Technical interlude
When rolling out the Carbon Black EDR agents we make sure to also deploy our custom watchlists to monitor the usage of certain tools. The following morning, we noticed that a watchlist had been triggered that monitors for the usage of rclone, a popular data exfiltration tool used by adversaries. What made this process even more suspicious was the fact that it was trying to hide behind the veeam.exe name, the name of a popular backup solution. Luckily, we're able to match on the actual product name in the binary which is telling us exactly what the actual name of the process should be, no matter how it is being renamed by the user.
The combination of watchlist queries like these with a platform like Carbon Black helps to make split second decisions.
Saturday 10:30. H + 16.5
We give a summary presentation to the client and discuss several scenarios based on business impact and effectiveness for containment. Decision by the client about the specific containment scenario to cut off internet access is made.
Saturday 14:00. H + 20
We have a “start of day” meeting with the client and discuss the approach for today. The goal is to do analysis and get a view on initial foothold, scope of compromise and a timeline.
Technical interlude
While investigation data uploads trickle in, we perform our standard processing steps. This consists of running almost all Dissect plugins and artefact parsers against the data and storing the result on disk, followed by sending the exported artefacts to Splunk, our preferred search platform.
For example, to export the Windows event logs in this way, you could do something like:
target-query /t/hostname.vmdk -f evtx > /h/hostname/evtx
rdump /h/hostname/evtx -w splunk://:1337
You can optionally use commands like xargs or tee to speed things up or parallelize the processing.
Besides using Splunk for things like IOC (Indicators of Compromise) checks and statistical analysis, some of our analysts prefer to use command line tools for a large part of their analysis. For this reason, we export to files and Splunk, so that analysts with different preferred workflows can efficiently work together on the case data. One analyst can consult the many dashboards we have created and create complex queries while the other can use a combination of Dissect (e.g. target-query, rdump) and simple text processing tools (e.g. (rip)grep, less).
Saturday 19:00. H + 25
We present the client with findings, including an initial timeline and scope of compromise. In the meantime discussions are ongoing on how to bring back some of the clients’ core processes in a safe way for the start of the business week.
Sunday 00:00. H + 30
We share additional findings with the client. Our team is going to sleep.
Sunday 10:00. H + 40
FoxCERT management is getting a team together that can provide on-site containment and remediation support for the client. Several calls, chats and mails take place with remediation, RM&G (Risk Management & Governance) and FoxCERT practices throughout the group. Later that morning there is confirmation that a containment lead and a pentester will fly to the customer’s premises the next day.
Sunday 14:00. H + 44
“Start of day” meeting including discussion of our approach for today. A new potential backdoor into the network is reported by the investigation team.
Sunday 19:00. H + 48
Findings meeting with the client. The timeline is expanding. While the meeting is ongoing the team finds evidence regarding the adversary and initial foothold. Decision is made to share the news later that night when there are additional details and observations made regarding the behavior of the attacker, also called an adversary profile.
Technical interlude
Using the different plugins available in the Dissect framework we can quickly triage different artefacts. The initial foothold of the actor was found by checking different sources like PowerShell logging (with the evtx plugin), scheduled tasks present on systems (with the tasks plugin), and an MFT (Master File Table) timeline (with the mft_timeline plugin). Dissect allows us to do this quickly and efficiently across a large number of data sources.
Sunday 23:00. H + 53
End of day meeting. We share information about the (medium confidence) adversary. The TTP’s (Tools, Techniques and Procedures) we observed match with content in several blogs. Information on potential goals and TTP of the adversary is used by the client and the investigation team to improve decision making on how to further mitigate the attack. It turns out that cutting the internet off was probably a very good idea and was taken at just the right time.
Monday 14:00. H + 68
“Start of day” meeting. Each member of the team worked at least 30 hours over the weekend. The team discusses the approach for today and agrees with the client on a bit shorter working day that day.
Monday 18:00. H + 72
We develop a more complete picture of the incident. The timeline of the attack will likely extend significantly (multiple months) into the past, and the initial intrusion vector is confirmed. Meanwhile, the on-site Fox containment team arrived on-site and started in their continuous support of the client in advising about mitigation and containment and even how to re-start some business processes, while doing this in a secure way. This secure way is devised by looking at the techniques used by the attacker which were found by the investigation team and making sure that this process is not sabotaged or disturbed by the attacker.
Wrapping up
Combining different artefact sources for triaging purposes is nothing new when we’re talking about incident response. With Dissect we were able to provide the client with a picture of the incident using the data sources of ~100 systems and only two host investigators within 72 hours. During the remainder of the incident this number reached around ~250 systems, but still only two host investigators were involved.
By combining the Dissect framework with Splunk we were able to create a complete timeline of the path the attacker took within the network, as well as all the pivots and toolsets that were used. With this information we could quickly and aptly answer the questions the client had regarding the scope of the compromise, as well as visually show this. Dissect and Splunk enable us to effectively analyze historical artefacts. The combination with an EDR agent such as Carbon Black gives us the ability to simultaneously monitor the environment in real-time. Both sides feeding each other with indicators and more insights into the environment and the attack.
We hope you enjoyed reading this short war story where we give you some insights in how we at Fox-IT leverage Dissect in our incident response practice.
We hope this story gives you some insights as to how Dissect can possibly help you in your own incident response process.