Table of contents

Introduction

An alert comes into the queue. Its name? “Possible Grandoreiro malware execution.” The alert details? From what you can decipher, it appears that Microsoft is unhappy with some signed executable at C:\ProgramData\AMD\ATISSDRIVER.exe making a network connection to dns.google.

If this is all the information you are presented with (as is too often the case), it can be easy to write this off as a false positive. Very rarely do alerts present themselves with enough context to make determinations on the validity of an alert. Instead, alerts usually act as pivot points to evidence sources where we are driven to answer investigative questions.

The answers to good investigative questions provide critical insight into the disposition, nature, and severity of an event. Of the ATISSDRIVER.exe executable, you might ask, “Where was this file downloaded from?” If the answer is a web domain registered less than seven days ago composed of random letters, you’ve all but confirmed the disposition as malicious. You may also then ask of the binary, “What did it do?” If we answer with a network connection to dns.google, a module load of secman.dll, and we review publicly-available reporting from X-Force on Grandoreiro, we can make the inference that it is likely using dns.google for command and control (C2) communication and may be harvesting information from Outlook. That answers the nature of what the binary is doing, and the severity of the threat is also clear: high.

Through the many security incidents (and nothingburgers) I’ve observed in my career, I’ve developed a strong understanding of what investigative questions an analyst must answer and how those puzzles are best solved. The remainder of this blog will discuss the top ten common investigative questions in security operations and where to look to go about answering them.

Top Ten Investigative Questions

1: Are There Any (Other) Alerts for This Observable?

The longer and more severe a cyber attack, the more likely alerts from something are going to fire. One of the nicer things about modern defense is that, usually, a single incorrect disposition determination on an alert will not directly result in an entire environment being consumed by ransomware. What becomes a major red flag, however, is if a single endpoint, identity, IP address, filename, etc. (an observable), has lots of recent alerts.

Ideally, your case management system brings together alerts from a wide variety of platforms and correctly correlates them together based on observables such that you never have the blindspot of not realizing how many alerts are firing for a single entity. The concept of “other alerts” is also what can turn a low-fidelity detection analytic (e.g., the invocation of nltest.exe /dclist) into a high-fidelity one (invocation of nltest.exe /dclist + another alert), though this meta-correlation can be difficult to accomplish.

To answer this investigative question, review all your security platforms, including but not limited to:

• EDR
• NIDS
• Security Information and Event Management platform (SIEM)
• Identity Threat Detection & Response platform (ITDR)

Example Alerts & Incidents Page for a Device in Microsoft Defender for Endpoint

2: Is There Any Intel Available for This Observable?

Another quick and easy win (that is ideally automated) is to check observables against known intelligence sources. This applies to both threat intelligence (e.g., malicious IoC lists) and friendly intelligence (e.g., CMDB for endpoints, user directories for users).

The purpose of threat intel in this context is to either condemn something as malicious (e.g., due to a binary having a malicious reputation) or to absolve it of wrongdoing (e.g., a binary has a benign reputation (but is not a LOLBIN and is not performing otherwise malicious activity)). The purpose of friendly intel in this context is to enrich your alerting with pertinent information. For example, extracting user information details like name and job title can help identify if behavior is unexpected. Consider the difference between the uninteresting event of Alice, a known network administrator, running Angry IP Scanner, and the worrisome nature of Bob, a financial analyst, doing the same thing.

There’s an endless pool of intel sources in the world. Note that you generally should not upload samples to any third-party public intel services provider unless you are comfortable with them being actually or effectively public. To list just a few of the heavy hitters:

• https://www.virustotal.com - Great for file/binary reputation
• https://www.hybrid-analysis.com - An online dynamic malware analysis sandbox
• https://zulu.zscaler.com - Dynamic risk scoring engine for web pages
• https://www.abuseipdb.com - My favorite IP reputation service
• Active Directory - For user identity information
• Configuration Management Database (CMDB) including device ownership information

Pro tip: Enrich your SIEM with your threat and friendly intelligence sources. This can be a great way to provide valuable context in-line with alerting.

3: What’s in the Command Line?

If I were to build a decision tree for determining if a process-related event is evil or not, I would begin the root of that tree with analyzing the command line. Not all evil command line looks suspicious (e.g., file.exe --startup), but when an evil command line looks suspicious, it often looks very suspicious. Pay particularly close attention to anomalously mixed casing, inconsistent usage of separator characters (e.g., many spaces), and obfuscated or encoded commands.

PowERshEll.exe   -enc  "aQBlAHgAIAAiAHcAcgBpAHQAZQAtAGgAbwBzAHQAIAAnAGgAaQAgAGIAbwBiACcAIgA=

You can find process command line in the following evidence sources:

• EDR process execution telemetry
• Sysmon event ID 1
• Windows Security event ID 4688
• PowerShell transcript log files
• Linux .history files (e.g., .bash_history)

4: What’s in the Process Tree?

Process trees can be deeply revealing. In the olden days (when these attacks were more common), one of the more obvious ways of identifying evil was if an Office program (e.g., Excel, Word) spawned a command processor like Command Prompt or PowerShell. Unexpected parent/child relationships are still a strong indicator of malicious activity. Many command and control frameworks spawn a legitimate, sacrificial process in a suspended state, populate its memory with malicious content, and then resume the process such that the injected evil instructions are carried out. In these cases, a process creation log may not feature any anomalies for threat identification, save for one possibility: an unexpected parent process. Rather notoriously, for example, the default sacrificial process spawned by Cobalt Strike is rundll32.exe with no command line arguments. This — the execution of rundll32.exe with no command line from an uncommon parent process (e.g., not svchost.exe) — is a fairly strong detection analytic, and the parent process that spawns such an instance of rundll32.exe should be treated as highly suspect.

You can find parent/child process lineage information in the same evidence sources as command line and general process execution, though it can be more difficult to stitch together native event logs to tell a story. EDR platforms are the ideal evidence source to review for this. To identify what is and is not an unexpected parent/child process relationship, it’s best to baseline these with queries in your EDR or SIEM platform. EchoTrail is another excellent place to begin your Windows process anomaly identification journey.

Example Process Tree from Carbon Black, pulled from Red Canary’s Blog

5: What Network Connections Were Made?

Most flavors of evil include some form of external network-based command and control. Lateral movement to internal endpoints is also common. One of the very first things I do when hunting for threats on an endpoint that has just received numerous alerts is to look for recent connections to uncommon domains. How to go about this depends on the tooling you are using, but, broadly speaking, I will simply look at Internet-based network connection logs minus the entries of the domains in Cisco Umbrella’s top 1 million domain list. This method catches a lot of evil, and can even catch abuse of trusted sites, as their abuse often involves the use of unique subdomains that will not be in the top 1 million domain list.

The best sources to identify network connections are, in order:

• Web proxy logs
• Sysmon event ID 3 (network connection)
• EDR network telemetry (less reliable, but more convenient to review)
• Firewall traffic logs
• NIDS alert logs

6: How Prevalent is This Observable?

When reviewing an odd observable (e.g., a strange network connection, a binary in a mysterious directory), one of my favorite questions to ask is “Is this common?” In different contexts, commonality (or prevalence) can be a strong indicator that something is benign or that there is widespread evil afoot.

Consider an investigation involving a legitimate but frequently abused remote access tool like ScreenConnect. If you know that ScreenConnect is prevalent in an environment, you can safely conclude that its presence does not necessarily constitute a threat. Conversely, if ScreenConnect is not prevalent in an environment, and is indeed only to be found on a singular endpoint (that you happen to be investigating), that constitutes a probable very serious cause for concern. Assuming that the presence of something like ScreenConnect is common and expected, you can still identify abuse by reviewing the prevalence of another observable: network connection domains. If you aggregate all the connections ending with screenconnect.com in an environment, and find that almost all are for the subdomain companyname.screenconnect.com, save for one anomaly in the form of fisa99.screenconnect.com, you’ve likely identified an adversary!

Prevalence calculations are best made from a SIEM or other log aggregation platform. Excellent evidence sources include:

• EDR telemetry of all flavors
• Sysmon & native Windows event logs
• Web proxy logs
• Firewall traffic logs
• Authentication logs (for the prevalence of IP addresses, ISPs, etc.)

7: Is This Binary Signed?

In the olden days, good software was (maybe) signed, and evil software was never signed. Today, unfortunately, a lot of malicious software is digitally signed. This is due to a rise in the use of stolen code-signing certificates and issues in the quality control process of digitally signed software. Still, it’s good to check. In fact, a digital signature, in the right context, can be an indicator that a file is malicious.

Consider, for example, a file called Adobe-Update-ClientSetup.exe that is signed by ConnectWise (the company behind the ScreenConnect remote management tool). This mismatch between the file name and the digital signer is a nearly perfect indicator that a binary is malicious.

The best place to identify if a file is signed (other than by direct examination) is through EDR telemetry. Many EDR providers provide signing information for executables in one place or another, though it is not always easy to access. For example, Microsoft Defender for Endpoint’s DeviceProcessEvents table does not include signing information by default, though process execution logs can be enriched to include it via the FileProfile() function. If you don’t have direct access to a file, but the file has been uploaded to VirusTotal (don’t upload anything to VirusTotal yourself that you wouldn’t be comfortable publicly uploading!), you can find the signing status on the “Details” tab.

Modern Digital Signatures

8: What Is This Email? Who Is This Sender?

The phishing investigations of today are slightly more nuanced than they used to be. While adversaries still send emails related to Ukrainian Brides from domains associated with Zimbabwe, due to modern email security solutions and spam filtering, not nearly as many of them make their way in front of a human. The greater and more difficult challenge of modern email security is when adversaries pop legitimate business email accounts and use them as a vehicle to spread their infestation. Email threat identification is becoming more and more like modern identity threat identification: anomaly-based.

In these cases, it becomes useful to be able to identify a baseline and history associated with an email sender, a sender domain, and/or an email subject line. Particularly in cases where you are unable to view direct email contents to protect users due to privacy standards or due to issues with RBAC, reviewing email can be your best option in identifying a malicious email.

To answer this investigative question, review your email send/receive logs for:

• Emails with similar subject lines (adversaries sometimes send similar, though slightly different, phishing emails from multiple email addresses)
• Emails containing URLs for the same unique domain as the suspect email
• Emails from or to the same sending domain (e.g., contoso.com)
• Emails in the same email chain (follow a trail back to its origin by following replies to the same subject line)

The Modern Phish

9: Is This Authentication Anomalous?

The basis for modern identity threat detection is, at its core, anomaly identification. What is completely normal and benign for one user may be completely abnormal and indicative of a threat for another. Bob uses NordVPN. So does LAPSUS$. If a user authenticates in a way that is irregular for the user, you probably need to ask, “was this you?”

To identify if an authentication is anomalous, look through your existing authentication logs (e.g., Entra, Okta) and review the following aggregations & statistics:

• Authentications by ISP (if available)
• Authentications by IP address
• Authentications by user agent
• Authentications by whether the authenticating device is managed, registered, or compliant with mobile device management (MDM).
• Most recent observed timestamps for the above properties
• Earliest observed timestamps for the above properties
• Trends in the above properties over time (e.g., count of ISP per 24-hour period for last 30 days)

To identify VPN usage for a given IP address, my favorite tool is Spur. Simply go to spur.us/context/<IP address> to identify if an IP address is associated with a VPN and what VPN that might be.

10: Was This You?

The unfortunate truth is that sometimes, the difference between an anomaly and the initial actions of an attack is intent. This drives us to prompt users with the most frequent and painful of questions in security operations: “Was this you?”

I wrote an entire blog post on the wasted time, pain, and suffering that this question causes. But, it is unavoidable. The TL;DR of that blog post is that you must ask users if they did indeed do a thing (“Did you just log in using ExpressVPN?”), but you should endeavor to do so in an automated fashion, ideally using a trusted, out-of-band means of communication to avoid the possibility of an adversary replying, “Yes, that was me!”