Validate your defenses with Atomic Test Harnesses for Linux and macOS

The POSIX Atomic Test Harnesses suite leverages Python to emulate multiple variations of a given ATT&CK technique on Linux and macOS systems.

What are Atomic Test Harnesses? 

Atomic Test Harnesses provide a way to understand in-depth coverage of a given MITRE ATT&CK® technique. In the most simple terms, Atomic Test Harnesses streamline the execution of attack technique variations and validate that the expected telemetry surfaces in the process. It has been a valuable tool in determining whether a technique is being detected correctly.

Originally, AtomicTestHarness was a PowerShell module designed only for Windows. We are excited to announce AtomicTestHarness support for both macOS and Linux. Leveraging Python instead of PowerShell, the POSIX suite can be installed using pip or pipenv.

How do they differ from Atomic Red Team tests?

Atomic Red Team tests focus on highlighting the end behaviors (procedural examples) of ATT&CK techniques. By contrast, Atomic Test Harnesses provide a comprehensive programmatic implementation of an ATT&CK technique agnostic of adversarial behaviors. In other words, regular atomics exercise single, atomic behavior, whereas test harnesses seek to demonstrate all the different ways an adversary can execute a given technique.

Why do we need them for macOS and Linux?

Over time, we saw the benefit of Atomic Test Harnesses on Windows. We were able to provide a much deeper analysis of a given ATT&CK technique and gained insight into our detection abilities. In the past few years, we have seen a significant rise in the amount of malware targeted towards macOS and Linux. We felt a strong need to bring Atomic Test Harnesses to those platforms in order to stay ahead of the emerging threats that we are seeing.

How do the POSIX Atomic Test Harnesses work?

When we were designing the macOS and Linux versions of Atomic Test Harnesses we wanted to use a language that was accessible to as many people as possible that could also work across multiple platforms. We also wanted to use a language that was easy to understand and modify for ease of user customization. It was for these reasons that we chose Python. Similar to how the Windows Atomic Test Harnesses use Pester to orchestrate tests and verify correct execution of the test, we used pytest as a test runner.

Let’s look at an example of how one would install and run a test using the POSIX version of Atomic Test Harness. The easiest way to get up and running is to install the posixath Python package with either:

pip install posixath

or

pipenv install posixath

Once the package is installed, users can run python -m posxiath –help to see the list of available commands as well as examples of how to run the tests. For more extensive documentation, there is a docs folder in the GitHub repo that houses a set of Markdown files for each attack technique. These can be viewed locally by following the instructions in the README.

To run a test there are two fields that are required: the platform (i.e., macOS or Linux) and the attack technique to run. Below is an example run of T1018: Remote System Discovery:

python -m posixath linux -t t1018

Test walkthrough

ATT&CK technique coverage assessments can be confusing because every environment is unique. For example, two ATT&CK techniques frequently leveraged by adversaries in the wild targeting macOS have been AppleScript and JavaScript for Automation (JXA). A test might be able to detect an adversary executing OSA code at the command line (via the /usr/bin/osascript platform binary), but might not detect programmatic execution via the OSAKit framework.

This has traditionally been a black box for detection. However, with a combination of Atomic Test Harnesses and Endpoint Security Framework (ESF) data collection tools, we can gain great insight into the raw telemetry exposed by ESF and freely available to use by endpoint detection and response (EDR) vendors. We will walk through a case study where we leverage the T1059.002: AppleScript and T1059.007: JavaScript Atomic Test Harnesses in combination with our research ESF sensor to pull back the curtain and observe what’s going on behind the scenes.

1. Install the POSIX AtomicTestHarness suite a. Install Python on macOS. To do this we recommend using pyenv. b. Install the POSIX AtomicTestHarness suite: python -m pip install posixath
2. Before running the AtomicTestHarnesses, start a system monitor of some kind. We will monitor ESF directly, but additional options include: a. /usr/bin/eslogger for an ESF logging platform binary. (macOS 13 Ventura and greater) b. Activity Monitor.app for generic high level system activity c. Console.app for the unified system log (with private logging enabled) d. Osquery: Free endpoint behavioral monitoring e. Utilities from The Mitten Mac and Objective-See.
3.  Run the tests for AppleScript and JXA: a. python -m posixath macos -t T1059_002 b. python -m posixath macos -t T1059_007
4. Stop the recording of system events or start hunting using the the tool chosen in step #2.

Brandon Dalton demonstrated these tests and more in our recent Detection Series webinar:

Interpretation of these results is dependent on the EDR product chosen. While most EDR products targeting macOS leverage ESF, there isn’t feature parity among all vendors. Using the data below, keep in mind whether your endpoint security solution can see the same data. If data is missing, you’ve found a gap in applied detections.

Test case	Detection with ESF	Event distribution
T1059.002/7 (NSAppleScript / OSAKit)	ES_EVENT_TYPE_NOTIFY_EXEC Of a binary with an adhoc signature? ES_EVENT_TYPE_NOTIFY_MMAP Mapping OSA components into memory? /System/Library/Components/AppleScript.component /System/Library/Components/JavaScript.component /System/Library/ScriptingAdditions/StandardAdditions.osax
T1059.002/7 (command line)	ES_EVENT_TYPE_NOTIFY_EXEC Ample command-line detection opportunities ES_EVENT_TYPE_NOTIFY_MMAP Mapping /usr/bin/osascript into memory Mapping OSA components into memory? /System/Library/Components/AppleScript.component /System/Library/Components/JavaScript.component /System/Library/ScriptingAdditions/StandardAdditions.osax
T1059.002/7 (shell script)	ES_EVENT_TYPE_NOTIFY_EXEC Ample command-line detection opportunities. ES_EVENT_TYPE_NOTIFY_MMAP Mapping /usr/bin/osascript into memory Mapping OSA components into memory? /System/Library/Components/AppleScript.component /System/Library/Components/JavaScript.component /System/Library/ScriptingAdditions/StandardAdditions.osax
T1059.002/7 (applet)	ES_EVENT_TYPE_NOTIFY_EXEC Of a binary with an adhoc signature? ES_EVENT_TYPE_NOTIFY_MMAP Mapping OSA components into memory? /System/Library/Components/AppleScript.component /System/Library/Components/JavaScript.component /System/Library/ScriptingAdditions/StandardAdditions.osax
T1059.002/7 (stay-open-script)	ES_EVENT_TYPE_NOTIFY_EXEC Of a binary with an adhoc signature? ES_EVENT_TYPE_NOTIFY_EXEC com.apple.plutil inserting the stay-open plist key: OSAAppletStayOpen? `ES_EVENT_TYPE_NOTIFY_MMAP` Mapping OSA components into memory? /System/Library/Components/AppleScript.component /System/Library/Components/JavaScript.component /System/Library/ScriptingAdditions/StandardAdditions.osax

KEEP UP WITH ALL THINGS ATOMIC

Get monthly updates on Atomic Tests Harnesses and other open source security tools by subscribing to the Atomic Newsletter!