SUMMARY

The Federal Bureau of Investigation (FBI) and Cybersecurity and Infrastructure Security Agency (CISA) are releasing this joint Cybersecurity Advisory (CSA) in response to the active exploitation of CVE-2023-27350. This vulnerability occurs in certain versions of PaperCut NG and PaperCut MF and enables an unauthenticated actor to execute malicious code remotely without credentials. PaperCut released a patch in March 2023.

According to FBI observed information, malicious actors exploited CVE-2023-27350 beginning in mid-April 2023 and continuing through the present. In early May 2023, also according to FBI information, a group self-identifying as the Bl00dy Ransomware Gang attempted to exploit vulnerable PaperCut servers against the Education Facilities Subsector.

This joint advisory provides detection methods for exploitation of CVE-2023-27350 as well and indicators of compromise (IOCs) associated with Bl00dy Ransomware Gang activity. FBI and CISA strongly encourage users and administrators to immediately apply patches, and workarounds if unable to patch. FBI and CISA especially encourage organizations who did not patch immediately to assume compromise and hunt for malicious activity using the detection signatures in this CSA. If potential compromise is detected, organizations should apply the incident response recommendations included in this CSA.

Download the PDF version of this report:

TECHNICAL DETAILS

Vulnerability Overview

CVE-2023-27350 allows a remote actor to bypass authentication and conduct remote code execution on the following affected installations of PaperCut:[1]

• Version 8.0.0 to 19.2.7
• Version 20.0.0 to 20.1.6
• Version 21.0.0 to 21.2.10
• Version 22.0.0 to 22.0.8

PaperCut servers vulnerable to CVE-2023-27350 implement improper access controls in the SetupCompleted Java class, allowing malicious actors to bypass user authentication and access the server as an administrator. After accessing the server, actors can leverage existing PaperCut software features for remote code execution (RCE). There are currently two publicly known proofs of concept for achieving RCE in vulnerable PaperCut software:

• Using the print scripting interface to execute shell commands.
• Using the User/Group Sync interface to execute a living-off-the-land-style attack.

FBI and CISA note that actors may develop other methods for RCE.

The PaperCut server process pc-app.exe runs with SYSTEM- or root-level privileges. When the software is exploited to execute other processes such as cmd.exe or powershell.exe, these child processes are created with the same privileges. Commands supplied with the execution of these processes will also run with the same privileges. As a result, a wide range of post-exploitation activity is possible following initial access and compromise.

This CVE was added to CISA’s Known Exploited Vulnerabilities (KEV) Catalog on April 21, 2023.

Threat Actor Activity

Education Facilities Subsector entities maintained approximately 68% of exposed, but not necessarily vulnerable, U.S.-based PaperCut servers. In early May 2023, according to FBI information, the Bl00dy Ransomware Gang gained access to victim networks across the Education Facilities Subsector where PaperCut servers vulnerable to CVE-2023-27350 were exposed to the internet. Ultimately, some of these operations led to data exfiltration and encryption of victim systems. The Bl00dy Ransomware Gang left ransom notes on victim systems demanding payment in exchange for decryption of encrypted files (see Figure 1).

According to FBI information, legitimate remote management and maintenance (RMM) software was downloaded and executed on victim systems via commands issued through PaperCut’s print scripting interface. External network communications through Tor and/or other proxies from inside victim networks helped Bl00dy Gang ransomware actors mask their malicious network traffic. The FBI also identified information relating to the download and execution of command and control (C2) malware such as DiceLoader, TrueBot, and Cobalt Strike Beacons, although it is unclear at which stage in the attack these tools were executed.

DETECTION METHODS

Network defenders should focus detection efforts on three key areas:

• Network traffic signatures – Look for network traffic attempting to access the SetupCompleted page of an exposed and vulnerable PaperCut server.
• System monitoring – Look for child processes spawned from a PaperCut server’s pc-app.exe process.
• Server settings and log files – Look for evidence of malicious activity in PaperCut server settings and log files.

Network Traffic Signatures

To exploit CVE-2023-27350, a malicious actor must first visit the SetupCompleted page of the intended target, which will provide the adversary with authentication to the targeted PaperCut server. Deploy the following Emerging Threat Suricata signatures to detect when GET requests are sent to the SetupCompleted page. (Be careful of improperly formatted double-quotation marks if copying and pasting signatures from this advisory.)

Note that some of the techniques identified in this section can affect the availability or stability of a system. Defenders should follow organizational policies and incident response best practices to minimize the risk to operations while threat hunting. 

alert http any any -> $HOME_NET any (
  msg:"ET EXPLOIT PaperCut MF/NG SetupCompleted Authentication Bypass (CVE-2023-27350)";
  flow:established,to_server;
  http.method; content:"GET";
  http.uri; content:"/app?service=page/SetupCompleted"; bsize:32; fast_pattern;
  reference:cve,2023-27350;
  classtype:attempted-admin;

alert http any any -> $HOME_NET any (msg:"ET EXPLOIT PaperCut MF/NG SetupCompleted Authentication Bypass (CVE-2023-27350)"; flow:established,to_server; http.method; content:"GET"; http.uri; content:"page/SetupCompleted"; fast_pattern; reference:url,www.huntress.com/blog/critical-vulnerabilities-in-papercut-print-management-software; reference:cve,2023-27350; classtype:attempted-admin; metadata:attack_target Server, cve CVE_2023_27350, deployment Perimeter, deployment Internal, deployment SSLDecrypt, former_category EXPLOIT, performance_impact Low, confidence High, signature_severity Major, updated_at 2023_05_05;)

Note that these signatures and other rule-based detections, including YARA rules, may fail to detect more advanced iterations of CVE-2023-27350 exploits. Actors are known to adapt exploits to circumvent rule-based detections formulated for the original iterations of exploits observed in the wild. For example, the first rule above detected some of the first known exploits of CVE-2023-27350, but a slight modification of the exploit’s GET request can evade that rule. The second rule was designed to detect a broader range of activity than the first rule.

The following additional Emerging Threat Suricata signatures are designed to detect Domain Name System (DNS) lookups of known malicious domains associated with recent PaperCut exploitation:

alert dns $HOME_NET any -> any any (msg:"ET TROJAN Possible PaperCut MF/NG Post Exploitation Domain in DNS Lookup (windowcsupdates .com)"; dns_query; content:"windowcsupdates.com"; nocase; isdataat:!1,relative; pcre:"/(?:^|.)windowcsupdates.com$/"; reference:url,www.huntress.com/blog/critical-vulnerabilities-in-papercut-print-management-software; classtype:trojan-activity; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, former_category MALWARE, performance_impact Low, signature_severity Major, updated_at 2023_04_21;)

alert dns $HOME_NET any -> any any (msg:"ET ATTACK_RESPONSE Possible PaperCut MF/NG Post Exploitation Domain in DNS Lookup (anydeskupdate .com)"; dns_query; content:"anydeskupdate.com"; nocase; isdataat:!1,relative; pcre:"/(?:^|.)anydeskupdate.com$/"; reference:url,www.huntress.com/blog/critical-vulnerabilities-in-papercut-print-management-software; classtype:trojan-activity; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, former_category MALWARE, performance_impact Low, signature_severity Major, updated_at 2023_04_21;)

alert dns $HOME_NET any -> any any (msg:"ET TROJAN Possible PaperCut MF/NG Post Exploitation Domain in DNS Lookup (anydeskupdates .com)"; dns_query; content:"anydeskupdates.com"; nocase; isdataat:!1,relative; pcre:"/(?:^|.)anydeskupdates.com$/"; reference:url,www.huntress.com/blog/critical-vulnerabilities-in-papercut-print-management-software; classtype:trojan-activity; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, former_category MALWARE, performance_impact Low, signature_severity Major, updated_at 2023_04_21;)

alert dns $HOME_NET any -> any any (msg:"ET TROJAN Possible PaperCut MF/NG Post Exploitation Domain in DNS Lookup (windowservicecemter .com)"; dns_query; content:"windowservicecemter.com"; nocase; isdataat:!1,relative; pcre:"/(?:^|.)windowservicecemter.com$/"; reference:url,www.huntress.com/blog/critical-vulnerabilities-in-papercut-print-management-software; classtype:trojan-activity; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, former_category MALWARE, performance_impact Low, signature_severity Major, updated_at 2023_04_21;)

alert dns $HOME_NET any -> any any (msg:"ET ATTACK_RESPONSE Possible PaperCut MF/NG Post Exploitation Domain in DNS Lookup (winserverupdates .com)"; dns_query; content:"winserverupdates.com"; nocase; isdataat:!1,relative; pcre:"/(?:^|.)winserverupdates.com$/"; reference:url,www.huntress.com/blog/critical-vulnerabilities-in-papercut-print-management-software; classtype:trojan-activity; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, former_category MALWARE, performance_impact Low, signature_severity Major, updated_at 2023_04_21;)

alert dns $HOME_NET any -> any any (msg:"ET TROJAN Possible PaperCut MF/NG Post Exploitation Domain in DNS Lookup (netviewremote .com)"; dns_query; content:"netviewremote.com"; nocase; isdataat:!1,relative; pcre:"/(?:^|.)netviewremote.com$/"; reference:url,www.huntress.com/blog/critical-vulnerabilities-in-papercut-print-management-software; classtype:trojan-activity; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, former_category MALWARE, performance_impact Low, signature_severity Major, updated_at 2023_04_21;)

alert dns $HOME_NET any -> any any (msg:"ET TROJAN Possible PaperCut MF/NG Post Exploitation Domain in DNS Lookup (updateservicecenter .com)"; dns_query; content:"updateservicecenter.com"; nocase; isdataat:!1,relative; pcre:"/(?:^|.)updateservicecenter.com$/"; reference:url,www.huntress.com/blog/critical-vulnerabilities-in-papercut-print-management-software; classtype:trojan-activity; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, former_category MALWARE, performance_impact Low, signature_severity Major, updated_at 2023_04_21;)

alert dns $HOME_NET any -> any any (msg:"ET TROJAN Possible PaperCut MF/NG Post Exploitation Domain in DNS Lookup (windowservicecenter .com)"; dns_query; content:"windowservicecenter.com"; nocase; isdataat:!1,relative; pcre:"/(?:^|.)windowservicecenter.com$/"; reference:url,www.huntress.com/blog/critical-vulnerabilities-in-papercut-print-management-software; classtype:trojan-activity; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, former_category MALWARE, performance_impact Low, signature_severity Major, updated_at 2023_04_21;)

alert dns $HOME_NET any -> any any (msg:"ET TROJAN Possible PaperCut MF/NG Post Exploitation Domain in DNS Lookup (windowservicecentar .com)"; dns_query; content:"windowservicecentar.com"; nocase; isdataat:!1,relative; pcre:"/(?:^|.)windowservicecentar.com$/"; reference:url,www.huntress.com/blog/critical-vulnerabilities-in-papercut-print-management-software; classtype:trojan-activity; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, former_category ATTACK_RESPONSE, performance_impact Low, signature_severity Major, updated_at 2023_04_21;)

Note that these signatures may also not work if the actor modified activity to evade detection by known rules.

System Monitoring

A child process is spawned under pc-app.exe when the vulnerable PaperCut software is used to execute another process, which is the PaperCut server process. Malicious activity against PaperCut servers in mid-April used the RCE to supply commands to a cmd.exe or powershell.exe child process, which were then used to conduct further network exploitation. The following YARA rule may detect malicious activity[2].

title: PaperCut MF/NG Vulnerability 
authors: Huntress DE&TH Team
description: Detects suspicious code execution from vulnerable PaperCut versions MF and NG 
logsource:
  category: process_creation 
  product: windows 
detection: 
  selection: 
    ParentImage|endswith: “pc-app.exe” 
    Image|endswith:  
      - “cmd.exe” 
      - “powershell.exe” 
  condition: selection 
level: high 
falsepositives:     
  - Expected admin activity

More advanced versions of the exploit can drop a backdoor executable, use living-off-the-land binaries, or attempt to evade the above YARA rule by spawning an additional child process in-between pc-app.exe and a command-line interpreter.

Server Settings and Log Files

Network defenders may be able to identify suspicious activity by reviewing the PaperCut server options to identify unfamiliar print scripts or User/Group Sync settings.

If the PaperCut Application Server logs have debug mode enabled, lines containing SetupCompleted at a time not correlating with the server installation or upgrade may be indicative of a compromise. Server logs can be found in [app-path]/server/logs/*.* where server.log is normally the most recent log file.
Any of the following server log entries may be indicative of a compromise:

• User "admin" updated the config key “print.script.sandboxed”
• User "admin" updated the config key “device.script.sandboxed”
• Admin user "admin" modified the print script on printer
• User/Group Sync settings changed by "admin"

Indicators of Compromise

See Table 1 through Table 6 for IOCs obtained from FBI investigations and open-source information as of early May 2023.

INCIDENT RESPONSE

If compromise is suspected or detected, organizations should:

1. Create a backup of the current PaperCut server(s).
2. Wipe the PaperCut Application Server and/or Site Server and rebuild it.
3. Restore the database from a “safe” backup point. Using a backup dated prior to April 2023 would be prudent, given that exploitation in-the-wild exploitation began around early April.
4. Execute additional security response procedures and carry out best practices around potential compromise.
5. Report the compromise to CISA via CISA’s 24/7 Operations Center (report@cisa.gov or 888-282-0870). The FBI encourages recipients of this document to report information concerning suspicious or criminal activity to their local FBI field office or IC3.gov. Regarding specific information that appears in this communication, the context and individual indicators, particularly those of a non-deterministic or ephemeral nature (such as filenames or IP addresses), may not be indicative of a compromise. Indicators should always be evaluated in light of an organization’s complete information security situation. 

MITIGATIONS

FBI and CISA recommend organizations:

• Upgrade PaperCut to the latest version.
• If unable to immediately patch, ensure vulnerable PaperCut servers are not accessible over the internet and implement one of the following network controls:
  • Option 1: External controls: Block all inbound traffic from external IP addresses to the web management portal (port 9191 and 9192 by default).
  • Option 2: Internal and external controls: Block all traffic inbound to the web management portal. Note: The server cannot be managed remotely after this step.
• Follow best cybersecurity practices in your production and enterprise environments, including mandating phishing-resistant multifactor authentication (MFA) for all staff and for all services. For additional best practices, see CISA’s Cross-Sector Cybersecurity Performance Goals (CPGs). The CPGs, developed by CISA and the National Institute of Standards and Technology (NIST), are a prioritized subset of IT and OT security practices that can meaningfully reduce the likelihood and impact of known cyber risks and common TTPs. Because the CPGs are a subset of best practices, CISA and FBI also recommend all organizations implement a comprehensive information security program based on a recognized framework, such as the NIST Cybersecurity Framework (CSF).

ACKNOWLEDGMENTS

The Multi-State Information Sharing and Analysis Center (MS-ISAC) contributed to this advisory.
REFERENCES
[1] PaperCut: URGENT | PaperCut MF/NG vulnerability bulletin (March 2023)
[2] Huntress: Critical Vulnerabilities in PaperCut Print Management Software

This product is provided subject to this Notification and this Privacy & Use policy.

SUMMARY

Note: this joint Cybersecurity Advisory (CSA) is part of an ongoing #StopRansomware effort to publish advisories for network defenders that detail ransomware variants and ransomware threat actors. These #StopRansomware advisories include recently and historically observed tactics, techniques, and procedures (TTPs) and indicators of compromise (IOCs) to help organizations protect against ransomware. Visit stopransomware.gov to see all #StopRansomware advisories and to learn more about other ransomware threats and no-cost resources.

Actions to take today to mitigate cyber threats from ransomware:

• Prioritize remediating known exploited vulnerabilities.
• Train users to recognize and report phishing attempts.
• Enable and enforce phishing- resistant multifactor authentication.

The Federal Bureau of Investigation (FBI), the Cybersecurity and Infrastructure Security Agency (CISA), and the Multi-State Information Sharing & Analysis Center (MS-ISAC) are releasing this joint CSA to disseminate known LockBit 3.0 ransomware IOCs and TTPs identified through FBI investigations as recently as March 2023.

The LockBit 3.0 ransomware operations function as a Ransomware-as-a-Service (RaaS) model and is a continuation of previous versions of the ransomware, LockBit 2.0, and LockBit. Since January 2020, LockBit has functioned as an affiliate-based ransomware variant; affiliates deploying the LockBit RaaS use many varying TTPs and attack a wide range of businesses and critical infrastructure organizations, which can make effective computer network defense and mitigation challenging.

The FBI, CISA, and the MS-ISAC encourage organizations to implement the recommendations in the mitigations section of this CSA to reduce the likelihood and impact of ransomware incidents.

Download the PDF version of this report: 

TECHNICAL DETAILS

Note: This advisory uses the MITRE ATT&CK® for Enterprise framework, version 12. See the MITRE ATT&CK Tactics and Techniques section for a table of the threat actors’ activity mapped to MITRE ATT&CK for Enterprise.

CAPABILITIES

LockBit 3.0, also known as “LockBit Black,” is more modular and evasive than its previous versions and shares similarities with Blackmatter and Blackcat ransomware.

LockBit 3.0 is configured upon compilation with many different options that determine the behavior of the ransomware. Upon the actual execution of the ransomware within a victim environment, various arguments can be supplied to further modify the behavior of the ransomware. For example, LockBit 3.0 accepts additional arguments for specific operations in lateral movement and rebooting into Safe Mode (see LockBit Command Line parameters under Indicators of Compromise). If a LockBit affiliate does not have access to passwordless LockBit 3.0 ransomware, then a password argument is mandatory during the execution of the ransomware. LockBit 3.0 affiliates failing to enter the correct password will be unable to execute the ransomware [T1480.001]. The password is a cryptographic key which decodes the LockBit 3.0 executable. By protecting the code in such a manner, LockBit 3.0 hinders malware detection and analysis with the code being unexecutable and unreadable in its encrypted form. Signature-based detections may fail to detect the LockBit 3.0 executable as the executable’s encrypted potion will vary based on the cryptographic key used for encryption while also generating a unique hash. When provided the correct password, LockBit 3.0 will decrypt the main component, continue to decrypt or decompress its code, and execute the ransomware.

LockBit 3.0 will only infect machines that do not have language settings matching a defined exclusion list. However, whether a system language is checked at runtime is determined by a configuration flag originally set at compilation time. Languages on the exclusion list include, but are not limited to, Romanian (Moldova), Arabic (Syria), and Tatar (Russia). If a language from the exclusion list is detected [T1614.001], LockBit 3.0 will stop execution without infecting the system.

INITIAL ACCESS

Affiliates deploying LockBit 3.0 ransomware gain initial access to victim networks via remote desktop protocol (RDP) exploitation [T1133], drive-by compromise [T1189], phishing campaigns [T1566], abuse of valid accounts [T1078], and exploitation of public-facing applications [T1190].

EXECUTION AND INFECTION PROCESS

During the malware routine, if privileges are not sufficient, LockBit 3.0 attempts to escalate to the required privileges [TA0004]. LockBit 3.0 performs functions such as:

• Enumerating system information such as hostname, host configuration, domain information, local drive configuration, remote shares, and mounted external storage devices [T1082]
• Terminating processes and services [T1489]
• Launching commands [TA0002]
• Enabling automatic logon for persistence and privilege escalation [T1547]
• Deleting log files, files in the recycle bin folder, and shadow copies residing on disk [T1485], [T1490]

LockBit 3.0 attempts to spread across a victim network by using a preconfigured list of credentials hardcoded at compilation time or a compromised local account with elevated privileges [T1078]. When compiled, LockBit 3.0 may also enable options for spreading via Group Policy Objects and PsExec using the Server Message Block (SMB) protocol. LockBit 3.0 attempts to encrypt [T1486] data saved to any local or remote device, but skips files associated with core system functions.

After files are encrypted, LockBit 3.0 drops a ransom note with the new filename .README.txt and changes the host’s wallpaper and icons to LockBit 3.0 branding [T1491.001]. If needed, LockBit 3.0 will send encrypted host and bot information to a command and control (C2) server [T1027].

Once completed, LockBit 3.0 may delete itself from the disk [T1070.004] as well as any Group Policy updates that were made, depending on which options were set at compilation time.

EXFILTRATION

LockBit 3.0 affiliates use Stealbit, a custom exfiltration tool used previously with LockBit 2.0 [TA0010]; rclone, an open-source command line cloud storage manager [T1567.002]; and publicly available file sharing services, such as MEGA [T1567.002], to exfiltrate sensitive company data files prior to encryption. While rclone and many publicly available file sharing services are primarily used for legitimate purposes, they can also be used by threat actors to aid in system compromise, network exploration, or data exfiltration. LockBit 3.0 affiliates often use other publicly available file sharing services to exfiltrate data as well [T1567] (see Table 1).

LEVERAGING FREEWARE AND OPEN-SOURCE TOOLS

LockBit affiliates have been observed using various freeware and open-source tools during their intrusions. These tools are used for a range of activities such as network reconnaissance, remote access and tunneling, credential dumping, and file exfiltration. Use of PowerShell and Batch scripts
are observed across most intrusions, which focus on system discovery, reconnaissance, password/credential hunting, and privilege escalation. Artifacts of professional penetration-testing tools such as Metasploit and Cobalt Strike have also been observed. See Table 2 for a list of legitimate freeware and open-source tools LockBit affiliates have repurposed for ransomware operations:

Indicators of Compromise (IOCs)

The IOCs and malware characteristics outlined below were derived from field analysis. The following samples are current as of March 2023.

LockBit 3.0 Black Icon

LockBit 3.0 Wallpaper

LockBit Command Line Parameters

-del

-gdel

-gspd

-pass (32 character value)

-path (File or path)

-psex

-safe

-wall

Mutual Exclusion Object (Mutex) Created

When executed, LockBit 3.0 will create the mutex, Global,
and check to see if this mutex has already been created to avoid running more than one instance of the ransomware.

UAC Bypass via Elevated COM Interface

LockBit 3.0 is capable of bypassing User Account Control (UAC) to execute code with elevated privileges via elevated Component Object Model (COM) Interface. C:WindowsSystem32dllhost.exe is spawned with high integrity with the command line GUID 3E5FC7F9-9A51-4367-9063-A120244FBEC.

For example, %SYSTEM32%dllhost.exe/Processid:{3E5FC7F9-9A51-4367-9063- A120244FBEC7}.

Volume Shadow Copy Deletion

LockBit 3.0 uses Windows Management Instrumentation (WMI) to identify and delete Volume Shadow Copies. LockBit 3.0 uses select * from Win32_ShadowCopy to query for Volume Shadow copies, Win32_ShadowCopy.ID to obtain the ID of the shadow copy, and DeleteInstance to delete any shadow copies.

Registry Artifacts

LockBit 3.0 Icon

HKCR. 

(Default)

HKCRDefaultIcon

(Default)

C:ProgramData.ico

LockBit 3.0 Wallpaper

HKCUControl PanelDesktopWallPaper

(Default)

C:ProgramData.bmp

Disable Privacy Settings Experience

SOFTWAREPoliciesMicrosoftWin
dowsOOBE

DisablePrivacyE
xperience

Enable Automatic Logon

SOFTWAREMicrosoftWindows
NTCurrentVersionWinlogon

AutoAdminLogon

1

DefaultUserName

DefaultDomainNa
me

DefaultPassword

Disable and Clear Windows Event Logs

HKLMSOFTWAREMicrosoftWindows
CurrentVersionWINEVTChannels
*

Enabled

0

HKLMSOFTWAREMicrosoftWindows
CurrentVersionWINEVTChannels
* ChannelAccess

ChannelAccess

AO:BAG:SYD:(A;;0x1;;
;SY)(A;;0x5;;;BA)(A;
;0x1;;;LA)

Ransom Locations

ADMIN$Temp.exe

%SystemRoot%Temp.exe

sysvolscripts.exe (Domain Controller)

Safe Mode Launch Commands

LockBit 3.0 has a Safe Mode feature to circumvent endpoint antivirus and detection. Depending upon the host operating system, the following command is launched to reboot the system to Safe Mode with Networking:

bcdedit /set {current} safeboot network

bootcfg /raw /a /safeboot:network /id 1

bcdedit /deletevalue {current} safeboot

bootcfg /raw /fastdetect /id 1

Group Policy Artifacts

The following are Group Policy Extensible Markup Language (XML) files identified after a LockBit 3.0 infection:

Services.xml stops and disables services on the Active Directory (AD) hosts.

Registry.pol

The following registry configuration changes values for the Group Policy refresh time, disable SmartScreen, and disable Windows Defender.

HKLMSOFTWAREPoliciesMicrosoftWindow
sSystem

GroupPolicyRefresh
TimeDC

REG_D
WORD

1

HKLMSOFTWAREPoliciesMicrosoftWindow
sSystem

GroupPolicyRefresh
TimeOffsetDC

REG_D
WORD

1

HKLMSOFTWAREPoliciesMicrosoftWindow
sSystem

GroupPolicyRefresh
Time

REG_D
WORD

1

HKLMSOFTWAREPoliciesMicrosoftWindow
sSystem

GroupPolicyRefresh
TimeOffset

REG_D
WORD

1

HKLMSOFTWAREPoliciesMicrosoftWindow
sSystem

EnableSmartScreen

REG_D
WORD

0

HKLMSOFTWAREPoliciesMicrosoftWindow
sSystem

**del.ShellSmartSc
reenLevel

REG_S
Z

HKLMSOFTWAREPoliciesMicrosoftWindow
s Defender

DisableAntiSpyware

REG_D
WORD

1

HKLMSOFTWAREPoliciesMicrosoftWindow
s Defender

DisableRoutinelyTa
kingAction

REG_D
WORD

1

HKLMSOFTWAREPoliciesMicrosoftWindow
s DefenderReal-Time Protection

DisableRealtimeMon
itoring

REG_D
WORD

1

HKLMSOFTWAREPoliciesMicrosoftWindow
s DefenderReal-Time Protection

DisableBehaviorMon
itoring

REG_D
WORD

1

HKLMSOFTWAREPoliciesMicrosoftWindow
s DefenderSpynet

SubmitSamplesConse
nt

REG_D
WORD

2

HKLMSOFTWAREPoliciesMicrosoftWindow
s DefenderSpynet

SpynetReporting

REG_D
WORD

0

HKLMSOFTWAREPoliciesMicrosoftWindow
sFirewallDomainProfile

EnableFirewall

REG_D
WORD

0

HKLMSOFTWAREPoliciesMicrosoftWindow
sFirewallStandardProfile

EnableFirewall

REG_D
WORD

0

Force GPUpdate

Once new group policies are added, a PowerShell command using Group Policy update (GPUpdate) applies the new group policy changes to all computers on the AD domain.

Services Killed

Processes Killed

LockBit 3.0 Ransom Note

~~~ LockBit 3.0 the world’s fastest and most stable ransomware from 2019~~~
>>>>> Your data is stolen and encrypted.
If you don’t pay the ransom, the data will be published on our TOR darknet sites. Keep in mind that once your data appears on our leak site, it could be bought by your competitors at any second, so don’t hesitate for a long time. The sooner you pay the ransom, the sooner your company will be safe.

Network Connections

If configured, Lockbit 3.0 will send two HTTP POST requests to one of the C2servers. Information about the victim host and bot are encrypted with an Advanced Encryption Standard (AES) key and encoded in Base64.

Example of HTTP POST request
POST /?7F6Da=u5a0TdP0&Aojq=&NtN1W=OuoaovMvrVJSmPNaA5&fckp9=FCYyT6b7kdyeEXywS8I8 HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate, br Content-Type: text/plain
User-Agent: Safari/537.36 
Host: 
Connection: Keep-Alive LIWy=RJ51lB5GM&a4OuN=&LoSyE3=8SZ1hdlhzld4&DHnd99T=rTx9xGlInO6X0zWW&2D6=Bokz&T1guL=MtRZsFCRMKyBmfmqI& 6SF3g=JPDt9lfJIQ&wQadZP= Xni=AboZOXwUw&2rQnM4=94L&0b=ZfKv7c&NO1d=M2kJlyus&AgbDTb=xwSpba&8sr=EndL4n0HVZjxPR& m4ZhTTH=sBVnPY&xZDiygN=cU1pAwKEztU&=5q55aFIAfTVQWTEm&4sXwVWcyhy=l68FrIdBESIvfCkvYl
Example of information found in encrypted data
{
"bot_version":"X",
"bot_id":"X",
"bot_company":"X", "host_hostname":"X", "host_user":"X",
"host_os":"X",
"host_domain":"X",
"host_arch":"X",
"host_lang":"X", "disks_info":[
{
"disk_name":"X",
"disk_size":"XXXX", "free_size":"XXXXX"
}

User Agent Strings

MITRE ATT&CK TECHNIQUES

See Table 3 for all referenced threat actor tactics and techniques in this advisory. For assistance with mapping to the MITRE ATT&CK framework, see CISA’s Decider Tool and Best Practices for MITRE ATT&CK Mapping Guide.

MITIGATIONS

The FBI, CISA, and the MS-ISAC recommend organizations implement the mitigations below to improve your organization’s cybersecurity posture on the basis of LockBit 3.0’s activity. These mitigations align with the Cross-Sector Cybersecurity Performance Goals (CPGs) developed by CISA and the National Institute of Standards and Technology (NIST). The CPGs provide a minimum set of practices and protections that CISA and NIST recommend all organizations implement. CISA and NIST based the CPGs on existing cybersecurity frameworks and guidance to protect against the most common and impactful TTPs. Visit CISA’s Cross-Sector Cybersecurity Performance Goals for more information on the CPGs, including additional recommended baseline protections.

• Implement a recovery plan to maintain and retain multiple copies of sensitive or proprietary data and servers [CPG 7.3] in a physically separate, segmented, and secure location (e.g., hard drive, storage device, the cloud).
• Require all accounts with password logins (e.g., service account, admin accounts, and domain admin accounts) to comply with National Institute for Standards and Technology (NIST) standards for developing and managing password policies [CPG 3.4].
  • Use longer passwords consisting of at least 8 characters and no more than 64 characters in length [CPG 1.4]
  • Store passwords in hashed format using industry-recognized password managers
  • Add password user “salts” to shared login credentials
  • Avoid reusing passwords
  • Implement multiple failed login attempt account lockouts [CPG 1.1]
  • Disable password “hints”
  • Refrain from requiring password changes more frequently than once per year. Note: NIST guidance suggests favoring longer passwords instead of requiring regular and frequent password resets. Frequent password resets are more likely to result in users developing password “patterns” cyber criminals can easily decipher.
  • Require administrator credentials to install software
• Require phishing-resistant multifactor authentication [CPG 1.3] for all services to the extent possible, particularly for webmail, virtual private networks, and accounts that access critical systems.
• Keep all operating systems, software, and firmware up to date. Timely patching is one of the most efficient and cost-effective steps an organization can take to minimize its exposure to cybersecurity threats.
• Segment networks [CPG 8.1] to prevent the spread of ransomware. Network segmentation can help prevent the spread of ransomware by controlling traffic flows between—and access to—various subnetworks and by restricting adversary lateral movement.
• Identify, detect, and investigate abnormal activity and potential traversal of the indicated ransomware with a networking monitoring tool. To aid in detecting the ransomware, implement a tool that logs and reports all network traffic, including lateral movement activity on a network [CPG 5.1]. Endpoint detection and response (EDR) tools are particularly useful for detecting lateral connections as they have insight into common and uncommon network connections for each host.
• Install, regularly update, and enable real time detection for antivirus software on all hosts.
• Review domain controllers, servers, workstations, and active directories for new and/or unrecognized accounts.
• Audit user accounts with administrative privileges and configure access controls according to the principle of least privilege [CPG 1.5].
• Disable unused ports.
• Consider adding an email banner to emails [CPG 8.3] received from outside your organization.
• Disable hyperlinks in received emails.
• Implement time-based access for accounts set at the admin level and higher. For example, the Just-in-Time (JIT) access method provisions privileged access when needed and can support enforcement of the principle of least privilege (as well as the Zero Trust model). This is a process where a network-wide policy is set in place to automatically disable admin accounts at the Active Directory level when the account is not in direct need. Individual users may submit their requests through an automated process that grants them access to a specified system for a set timeframe when they need to support the completion of a certain task.
• Disable command-line and scripting activities and permissions. Privilege escalation and lateral movement often depend on software utilities running from the command line. If threat actors are not able to run these tools, they will have difficulty escalating privileges and/or moving laterally.
• Maintain offline backups of data, and regularly maintain backup and restoration [CPG 7.3]. By instituting this practice, the organization ensures they will not be severely interrupted, and/or only have irretrievable data.
• Ensure all backup data is encrypted, immutable (i.e., cannot be altered or deleted), and covers the entire organization’s data infrastructure [CPG 3.3].

VALIDATE SECURITY CONTROLS

In addition to applying mitigations, the FBI, CISA, and the MS-ISAC recommend exercising, testing, and validating your organization’s security program against the threat behaviors mapped to the MITRE ATT&CK for Enterprise framework in this advisory. The FBI, CISA, and the MS-ISAC authoring agencies recommend testing your existing security controls inventory to assess how they perform against the ATT&CK techniques described in this advisory.
To get started:

1. Select an ATT&CK technique described in this advisory (see Table 3).
2. Align your security technologies against the technique.
3. Test your technologies against the technique.
4. Analyze your detection and prevention technologies performance.
5. Repeat the process for all security technologies to obtain a set of comprehensive performance data.
6. Tune your security program, including people, processes, and technologies, based on the data generated by this process.

The FBI, CISA, and the MS-ISAC recommend continually testing your security program at scale and in a production environment to ensure optimal performance against the MITRE ATT&CK techniques identified in this advisory.

RESOURCES

• Stopransomware.gov is a whole-of-government approach that gives one central location for ransomware resources and alerts.
• Resource to mitigate a ransomware attack: CISA-Multi-State Information Sharing and Analysis Center (MS-ISAC) Joint Ransomware Guide.
• No-cost cyber hygiene services: Cyber Hygiene Services and Ransomware Readiness Assessment.

REPORTING

The FBI is seeking any information that can be legally shared, including:

• Boundary logs showing communication to and from foreign IP addresses
• Sample ransom note
• Communications with LockBit 3.0 actors
• Bitcoin wallet information
• Decryptor files
• Benign sample of an encrypted file

The FBI, CISA, and MS-ISAC do not encourage paying ransom, as payment does not guarantee victim files will be recovered. Furthermore, payment may also embolden adversaries to target additional organizations, encourage other criminal actors to engage in the distribution of ransomware, and/or fund illicit activities. Regardless of whether you or your organization have decided to pay the ransom, the FBI and CISA urge you to promptly report ransomware incidents to a local FBI Field Office or CISA at report@cisa.gov. State, local, tribal, and territorial (SLTT) government entities can also report to the MS-ISAC (SOC@cisecurity.org or 866-787-4722).

DISCLAIMER

The information in this report is being provided “as is” for informational purposes only. The FBI, CISA, and the MS-ISAC do not endorse any commercial product or service, including any subjects of analysis. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply endorsement, recommendation, or favoring by the FBI, CISA, or the MS-ISAC.

SUMMARY

The Cybersecurity and Infrastructure Security Agency (CISA) is releasing this Cybersecurity Advisory (CSA) detailing activity and key findings from a recent CISA red team assessment—in coordination with the assessed organization—to provide network defenders recommendations for improving their organization’s cyber posture.

Actions to take today to harden your local environment:

• Establish a security baseline of normal network activity; tune network and host-based appliances to detect anomalous behavior.
• Conduct regular assessments to ensure appropriate procedures are created and can be followed by security staff and end users.
• Enforce phishing-resistant MFA to the greatest extent possible.

In 2022, CISA conducted a red team assessment (RTA) at the request of a large critical infrastructure organization with multiple geographically separated sites. The team gained persistent access to the organization’s network, moved laterally across the organization’s multiple geographically separated sites, and eventually gained access to systems adjacent to the organization’s sensitive business systems (SBSs). Multifactor authentication (MFA) prompts prevented the team from achieving access to one SBS, and the team was unable to complete its viable plan to compromise a second SBSs within the assessment period.

Despite having a mature cyber posture, the organization did not detect the red team’s activity throughout the assessment, including when the team attempted to trigger a security response.

CISA is releasing this CSA detailing the red team’s tactics, techniques, and procedures (TTPs) and key findings to provide network defenders of critical infrastructure organizations proactive steps to reduce the threat of similar activity from malicious cyber actors. This CSA highlights the importance of collecting and monitoring logs for unusual activity as well as continuous testing and exercises to ensure your organization’s environment is not vulnerable to compromise, regardless of the maturity of its cyber posture.

CISA encourages critical infrastructure organizations to apply the recommendations in the Mitigations section of this CSA—including conduct regular testing within their security operations center—to ensure security processes and procedures are up to date, effective, and enable timely detection and mitigation of malicious activity.

Download the PDF version of this report:

TECHNICAL DETAILS

Note: This advisory uses the MITRE ATT&CK® for Enterprise framework, version 12. See the appendix for a table of the red team’s activity mapped to MITRE ATT&CK tactics and techniques.

Introduction

CISA has authority to, upon request, provide analyses, expertise, and other technical assistance to critical infrastructure owners and operators and provide operational and timely technical assistance to Federal and non-Federal entities with respect to cybersecurity risks. (See generally 6 U.S.C. §§ 652[c][5], 659[c][6].) After receiving a request for a red team assessment (RTA) from an organization and coordinating some high-level details of the engagement with certain personnel at the organization, CISA conducted the RTA over a three-month period in 2022.

During RTAs, a CISA red team emulates cyber threat actors to assess an organization’s cyber detection and response capabilities. During Phase I, the red team attempts to gain and maintain persistent access to an organization’s enterprise network while avoiding detection and evading defenses. During Phase II, the red team attempts to trigger a security response from the organization’s people, processes, or technology.

The “victim” for this assessment was a large organization with multiple geographically separated sites throughout the United States. For this assessment, the red team’s goal during Phase I was to gain access to certain sensitive business systems (SBSs).

Phase I: Red Team Cyber Threat Activity

Overview

The organization’s network was segmented with both logical and geographical boundaries. CISA’s red team gained initial access to two organization workstations at separate sites via spearphishing emails. After gaining access and leveraging Active Directory (AD) data, the team gained persistent access to a third host via spearphishing emails. From that host, the team moved laterally to a misconfigured server, from which they compromised the domain controller (DC). They then used forged credentials to move to multiple hosts across different sites in the environment and eventually gained root access to all workstations connected to the organization’s mobile device management (MDM) server. The team used this root access to move laterally to SBS-connected workstations. However, a multifactor authentication (MFA) prompt prevented the team from achieving access to one SBS, and Phase I ended before the team could implement a seemingly viable plan to achieve access to a second SBS.

Initial Access and Active Directory Discovery

The CISA red team gained initial access [TA0001] to two workstations at geographically separated sites (Site 1 and Site 2) via spearphishing emails. The team first conducted open-source research [TA0043] to identify potential targets for spearphishing. Specifically, the team looked for email addresses [T1589.002] as well as names [T1589.003] that could be used to derive email addresses based on the team’s identification of the email naming scheme. The red team sent tailored spearphishing emails to seven targets using commercially available email platforms [T1585.002]. The team used the logging and tracking features of one of the platforms to analyze the organization’s email filtering defenses and confirm the emails had reached the target’s inbox.

The team built a rapport with some targeted individuals through emails, eventually leading these individuals to accept a virtual meeting invite. The meeting invite took them to a red team-controlled domain [T1566.002] with a button, which, when clicked, downloaded a “malicious” ISO file [T1204]. After the download, another button appeared, which, when clicked, executed the file.

Two of the seven targets responded to the phishing attempt, giving the red team access to a workstation at Site 1 (Workstation 1) and a workstation at Site 2. On Workstation 1, the team leveraged a modified SharpHound collector, ldapsearch, and command-line tool, dsquery, to query and scrape AD information, including AD users [T1087.002], computers [T1018], groups [T1069.002], access control lists (ACLs), organizational units (OU), and group policy objects (GPOs) [T1615]. Note: SharpHound is a BloodHound collector, an open-source AD reconnaissance tool. Bloodhound has multiple collectors that assist with information querying.

There were 52 hosts in the AD that had Unconstrained Delegation enabled and a lastlogon timestamp within 30 days of the query. Hosts with Unconstrained Delegation enabled store Kerberos ticket-granting tickets (TGTs) of all users that have authenticated to that host. Many of these hosts, including a Site 1 SharePoint server, were Windows Server 2012R2. The default configuration of Windows Server 2012R2 allows unprivileged users to query group membership of local administrator groups.

The red team queried parsed Bloodhound data for members of the SharePoint admin group and identified several standard user accounts with administrative access. The team initiated a second spearphishing campaign, similar to the first, to target these users. One user triggered the red team’s payload, which led to installation of a persistent beacon on the user’s workstation (Workstation 2), giving the team persistent access to Workstation 2.

Lateral Movement, Credential Access, and Persistence

The red team moved laterally [TA0008] from Workstation 2 to the Site 1 SharePoint server and had SYSTEM level access to the Site 1 SharePoint server, which had Unconstrained Delegation enabled. They used this access to obtain the cached credentials of all logged-in users—including the New Technology Local Area Network Manager (NTLM) hash for the SharePoint server account. To obtain the credentials, the team took a snapshot of lsass.exe [T1003.001] with a tool called nanodump, exported the output, and processed the output offline with Mimikatz.

The team then exploited the Unconstrained Delegation misconfiguration to steal the DC’s TGT. They ran the DFSCoerce python script (DFSCoerce.py), which prompted DC authentication to the SharePoint server using the server’s NTLM hash. The team then deployed Rubeus to capture the incoming DC TGT [T1550.002], [T1557.001]. (DFSCoerce abuses Microsoft’s Distributed File System [MS-DFSNM] protocol to relay authentication against an arbitrary server.[1])

The team then used the TGT to harvest advanced encryption standard (AES)-256 hashes via DCSync [T1003.006] for the krbtgt account and several privileged accounts—including domain admins, workstation admins, and a system center configuration management (SCCM) service account (SCCM Account 1). The team used the krbtgt account hash throughout the rest of their assessment to perform golden ticket attacks [T1558.001] in which they forged legitimate TGTs. The team also used the asktgt command to impersonate accounts they had credentials for by requesting account TGTs [T1550.003].

The team first impersonated the SCCM Account 1 and moved laterally to a Site 1 SCCM distribution point (DP) server (SCCM Server 1) that had direct network access to Workstation 2. The team then moved from SCCM Server 1 to a central SCCM server (SCCM Server 2) at a third site (Site 3). Specifically, the team:

1. Queried the AD using Lightweight Directory Access Protocol (LDAP) for information about the network’s sites and subnets [T1016]. This query revealed all organization sites and subnets broken down by classless inter-domain routing (CIDR) subnet and description.
2. Used LDAP queries and domain name system (DNS) requests to identify recently active hosts.
3. Listed existing network connections [T1049] on SCCM Server 1, which revealed an active Server Message Block (SMB) connection from SCCM Server 2.
4. Attempted to move laterally to the SCCM Server 2 via AppDomain hijacking, but the HTTPS beacon failed to call back.
5. Attempted to move laterally with an SMB beacon [T1021.002], which was successful.

The team also moved from SCCM Server 1 to a Site 1 workstation (Workstation 3) that housed an active server administrator. The team impersonated an administrative service account via a golden ticket attack (from SCCM Server 1); the account had administrative privileges on Workstation 3. The user employed a KeePass password manager that the team was able to use to obtain passwords for other internal websites, a kernel-based virtual machine (KVM) server, virtual private network (VPN) endpoints, firewalls, and another KeePass database with credentials. The server administrator relied on a password manager, which stored credentials in a database file. The red team pulled the decryption key from memory using KeeThief and used it to unlock the database [T1555.005].

At the organization’s request, the red team confirmed that SCCM Server 2 provided access to the organization’s sites because firewall rules allowed SMB traffic to SCCM servers at all other sites.

The team moved laterally from SCCM Server 2 to an SCCM DP server at Site 5 and from the SCCM Server 1 to hosts at two other sites (Sites 4 and 6). The team installed persistent beacons at each of these sites. Site 5 was broken into a private and a public subnet and only DCs were able to cross that boundary. To move between the subnets, the team moved through DCs. Specifically, the team moved from the Site 5 SCCM DP server to a public DC; and then they moved from the public DC to the private DC. The team was then able to move from the private DC to workstations in the private subnet.

The team leveraged access available from SCCM 2 to move around the organization’s network for post-exploitation activities (See Post-Exploitation Activity section).

See Figure 1 for a timeline of the red team’s initial access and lateral movement showing key access points.

While traversing the network, the team varied their lateral movement techniques to evade detection and because the organization had non-uniform firewalls between the sites and within the sites (within the sites, firewalls were configured by subnet). The team’s primary methods to move between sites were AppDomainManager hijacking and dynamic-link library (DLL) hijacking [T1574.001]. In some instances, they used Windows Management Instrumentation (WMI) Event Subscriptions [T1546.003].

The team impersonated several accounts to evade detection while moving. When possible, the team remotely enumerated the local administrators group on target hosts to find a valid user account. This technique relies on anonymous SMB pipe binds [T1071], which are disabled by default starting with Windows Server 2016. In other cases, the team attempted to determine valid accounts based on group name and purpose. If the team had previously acquired the credentials, they used asktgt to impersonate the account. If the team did not have the credentials, they used the golden ticket attack to forge the account.

Post-Exploitation Activity: Gaining Access to SBSs

With persistent, deep access established across the organization’s networks and subnetworks, the red team began post-exploitation activities and attempted to access SBSs. Trusted agents of the organization tasked the team with gaining access to two specialized servers (SBS 1 and SBS 2). The team achieved root access to three SBS-adjacent workstations but was unable to move laterally to the SBS servers:

• Phase I ended before the team could implement a plan to move to SBS 1.
• An MFA prompt blocked the team from moving to SBS 2, and Phase I ended before they could implement potential workarounds.

However, the team assesses that by using Secure Shell (SSH) session socket files (see below), they could have accessed any hosts available to the users whose workstations were compromised.

Plan for Potential Access to SBS 1

Conducting open-source research [1591.001], the team identified that SBS 1 and 2 assets and associated management/upkeep staff were located at Sites 5 and 6, respectively. Adding previously collected AD data to this discovery, the team was able to identify a specific SBS 1 admin account. The team planned to use the organization’s mobile device management (MDM) software to move laterally to the SBS 1 administrator’s workstation and, from there, pivot to SBS 1 assets.

The team identified the organization’s MDM vendor using open-source and AD information [T1590.006] and moved laterally to an MDM distribution point server at Site 5 (MDM DP 1). This server contained backups of the MDM MySQL database on its D: drive in the Backup directory. The backups included the encryption key needed to decrypt any encrypted values, such as SSH passwords [T1552]. The database backup identified both the user of the SBS 1 administrator account (USER 2) and the user’s workstation (Workstation 4), which the MDM software remotely administered.

The team moved laterally to an MDM server (MDM 1) at Site 3, searched files on the server, and found plaintext credentials [T1552.001] to an application programming interface (API) user account stored in PowerShell scripts. The team attempted to leverage these credentials to browse to the web login page of the MDM vendor but were unable to do so because the website directed to an organization-controlled single-sign on (SSO) authentication page.

The team gained root access to workstations connected to MDM 1—specifically, the team accessed Workstation 4—by:

1. Selecting an MDM user from the plaintext credentials in PowerShell scripts on MDM 1.
2. While in the MDM MySQL database,
   • Elevating the selected MDM user’s account privileges to administrator privileges, and
   • Modifying the user’s account by adding Create Policy and Delete Policy permissions [T1098], [T1548].
3. Creating a policy via the MDM API [T1106], which instructed Workstation 4 to download and execute a payload to give the team interactive access as root to the workstation.
4. Verifying their interactive access.
5. Resetting permissions back to their original state by removing the policy via the MDM API and removing Create Policy and Delete Policy and administrator permissions and from the MDM user’s account.

While interacting with Workstation 4, the team found an open SSH socket file and a corresponding netstat connection to a host that the team identified as a bastion host from architecture documentation found on Workstation 4. The team planned to move from Workstation 4 to the bastion host to SBS 1. Note: A SSH socket file allows a user to open multiple SSH sessions through a single, already authenticated SSH connection without additional authentication.

The team could not take advantage of the open SSH socket. Instead, they searched through SBS 1 architecture diagrams and documentation on Workstation 4. They found a security operations (SecOps) network diagram detailing the network boundaries between Site 5 SecOps on-premises systems, Site 5 non-SecOps on-premises systems, and Site 5 SecOps cloud infrastructure. The documentation listed the SecOps cloud infrastructure IP ranges [T1580]. These “trusted” IP addresses were a public /16 subnet; the team was able to request a public IP in that range from the same cloud provider, and Workstation 4 made successful outbound SSH connections to this cloud infrastructure. The team intended to use that connection to reverse tunnel traffic back to the workstation and then access the bastion host via the open SSH socket file. However, Phase 1 ended before they were able to implement this plan.

Attempts to Access SBS 2

Conducting open-source research, the team identified an organizational branch [T1591] that likely had access to SBS 2. The team queried the AD to identify the branch’s users and administrators. The team gathered a list of potential accounts, from which they identified administrators, such as SYSTEMS ADMIN or DATA SYSTEMS ADMINISTRATOR, with technical roles. Using their access to the MDM MySQL database, the team queried potential targets to (1) determine the target’s last contact time with the MDM and (2) ensure any policy targeting the target’s workstation would run relatively quickly [T1596.005]. Using the same methodology as described by the steps in the Plan for Potential Access to SBS 1 section above, the team gained interactive root access to two Site 6 SBS 2-connected workstations: a software engineering workstation (Workstation 5) and a user administrator workstation (Workstation 6).

The Workstation 5 user had bash history files with what appeared to be SSH passwords mistyped into the bash prompt and saved in bash history [T1552.003]. The team then attempted to authenticate to SBS 2 using a similar tunnel setup as described in the Access to SBS 1 section above and the potential credentials from the user’s bash history file. However, this attempt was unsuccessful for unknown reasons.

On Workstation 6, the team found a .txt file containing plaintext credentials for the user. Using the pattern discovered in these credentials, the team was able to crack the user’s workstation account password [T1110.002]. The team also discovered potential passwords and SSH connection commands in the user’s bash history. Using a similar tunnel setup described above, the team attempted to log into SBS 2. However, a prompt for an MFA passcode blocked this attempt.

See figure 2 for a timeline of the team’s post exploitation activity that includes key points of access.

Command and Control

The team used third-party owned and operated infrastructure and services [T1583] throughout their assessment, including in certain cases for command and control (C2) [TA0011]. These included:

• Cobalt Strike and Merlin payloads for C2 throughout the assessment. Note: Merlin is a post-exploit tool that leverages HTTP protocols for C2 traffic.
  • The team maintained multiple Cobalt Strike servers hosted by a cloud vendor. They configured each server with a different domain and used the servers for communication with compromised hosts. These servers retained all assessment data.
• Two commercially available cloud-computing platforms.
  • The team used these platforms to create flexible and dynamic redirect servers to send traffic to the team’s Cobalt Strike servers [T1090.002]. Redirecting servers make it difficult for defenders to attribute assessment activities to the backend team servers. The redirectors used HTTPS reverse proxies to redirect C2 traffic between the target organization’s network and the Cobalt Strike team servers [T1071.002]. The team encrypted all data in transit [T1573] using encryption keys stored on team’s Cobalt Strike servers.
• A cloud service to rapidly change the IP address of the team’s redirecting servers in the event of detection and eradication.
• Content delivery network (CDN) services to further obfuscate some of the team’s C2 traffic.
  • This technique leverages CDNs associated with high-reputation domains so that the malicious traffic appears to be directed towards a reputation domain but is actually redirected to the red team-controlled Cobalt Strike servers.
  • The team used domain fronting [T1090.004] to disguise outbound traffic in order to diversify the domains with which the persistent beacons were communicating. This technique, which also leverages CDNs, allows the beacon to appear to connect to third-party domains, such as nytimes.com, when it is actually connecting to the team’s redirect server.

Phase II: Red Team Measurable Events Activity

The red team executed 13 measurable events designed to provoke a response from the people, processes, and technology defending the organization’s network. See Table 1 for a description of the events, the expected network defender activity, and the organization’s actual response.

Findings

Key Issues

The red team noted the following key issues relevant to the security of the organization’s network. These findings contributed to the team’s ability to gain persistent, undetected access across the organization’s sites. See the Mitigations section for recommendations on how to mitigate these issues.

• Insufficient host and network monitoring. Most of the red team’s Phase II actions failed to provoke a response from the people, processes, and technology defending the organization’s network. The organization failed to detect lateral movement, persistence, and C2 activity via their intrusion detection or prevention systems, endpoint protection platform, web proxy logs, and Windows event logs. Additionally, throughout Phase I, the team received no deconflictions or confirmation that the organization caught their activity. Below is a list of some of the higher risk activities conducted by the team that were opportunities for detection:
  • Phishing
  • Lateral movement reuse
  • Generation and use of the golden ticket
  • Anomalous LDAP traffic
  • Anomalous internal share enumeration
  • Unconstrained Delegation server compromise
  • DCSync
  • Anomalous account usage during lateral movement
  • Anomalous outbound network traffic
  • Anomalous outbound SSH connections to the team’s cloud servers from workstations
• Lack of monitoring on endpoint management systems. The team used the organization’s MDM system to gain root access to machines across the organization’s network without being detected. Endpoint management systems provide elevated access to thousands of hosts and should be treated as high value assets (HVAs) with additional restrictions and monitoring.
• KRBTGT never changed. The Site 1 krbtgt account password had not been updated for over a decade. The krbtgt account is a domain default account that acts as a service account for the key distribution center (KDC) service used to encrypt and sign all Kerberos tickets for the domain. Compromise of the krbtgt account could provide adversaries with the ability to sign their own TGTs, facilitating domain access years after the date of compromise. The red team was able to use the krbtgt account to forge TGTs for multiple accounts throughout Phase I.
• Excessive permissions to standard users. The team discovered several standard user accounts that have local administrator access to critical servers. This misconfiguration allowed the team to use the low-level access of a phished user to move laterally to an Unconstrained Delegation host and compromise the entire domain.
• Hosts with Unconstrained Delegation enabled unnecessarily. Hosts with Unconstrained Delegation enabled store the Kerberos TGTs of all users that authenticate to that host, enabling actors to steal service tickets or compromise krbtgt accounts and perform golden ticket or “silver ticket” attacks. The team performed an NTLM-relay attack to obtain the DC’s TGT, followed by a golden ticket attack on a SharePoint server with Unconstrained Delegation to gain the ability to impersonate any Site 1 AD account.
• Use of non-secure default configurations. The organization used default configurations for hosts with Windows Server 2012 R2. The default configuration allows unprivileged users to query group membership of local administrator groups. The red team used and identified several standard user accounts with administrative access from a Windows Server 2012 R2 SharePoint server.

Additional Issues

The team noted the following additional issues.

• Ineffective separation of privileged accounts. Some workstations allowed unprivileged accounts to have local administrator access; for example, the red team discovered an ordinary user account in the local admin group for the SharePoint server. If a user with administrative access is compromised, an actor can access servers without needing to elevate privileges. Administrative and user accounts should be separated, and designated admin accounts should be exclusively used for admin purposes.
• Lack of server egress control. Most servers, including domain controllers, allowed unrestricted egress traffic to the internet.
• Inconsistent host configuration. The team observed inconsistencies on servers and workstations within the domain, including inconsistent membership in the local administrator group among different servers or workstations. For example, some workstations had “Server Admins” or “Domain Admins” as local administrators, and other workstations had neither.
• Potentially unwanted programs. The team noticed potentially unusual software, including music software, installed on both workstations and servers. These extraneous software installations indicate inconsistent host configuration (see above) and increase the attack surfaces for malicious actors to gain initial access or escalate privileges once in the network.
• Mandatory password changes enabled. During the assessment, the team keylogged a user during a mandatory password change and noticed that only the final character of their password was modified. This is potentially due to domain passwords being required to be changed every 60 days.
• Smart card use was inconsistent across the domain. While the technology was deployed, it was not applied uniformly, and there was a significant portion of users without smartcard protections enabled. The team used these unprotected accounts throughout their assessment to move laterally through the domain and gain persistence.

Noted Strengths

The red team noted the following technical controls or defensive measures that prevented or hampered offensive actions:

• The organization conducts regular, proactive penetration tests and adversarial assessments and invests in hardening their network based on findings.
  • The team was unable to discover any easily exploitable services, ports, or web interfaces from more than three million external in-scope IPs. This forced the team to resort to phishing to gain initial access to the environment.
  • Service account passwords were strong. The team was unable to crack any of the hashes obtained from the 610 service accounts pulled. This is a critical strength because it slowed the team from moving around the network in the initial parts of the Phase I.
  • The team did not discover any useful credentials on open file shares or file servers. This slowed the progress of the team from moving around the network.
• MFA was used for some SBSs. The team was blocked from moving to SBS 2 by an MFA prompt.
• There were strong security controls and segmentation for SBS systems. Direct access to SBS were located in separate networks, and admins of SBS used workstations protected by local firewalls.

MITIGATIONS

CISA recommends organizations implement the recommendations in Table 2 to mitigate the issues listed in the Findings section of this advisory. These mitigations align with the Cross-Sector Cybersecurity Performance Goals (CPGs) developed by CISA and the National Institute of Standards and Technology (NIST). The CPGs provide a minimum set of practices and protections that CISA and NIST recommend all organizations implement. CISA and NIST based the CPGs on existing cybersecurity frameworks and guidance to protect against the most common and impactful threats, tactics, techniques, and procedures. See CISA’s Cross-Sector Cybersecurity Performance Goals for more information on the CPGs, including additional recommended baseline protections.

Additionally, CISA recommends organizations implement the mitigations below to improve their cybersecurity posture:

• Provide users with regular training and exercises, specifically related to phishing emails [CPG 4.3]. Phishing accounts for majority of initial access intrusion events.
• Enforce phishing-resistant MFA to the greatest extent possible [CPG 1.3].
• Reduce the risk of credential compromise via the following:
  • Place domain admin accounts in the protected users group to prevent caching of password hashes locally; this also forces Kerberos AES authentication as opposed to weaker RC4 or NTLM.
  • Implement Credential Guard for Windows 10 and Server 2016 (Refer to Microsoft: Manage Windows Defender Credential Guard for more information). For Windows Server 2012R2, enable Protected Process Light for Local Security Authority (LSA).
  • Refrain from storing plaintext credentials in scripts [CPG 3.4]. The red team discovered a PowerShell script containing plaintext credentials that allowed them to escalate to admin.
• Upgrade to Windows Server 2019 or greater and Windows 10 or greater. These versions have security features not included in older operating systems.

As a long-term effort, CISA recommends organizations prioritize implementing a more modern, Zero Trust network architecture that:

• Leverages secure cloud services for key enterprise security capabilities (e.g., identity and access management, endpoint detection and response, policy enforcement).
• Upgrades applications and infrastructure to leverage modern identity management and network access practices.
• Centralizes and streamlines access to cybersecurity data to drive analytics for identifying and managing cybersecurity risks.
• Invests in technology and personnel to achieve these goals.

CISA encourages organizational IT leadership to ask their executive leadership the question: Can the organization accept the business risk of NOT implementing critical security controls such as MFA? Risks of that nature should typically be acknowledged and prioritized at the most senior levels of an organization.

VALIDATE SECURITY CONTROLS

In addition to applying mitigations, CISA recommends exercising, testing, and validating your organization’s security program against the threat behaviors mapped to the MITRE ATT&CK for Enterprise framework in this advisory. CISA recommends testing your existing security controls inventory to assess how they perform against the ATT&CK techniques described in this advisory.

To get started:

1. Select an ATT&CK technique described in this advisory (see Table 3).
2. Align your security technologies against the technique.
3. Test your technologies against the technique.
4. Analyze your detection and prevention technologies’ performance.
5. Repeat the process for all security technologies to obtain a set of comprehensive performance data.
6. Tune your security program, including people, processes, and technologies, based on the data generated by this process.

CISA recommends continually testing your security program, at scale, in a production environment to ensure optimal performance against the MITRE ATT&CK techniques identified in this advisory.

RESOURCES

See CISA’s RedEye tool on CISA’s GitHub page. RedEye is an interactive open-source analytic tool used to visualize and report red team command and control activities. See CISA’s RedEye tool overview video for more information.

REFERENCES
[1] Bleeping Computer: New DFSCoerce NTLM Relay attack allows Windows domain takeover

APPENDIX: MITRE ATT&CK TACTICS AND TECHNIQUES

See Table 3 for all referenced red team tactics and techniques in this advisory. Note: activity was from Phase I unless noted.

Please share your thoughts. We recently updated our anonymous Product Feedback Survey and we’d welcome your feedback.

Summary

Technical Details

Open-source reporting indicates that malicious actors are exploiting known vulnerabilities in VMware ESXi software to gain access to servers and deploy ESXiArgs ransomware. The actors are likely targeting end-of-life ESXi servers or ESXi servers that do not have the available ESXi software patches applied.[1] 

ESXiArgs ransomware encrypts certain configuration files on ESXi servers, potentially rendering VMs unusable. Specifically, the ransomware encrypts configuration files associated with the VMs; it does not encrypt flat files. As a result, it is possible, in some cases, for victims to reconstruct the encrypted configuration files based on the unencrypted flat file. The recovery script documented below automates the process of recreating configuration files. The full list of file extensions encrypted by the malware is: vmdk, vmx, vmxf, vmsd, vmsn, vswp, vmss, nvram, vmem.

Recovery Guidance

CISA and FBI do not encourage paying the ransom as payment does not guarantee victim files will be recovered. Furthermore, payment may also embolden adversaries to target additional organizations, encourage other criminal actors to engage in the distribution of ransomware, and/or fund illicit activities. Regardless of whether you or your organization have decided to pay the ransom, CISA and FBI urge you to promptly report ransomware incidents to a local FBI Field Office, or to CISA at cisa.gov/report. 

CISA is providing these steps to enable organizations to attempt recovery of their VMs. CISA’s GitHub ESXiArgs recovery script, which also outlines these steps, is available at github.com/cisagov/ESXiArgs-Recover. CISA is aware that some organizations have reported success in recovering files without paying ransoms. CISA’s script is based on findings published by third-party researchers.[2] 

Any organization seeking to use CISA’s ESXiArgs recovery script should carefully review the script to determine if it is appropriate for their environment before deploying it. This script does not seek to delete the encrypted configuration files, but instead seeks to create new configuration files that enable access to the VMs. While CISA works to ensure that scripts like this one are safe and effective, this script is delivered without warranty, either implicit or explicit. Do not use this script without understanding how it may affect your system. CISA does not assume liability for damage caused by this script. Note: Organizations that run into problems with the script can create a GitHub issue at https://github.com/cisagov/ESXiArgs-Recover/issues; CISA will do our best to resolve concerns.

1. Quarantine or take affected hosts offline to ensure that repeat infection does not occur.
2. Download CISA’s recovery script and save it as /tmp/recover.sh.
   For example, with wget: wget -O /tmp/recover.sh https://raw.githubusercontent.com/cisagov/ESXiArgs-Recover/main/recover.sh.
3. Give the script execute permissions: chmod +x /tmp/recover.sh
4. Navigate to the folder of a VM you would like to recover and run ls to view the files.
   • Note: You may browse these folders by running ls /vmfs/volumes/datastore1. For instance, if the folder is called example, run cd /vmfs/volumes/datastore1/example.
5. View files by running ls. Note the name of the VM (via naming convention: [name].vmdk).
6. Run the recovery script with /tmp/recover.sh [name], where [name] is the name of the VM determined previously. 
   • If the VM is a thin format, run /tmp/recover.sh [name] thin.
   • If successful, the recovery script will output that it has successfully run. If unsuccessful, it may not be possible for the recovery script to recover your VMs; consider engaging external incident response help.
7. If the script succeeded, re-register the VM.
   1. If the ESXi web interface is inaccessible, remove the ransom note and restore access via the following steps. (Note: Taking the steps below moves the ransom note to the file ransom.html. Consider archiving this file for future incident review.)
      • Run cd /usr/lib/vmware/hostd/docroot/ui/ && mv index.html ransom.html && mv index1.html index.html.
      • Run cd /usr/lib/vmware/hostd/docroot && mv index.html ransom.html && rm index.html && mv index1.html index.html.
      • Reboot the ESXi server (e.g., with the reboot command). After a few minutes, you should be able to navigate to the web interface.
      • In the ESXi web interface, navigate to the Virtual Machines page.
      • If the VM you restored already exists, right click on the VM and select Unregister (see figure 1).

Figure 1: Unregistering the virtual machine.

• Select Create / Register VM (see figure 2).
• Select Register an existing virtual machine (see figure 2).

Figure 2: Registering the virtual machine, selecting machine to register.

Click Select one or more virtual machines, a datastore or a directory to navigate to the folder of the VM you restored. Select the vmx file in the folder (see figure 3).

Figure 3: Registering the virtual machine, finalizing registration.

Select Next and Finish. You should now be able to use the VM as normal.

Figure 3: Registering the virtual machine, finalizing registration.

Select Next and Finish. You should now be able to use the VM as normal.

8. Update servers to the latest software version, disable the Service Location Protocol (SLP) service, and ensure the ESXi hypervisor is not configured to be exposed to the public internet before putting systems back online. 

Additional Incident Response

The above script only serves as a method to recover essential services. Although CISA and FBI have not seen any evidence that the actors have established persistence, we recommend organizations take the following additional incident response actions after applying the script:

1. Review network logging to and from ESXi hosts and the guest VMs for unusual scanning activity.
2. Review traffic from network segments occupied by the ESXi hosts and guests. Consider restricting non-essential traffic to and from these segments.

If you detect activity from the above, implement your incident response plan. CISA and FBI urge you to promptly report ransomware incidents to a local FBI Field Office, or to CISA at cisa.gov/report.

Organizations should also collect and review artifacts, such as running processes/services, unusual authentications, and recent network connections.

See the joint CSA from the cybersecurity authorities of Australia, Canada, New Zealand, the United Kingdom, and the United States on Technical Approaches to Uncovering and Remediating Malicious Activity for additional guidance on hunting or investigating a network, and for common mistakes in incident handling. CISA also encourages government network administrators to see CISA’s Federal Government Cybersecurity Incident and Vulnerability Response Playbooks. Although tailored to federal civilian branch agencies, these playbooks provide operational procedures for planning and conducting cybersecurity incident and vulnerability response activities and detail steps for both incident and vulnerability response.  

Additional resources for recovering .vmdk files can be found on a third-party researcher’s website.[2]

Mitigations

References

Revisions