Original release date: May 18, 2022

Summary

The Cybersecurity and Infrastructure Security Agency (CISA) is releasing this Cybersecurity Advisory (CSA) to warn organizations that malicious cyber actors, likely advanced persistent threat (APT) actors, are exploiting CVE-2022-22954 and CVE-2022-22960 separately and in combination. These vulnerabilities affect certain versions of VMware Workspace ONE Access, VMware Identity Manager (vIDM), VMware vRealize Automation (vRA), VMware Cloud Foundation, and vRealize Suite Lifecycle Manager. Exploiting these vulnerabilities permits malicious actors to trigger a server-side template injection that may result in remote code execution (RCE) (CVE-2022-22954) or escalation of privileges to root (CVE-2022-22960). 

VMware released updates for both vulnerabilities on April 6, 2022, and, according to a trusted third party, malicious cyber actors were able to reverse engineer the updates to develop an exploit within 48 hours and quickly began exploiting the disclosed vulnerabilities in unpatched devices. CISA was made aware of this exploit a week later and added CVE-2022-22954 and CVE-2022-22960 to its catalog of Known Exploited Vulnerabilities on April 14 and April 15, respectively. In accordance with Binding Operational Directive (BOD) 22-01, Reducing the Significant Risk of Known Exploited Vulnerabilities, federal agencies were required to apply updates for CVE-2022-22954 and CVE-2022-22960 by May 5, and May 6, 2022, respectively

Note: based on this activity, CISA expects malicious cyber actors to quickly develop a capability to exploit newly released vulnerabilities CVE-2022-22972 and CVE-2022-22973 in the same impacted VMware products. In response, CISA has released, Emergency Directive (ED) 22-03 Mitigate VMware Vulnerabilities, which requires emergency action from Federal Civilian Executive Branch agencies to either immediately implement the updates in VMware Security Advisory VMSA-2022-0014 or remove the affected software from their network until the updates can be applied.

CISA has deployed an incident response team to a large organization where the threat actors exploited CVE-2022-22954. Additionally, CISA has received information—including indicators of compromise (IOCs)—about observed exploitation at multiple other large organizations from trusted third parties.

This CSA provides IOCs and detection signatures from CISA as well as from trusted third parties to assist administrators with detecting and responding to this activity. Due to the rapid exploitation of these vulnerabilities, CISA strongly encourages all organizations with internet-facing affected systems—that did not immediately apply updates—to assume compromise and initiate threat hunting activities using the detection methods provided in this CSA. If potential compromise is detected, administrators should apply the incident response recommendations included in this CSA.

Download the PDF version of this report (pdf, 232kb).

Technical Details

CISA has deployed an incident response team to a large organization where the threat actors exploited CVE-2022-22954. Additionally, CISA has received information about observed exploitation of CVE-2022-22954 and CVE-2022-22960 by multiple threat actors at multiple other large organizations from trusted third parties.

• CVE-2022-22954 enables an actor with network access to trigger a server-side template injection that may result in RCE. This vulnerability affects the following products:[1]
  • VMware Workspace ONE Access, versions 21.08.0.1, 21.08.0.0, 20.10.0.1, 20.10.0.0
  • vIDM versions 3.3.6, 3.3.5, 3.3.4, 3.3.3
  • VMware Cloud Foundation, 4.x
  • vRealize Suite LifeCycle Manager, 8.
• CVE-2022-22960 enables a malicious actor with local access to escalate privileges to root due to improper permissions in support scripts. This vulnerability affects the following products:[2]
  • VMware Workspace ONE Access, versions 21.08.0.1, 21.08.0.0, 20.10.0.1, 20.10.0.0
  • vIDM, versions 3.3.6, 3.3.5, 3.3.4, 3.3.3
  • vRA, version 7.6 
  • VMware Cloud Foundation, 3.x, 4.x, 
  • vRealize Suite LifeCycle Manager, 8.x

According to trusted third-party reporting, threat actors may chain these vulnerabilities. At one compromised organization, on or around April 12, 2022, an unauthenticated actor with network access to the web interface leveraged CVE-2022-22954 to execute an arbitrary shell command as a VMware user. The actor then exploited CVE-2022-22960 to escalate the user’s privileges to root. With root access, the actor could wipe logs, escalate permissions, and move laterally to other systems.

Threat actors have dropped post-exploitation tools, including the Dingo J-spy webshell. During incident response activities, CISA observed, on or around April 13, 2022, threat actors leveraging CVE-2022-22954 to drop the Dingo J-spy webshell. Around the same period, a trusted third party observed threat actors leveraging CVE-2022-22954 to drop the Dingo J-spy webshell at one other organization. According to the third party, the actors may have also dropped the Dingo J-spy webshell at a third organization. Note: analysis of the first compromise and associated malware is ongoing, and CISA will update information about this case as we learn more.

Detection Methods

Signatures

Note: servers vulnerable to CVE-2022-22954 may use Hypertext Transfer Protocol Secure (HTTPS) to encrypt client/server communications. Secure Sockets Layer (SSL)/Transport Layer Security (TLS) decryption can be used as a workaround for network-based detection and threat hunting efforts.

The following CISA-created Snort signature may detect malicious network traffic related to exploitation of CVE-2022-22954:

The following third-party Snort signature may detect exploitation of VMware Workspace ONE Access server-side template injection:

The following third-party YARA rule may detect unmodified instances of the Dingo J-spy webshell on infected hosts:

Note: the Dingo J-spy webshell is an example of post-exploitation tools that actors have used. Administrators should examine their network for any sign of post-exploitation activity.

Behavioral Analysis and Indicators of Compromise

Administrators should conduct behavioral analysis on root accounts of vulnerable systems by:

• Using the indicators listed in table 1 to detect potential malicious activity.
• Reviewing systems logs and gaps in logs.
• Reviewing abnormal connections to other assets.
• Searching the command-line history.
• Auditing running processes.
• Reviewing local user accounts and groups.  
• Auditing active listening ports and connections.

Table 1: Third-party IOCs for Exploitation of CVE-2022-22954 and CVE-2022-22960

Incident Response

If administrators discover system compromise, CISA recommends they:

1. Immediately isolate affected systems. 
2. Collect and review relevant logs, data, and artifacts.
3. Consider soliciting support from a third-party incident response organization to provide subject matter expertise, ensure the actor is eradicated from the network, and avoid residual issues that could enable follow-on exploitation.
4. Report incidents to CISA via CISA’s 24/7 Operations Center (report@cisa.gov or 888-282-0870)

Mitigations

CISA recommends organizations update impacted VMware products to the latest version or remove impacted versions from organizational networks. CISA does not endorse alternative mitigation options. As noted in ED 22-03 Mitigate VMware Vulnerabilities, CISA expects malicious cyber actors to quickly develop a capability to exploit newly released vulnerabilities CVE-2022-22972 and CVE-2022-22973 in the same impacted VMware products. ED 22-03 directs all Federal Civilian Executive Branch agencies to enumerate all instances of impacted VMware products and deploy updates in VMware Security Advisory VMSA-2022-0014 or to remove the affected software from the agency network until the updates can be applied.

Resources

• ED 22-03 Mitigate VMware Vulnerabilities
• VMware Security Advisory VMSA-2022-0011
• VMware Security Advisory VMSA-2022-0014

Contact Information

CISA encourages recipients of this CSA to report incidents to CISA via CISA’s 24/7 Operations Center (report@cisa.gov or 888-282-0870)

References

• [1] VMware Security Advisory VMSA-2022-0011
• [2] Ibid

Revisions

• Initial Version: May 18, 2022

Original release date: May 18, 2022

Summary

Actions for administrators to take today:
• Do not expose management interfaces to the internet.
• Enforce multi-factor authentication.
• Consider using CISA’s Cyber Hygiene Services.

The Cybersecurity and Infrastructure Security Agency (CISA) and the Multi-State Information Sharing & Analysis Center (MS-ISAC) are releasing this joint Cybersecurity Advisory (CSA) in response to active exploitation of CVE-2022-1388. This recently disclosed vulnerability in certain versions of F5 Networks, Inc., (F5) BIG-IP enables an unauthenticated actor to gain control of affected systems via the management port or self-IP addresses. F5 released a patch for CVE-2022-1388 on May 4, 2022, and proof of concept (POC) exploits have since been publicly released, enabling less sophisticated actors to exploit the vulnerability. Due to previous exploitation of F5 BIG-IP vulnerabilities, CISA and MS-ISAC assess unpatched F5 BIG-IP devices are an attractive target; organizations that have not applied the patch are vulnerable to actors taking control of their systems.

According to public reporting, there is active exploitation of this vulnerability, and CISA and MS-ISAC expect to see widespread exploitation of unpatched F5 BIG-IP devices (mostly with publicly exposed management ports or self IPs) in both government and private sector networks. CISA and MS-ISAC strongly urge users and administrators to remain aware of the ramifications of exploitation and use the recommendations in this CSA—including upgrading their software to fixed versions—to help secure their organization’s systems against malicious cyber operations. Additionally, CISA and MS-ISAC strongly encourage administrators to deploy the signatures included in this CSA to help determine whether their systems have been compromised. CISA and MS-ISAC especially encourage organizations who did not patch immediately or whose F5 BIG-IP device management interface has been exposed to the internet to assume compromise and hunt for malicious activity using the detection signatures in this CSA. If potential compromise is detected, organizations should apply the incident response recommendations included in this CSA.

Download the PDF version of this report (pdf, 500kb).

Technical Details

CVE-2022-1388 is a critical iControl REST authentication bypass vulnerability affecting the following versions of F5 BIG-IP:[1]

• 16.1.x versions prior to 16.1.2.2 
• 15.1.x versions prior to 15.1.5.1 
• 14.1.x versions prior to 14.1.4.6 
• 13.1.x versions prior to 13.1.5 
• All 12.1.x and 11.6.x versions

An unauthenticated actor with network access to the BIG-IP system through the management port or self IP addresses could exploit the vulnerability to execute arbitrary system commands, create or delete files, or disable services. F5 released a patch for CVE-2022-1388 for all affected versions—except 12.1.x and 11.6.x versions—on May 4, 2022 (12.1.x and 11.6.x versions are end of life [EOL], and F5 has stated they will not release patches).[2]

POC exploits for this vulnerability have been publicly released, and on May 11, 2022, CISA added this vulnerability its Known Exploited Vulnerabilities Catalog, based on evidence of active exploitation. Due to the POCs and ease of exploitation, CISA and MS-ISAC expect to see widespread exploitation of unpatched F5 BIG-IP devices in government and private networks. 

Dection Methods

CISA recommends administrators, especially of organizations who did not immediately patch, to:

• See the F5 Security Advisory K23605346 for indicators of compromise. 
• See the F5 guidance K11438344 if you suspect a compromise. 
• Deploy the following CISA-created Snort signature:

Additional resources to detect possible exploitation or compromise are identified below:

• Emerging Threats suricata signatures. Note: CISA and MS-ISAC have verified these signatures are successful in detection of both inbound exploitation attempts (SID: 2036546) as well as post exploitation, indicating code execution (SID: 2036547).
  • SID 2036546

• SID SID 2036547

• Palo Alto Networks Unit 42 Threat Brief: CVE-2022-1388. This brief includes indicators of compromise. Note: due to the urgency to share this information, CISA and MS-ISAC have not yet validated this content.

• Cisco Talos Intelligence Group – Comprehensive Threat Intelligence: Threat Advisory: Critical F5 BIG-IP Vulnerability. This blog includes indicators of compromise. Note: due to the urgency to share this information, CISA and MS-ISAC have not yet validated this content.

• Randori’s bash script. This script can be used to identify vulnerable instances of BIG-IP. Note: MS-ISAC has verified this bash script identifies vulnerable instances of BIG-IP. 

Incident Response 

If an organization’s IT security personnel discover system compromise, CISA and MS-ISAC recommend they:

1. Quarantine or take offline potentially affected hosts.
2. Reimage compromised hosts.
3. Provision new account credentials.
4. Limit access to the management interface to the fullest extent possible.
5. Collect and review artifacts such as running processes/services, unusual authentications, and recent network connections.
6. Report the compromise to CISA via CISA’s 24/7 Operations Center (report@cisa.gov or 888-282-0870). State, local, tribal, or territorial government entities can also report to MS-ISAC (SOC@cisecurity.org or 866-787-4722).

See the joint CSA from the cybersecurity authorities of Australia, Canada, New Zealand, the United Kingdom, and the United States on Technical Approaches to Uncovering and Remediating Malicious Activity for additional guidance on hunting or investigating a network, and for common mistakes in incident handling. CISA and MS-ISAC also encourage government network administrators to see CISA’s Federal Government Cybersecurity Incident and Vulnerability Response Playbooks. Although tailored to federal civilian branch agencies, these playbooks provide operational procedures for planning and conducting cybersecurity incident and vulnerability response activities and detail steps for both incident and vulnerability response. 

Mitigations

CISA and MS-ISAC recommend organizations:

• Upgrade F5 BIG-IP software to fixed versions; organizations using versions 12.1.x and 11.6.x should upgrade to supported versions. 
• If unable to immediately patch, implement F5’s temporary workarounds:
  • Block iControl REST access through the self IP address.
  • Block iControl REST access through the management interface.
  • Modify the BIG-IP httpd configuration. 

See F5 Security Advisory K23605346 for more information on how to implement the above workarounds. 

CISA and MS-ISAC also recommend organizations apply the following best practices to reduce risk of compromise:

• Maintain and test an incident response plan.
• Ensure your organization has a vulnerability program in place and that it prioritizes patch management and vulnerability scanning. Note: CISA’s Cyber Hygiene Services (CyHy) are free to all SLTT organizations and public and private sector critical infrastructure organizations: https://www.cisa.gov/cyber-hygiene-services.
• Properly configure and secure internet-facing network devices.
  • Do not expose management interfaces to the internet.
  • Disable unused or unnecessary network ports and protocols.
  • Disable/remove unused network services and devices.
• Adopt zero-trust principles and architecture, including:
  • Micro-segmenting networks and functions to limit or block lateral movements.
  • Enforcing multifactor authentication (MFA) for all users and VPN connections.
  • Restricting access to trusted devices and users on the networks.

References

• [1] K23605346: BIG-IP iControl REST vulnerability CVE-2022-1388
• [2] K11438344: Considerations and guidance when you suspect a security compromise on a BIG-IP system

Revisions

• Initial Version: May 18, 2022

Original release date: May 17, 2022

Summary

Best Practices to Protect Your Systems:
• Control access.
• Harden Credentials.
• Establish centralized log management.
• Use antivirus solutions.
• Employ detection tools.
• Operate services exposed on internet-accessible hosts with secure configurations.
• Keep software updated.

Cyber actors routinely exploit poor security configurations (either misconfigured or left unsecured), weak controls, and other poor cyber hygiene practices to gain initial access or as part of other tactics to compromise a victim’s system. This joint Cybersecurity Advisory identifies commonly exploited controls and practices and includes best practices to mitigate the issues. This advisory was coauthored by the cybersecurity authorities of the United States,[1],[2],[3] Canada,[4] New Zealand,[5],[6] the Netherlands,[7] and the United Kingdom.[8]

Download the PDF version of this report (pdf, 430kb).

Technical Details

Malicious actors commonly use the following techniques to gain initial access to victim networks.[TA0001]

• Exploit Public-Facing Application [T1190] 
• External Remote Services [T1133]
• Phishing [T1566]
• Trusted Relationship [T1199] 
• Valid Accounts [T1078]

Malicious cyber actors often exploit the following common weak security controls, poor configurations, and poor security practices to employ the initial access techniques.

• Multifactor authentication (MFA) is not enforced. MFA, particularly for remote desktop access, can help prevent account takeovers. With Remote Desktop Protocol (RDP) as one of the most common infection vector for ransomware, MFA is a critical tool in mitigating malicious cyber activity. Do not exclude any user, particularly adminstrators, from an MFA requirement. 
• Incorrectly applied privileges or permissions and errors within access control lists. These mistakes can prevent the enforcement of access control rules and could allow unauthorized users or system processes to be granted access to objects. 
• Software is not up to date. Unpatched software may allow an attacker to exploit publicly known vulnerabilities to gain access to sensitive information, launch a denial-of-service attack, or take control of a system. This is one of the most commonly found poor security practices.
• Use of vendor-supplied default configurations or default login usernames and passwords. Many software and hardware products come “out of the box” with overly permissive factory-default configurations intended to make the products user-friendly and reduce the troubleshooting time for customer service. However, leaving these factory default configurations enabled after installation may provide avenues for an attacker to exploit. Network devices are also often pre-configured with default administrator usernames and passwords to simplify setup. These default credentials are not secure—they may be physically labeled on the device or even readily available on the internet. Leaving these credentials unchanged creates opportunities for malicious activity, including gaining unauthorized access to information and installing malicious software. Network defenders should also be aware that the same considerations apply for extra software options, which may come with preconfigured default settings.
• Remote services, such as a virtual private network (VPN), lack sufficient controls to prevent unauthorized access. During recent years, malicious threat actors have been observed targeting remote services. Network defenders can reduce the risk of remote service compromise by adding access control mechanisms, such as enforcing MFA, implementing a boundary firewall in front of a VPN, and leveraging intrusion detection system/intrusion prevention system sensors to detect anomalous network activity.  
• Strong password policies are not implemented. Malicious cyber actors can use a myriad of methods to exploit weak, leaked, or compromised passwords and gain unauthorized access to a victim system. Malicious cyber actors have used this technique in various nefarious acts and prominently in attacks targeting RDP. 
• Cloud services are unprotected. Misconfigured cloud services are common targets for cyber actors. Poor configurations can allow for sensitive data theft and even cryptojacking.
• Open ports and misconfigured services are exposed to the internet. This is one of the most common vulnerability findings. Cyber actors use scanning tools to detect open ports and often use them as an initial attack vector. Successful compromise of a service on a host could enable malicious cyber actors to gain initial access and use other tactics and procedures to compromise exposed and vulnerable entities. RDP, Server Message Block (SMB), Telnet, and NetBIOS are high-risk services. 
• Failure to detect or block phishing attempts. Cyber actors send emails with malicious macros—primarily in Microsoft Word documents or Excel files—to infect computer systems. Initial infection can occur in a variety of ways, such as when a user opens or clicks a malicious download link, PDF, or macro-enabled Microsoft Word document included in phishing emails. 
• Poor endpoint detection and response. Cyber actors use obfuscated malicious scripts and PowerShell attacks to bypass endpoint security controls and launch attacks on target devices. These techniques can be difficult to detect and protect against. 

Mitigations

Applying the following practices can help organizations strengthen their network defenses against common exploited weak security controls and practices.

Control Access

• Adopt a zero-trust security model that eliminates implicit trust in any one element, node, or service, and instead requires continuous verification of the operational picture via real-time information from multiple sources to determine access and other system responses.[9],[10] Zero-trust architecture enables granular privilege access management and can allow users to be assigned only the rights required to perform their assigned tasks.
• Limit the ability of a local administrator account to log in from a remote session (e.g., deny access to this computer from the network) and prevent access via an RDP session. Additionally, use dedicated administrative workstations for privileged user sessions to help limit exposure to all the threats associated with device or user compromise. 
• Control who has access to your data and services. Give personnel access only to the data, rights, and systems they need to perform their job. This role-based access control, also known as the principle of least priviledge, should apply to both accounts and physical access. If a malicious cyber actor gains access, access control can limit the actions malicious actors can take and can reduce the impact of misconfigurations and user errors. Network defenders should also use this role-based access control to limit the access of service, machine, and functional accounts, as well as the use of management privileges, to what is necessary. Consider the following when implementing access control models:
  • Ensure that access to data and services is specifically tailored to each user, with each employee having their own user account. 
  • Give employees access only to the resources needed to perform their tasks.
  • Change default passwords of equipment and systems upon installation or commissioning. 
  • Ensure there are processes in place for the entry, exit, and internal movement of employees. Delete unused accounts, and immediately remove access to data and systems from accounts of exiting employees who no longer require access. Deactivate service accounts, and activate them only when maintenance is performed.[11]
• Harden conditional access policies. Review and optimize VPN and access control rules to manage how users connect to the network and cloud services.
• Verify that all machines, including cloud-based virtual machine instances do not have open RDP ports. Place any system with an open RDP port behind a firewall and require users to use a VPN to access it through the firewall.[12]

Implement Credential Hardening

• Implement MFA. In particular, apply MFA on all VPN connections, external-facing services, and privileged accounts. Require phishing-resistant MFA (such as security keys or PIV cards) for critical services. Where MFA is not implemented, enforce a strong password policy alongside other attribute-based information, such as device information, time of access, user history, and geolocation data. See NSA’s Cybersecurity Information on Selecting Secure Multi-factor Authentication Solutions, the National Institute for Standards and Technology (NIST) Special Publication 800-63B – Digital Identity Guidelines: Authentication and Lifecycle Management, and CCCS’s Information Technology Security Guidance – User Authentication Guidance for Information Technology Systems for additional steps to take to enable in-depth authentication security.
• Change or disable vendor-supplied default usernames and passwords. Enforce the use of strong passwords. (See guidance from NIST.)
• Set up monitoring to detect the use of compromised credentials on your systems. Implement controls to prevent the use of compromised or weak passwords on your network. 

Establish Centralized Log Management

• Ensure that each application and system generates sufficient log information. Log files play a key role in detecting attacks and dealing with incidents. By implementing robust log collection and retention, organizations are able to have sufficient information to investigate incidents and detect threat actor behavior. Consider the following when implementing log collection and retention: 
  • Determine which log files are required. These files can pertain to system logging, network logging, application logging, and cloud logging. 
  • Set up alerts where necessary. These should include notifications of suspicious login attempts based on an analysis of log files. 
  • Ensure that your systems store log files in a usable file format, and that the recorded timestamps are accurate and set to the correct time zone. 
  • Forward logs off local systems to a centralized repository or security information and event management (SIEM) tools. Robustly protect SIEM tools with strong account and architectural safeguards.
  • Make a decision regarding the retention period of log files. If you keep log files for a long time, you can refer to them to determine facts long after incidents occur. On the other hand, log files may contain privacy-sensitive information and take up storage space. Limit access to log files and store them in a separate network segment. An incident investigation will be nearly impossible if attackers have been able to modify or delete the logfiles.[13]

Employ Antivirus Programs

• Deploy an anti-malware solution on workstations to prevent spyware, adware, and malware as part of the operating system security baseline.
• Monitor antivirus scan results on a routine basis.

Employ Detection Tools and Search for Vulnerabilities

• Implement endpoint and detection response tools. These tools allow a high degree of visibility into the security status of endpoints and can help effectively protect against malicious cyber actors.
• Employ an intrusion detection system or intrusion prevention system to protect network and on-premises devices from malicious activity. Use signatures to help detect malicious network activity associated with known threat activity.
• Conduct penetration testing to identify misconfigurations. See the Additional Resources section below for more information about CISA’s free cyber hygiene services, including remote penetration testing.
• Conduct vulnerability scanning to detect and address application vulnerabilities. 
• Use cloud service provider tools to detect overshared cloud storage and monitor for abnormal accesses.

Maintain Rigorous Configuration Management Programs

• Always operate services exposed on internet-accessible hosts with secure configurations. Never enable external access without compensating controls such as boundary firewalls and segmentation from other more secure and internal hosts like domain controllers. Continuously assess the business and mission need of internet-facing services. Follow best practices for security configurations, especially blocking macros in documents from the internet.[14]

Initiate a Software and Patch Management Program 

• Implement asset and patch management processes to keep software up to date. Identify and mitigate unsupported, end-of-life, and unpatched software and firmware by performing vulnerability scanning and patching activities. Prioritize patching known exploited vulnerabilities.

Additional Resources 

• NCSC-UK Guidance – Phishing Attacks: Defending Your Organisation 
• Open Web Application Security Project (OWASP) Proactive Controls: Enforce Access Controls

References 

[1] United States Cybersecurity and Infrastructure Security Agency 
[2] United States Federal Bureau of Investigation
[3] United States National Security Agency
[4] Canadian Centre for Cyber Security 
[5] New Zealand National Cyber Security Centre 
[6] New Zealand CERT NZ
[7] Netherlands National Cyber Security Centre
[8] United Kingdom National Cyber Security Centre 
[9] White House Executive Order on Improving the Nation’s Cybersecurity
[10] NCSC-NL Factsheet: Prepare for Zero Trust
[11] NCSC-NL Guide to Cyber Security Measures
[12] N-able Blog: Intrusion Detection System (IDS): Signature vs. Anomaly-Based
[13] NCSC-NL Guide to Cyber Security Measures
[14] National Institute of Standards and Technology SP 800-123 – Keeping Servers Secured

Contact

U.S. organizations: To report incidents and anomalous activity or to request incident response resources or technical assistance related to these threats, contact CISA at report@cisa.gov. To report computer intrusion or cybercrime activity related to information found in this advisory, contact your local FBI field office at www.fbi.gov/contact-us/field, or the FBI’s 24/7 Cyber Watch at 855-292-3937 or by email at CyWatch@fbi.gov. For NSA client requirements or general cybersecurity inquiries, contact Cybersecurity_Requests@nsa.gov. 

Canadian organizations: report incidents by emailing CCCS at contact@cyber.gc.ca. 

New Zealand organizations: report cyber security incidents to incidents@ncsc.govt.nz or call 04 498 7654. 

The Netherlands organizations: report incidents to cert@ncsc.nl. 

United Kingdom organizations: report a significant cyber security incident: ncsc.gov.uk/report-an-incident (monitored 24 hours) or, for urgent assistance, call 03000 200 973.

Caveats

The information you have accessed or received is being provided “as is” for informational purposes only. CISA, the FBI, NSA, CCCS, NCSC-NZ, CERT-NZ, NCSC-NL, and NCSC-UK do not endorse any commercial product or service, including any subjects of analysis. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply their endorsement, recommendation, or favoring.

Purpose

This document was developed by CISA, the FBI, NSA, CCCS, NCSC-NZ, CERT-NZ, NCSC-NL, and NCSC-UK in furtherance of their respective cybersecurity missions, including their responsibilities to develop and issue cybersecurity specifications and mitigations. This information may be shared broadly to reach all appropriate stakeholders. 

Revisions

• May 17, 2022: Initial version

Original release date: May 11, 2022

Summary

Tactical actions for MSPs and their customers to take today:
• Identify and disable accounts that are no longer in use.
• Enforce MFA on MSP accounts that access the customer environment and monitor for unexplained failed authentication.
• Ensure MSP-customer contracts transparently identify ownership of ICT security roles and responsibilities.

The cybersecurity authorities of the United Kingdom (NCSC-UK), Australia (ACSC), Canada (CCCS), New Zealand (NCSC-NZ), and the United States (CISA), (NSA), (FBI) are aware of recent reports that observe an increase in malicious cyber activity targeting managed service providers (MSPs) and expect this trend to continue.[1] This joint Cybersecurity Advisory (CSA) provides actions MSPs and their customers can take to reduce their risk of falling victim to a cyber intrusion. This advisory describes cybersecurity best practices for information and communications technology (ICT) services and functions, focusing on guidance that enables transparent discussions between MSPs and their customers on securing sensitive data. Organizations should implement these guidelines as appropriate to their unique environments, in accordance with their specific security needs, and in compliance with applicable regulations. MSP customers should verify that the contractual arrangements with their provider include cybersecurity measures in line with their particular security requirements.

The guidance provided in this advisory is specifically tailored for both MSPs and their customers and is the result of a collaborative effort from the United Kingdom National Cyber Security Centre (NCSC-UK), the Australian Cyber Security Centre (ACSC), the Canadian Centre for Cyber Security (CCCS), the New Zealand National Cyber Security Centre (NCSC-NZ), the United States’ Cybersecurity and Infrastructure Security Agency (CISA), National Security Agency (NSA), and Federal Bureau of Investigation (FBI) with contributions from industry members of the Joint Cyber Defense Collaborative (JCDC). Organizations should read this advisory in conjunction with NCSC-UK guidance on actions to take when the cyber threat is heightened, CCCS guidance on Cyber Security Considerations for Consumers of Managed Services, and CISA guidance provided on the Shields Up and Shields Up Technical Guidance webpages.

Managed Service Providers

This advisory defines MSPs as entities that deliver, operate, or manage ICT services and functions for their customers via a contractual arrangement, such as a service level agreement. In addition to offering their own services, an MSP may offer services in conjunction with those of other providers. Offerings may include platform, software, and IT infrastructure services; business process and support functions; and cybersecurity services. MSPs typically manage these services and functions in their customer’s network environment—either on the customer’s premises or hosted in the MSP’s data center. Note: this advisory does not address guidance on cloud service providers (CSPs)—providers who handle the ICT needs of their customers via cloud services such as Software-as-a-Service, Platform-as-a-Service, and Infrastructure-as-a-Service; however, MSPs may offer these services as well. (See Appendix for additional definitions.)

MSPs provide services that usually require both trusted network connectivity and privileged access to and from customer systems. Many organizations—ranging from large critical infrastructure organizations to small- and mid-sized businesses—use MSPs to manage ICT systems, store data, or support sensitive processes. Many organizations make use of MSPs to scale and support network environments and processes without expanding their internal staff or having to develop the capabilities internally. 

Threat Actors Targeting MSP Access to Customer Networks

Whether the customer’s network environment is on premises or externally hosted, threat actors can use a vulnerable MSP as an initial access vector to multiple victim networks, with globally cascading effects. The UK, Australian, Canadian, New Zealand, and U.S. cybersecurity authorities expect malicious cyber actors—including state-sponsored advanced persistent threat (APT) groups—to step up their targeting of MSPs in their efforts to exploit provider-customer network trust relationships. For example, threat actors successfully compromising an MSP could enable follow-on activity—such as ransomware and cyber espionage—against the MSP as well as across the MSP’s customer base.

The UK, Australian, Canadian, New Zealand, and U.S. cybersecurity authorities have previously issued general guidance for MSPs and their customers.[2],[3],[4],[5],[6],[7],[8] This advisory provides specific guidance to enable transparent, well-informed discussions between MSPs and their customers that center on securing sensitive information and data. These discussions should result in a re-evaluation of security processes and contractual commitments to accommodate customer risk tolerance. A shared commitment to security will reduce risk for both MSPs and their customers, as well as the global ICT community. 

Download the Joint Cybersecurity Advisory: Protecting Against Cyber Threats to Managed Service Providers and their Customers (pdf, 697kb).

Recommendations 

MSPs and their Customers

The UK, Australian, Canadian, New Zealand, and U.S. cybersecurity authorities recommend MSPs and their customers implement the baseline security measures and operational controls listed in this section. Additionally, customers should ensure their contractual arrangements specify that their MSP implements these measures and controls.

Prevent initial compromise. 

In their efforts to compromise MSPs, malicious cyber actors exploit vulnerable devices and internet-facing services, conduct brute force attacks, and use phishing techniques. MSPs and their customers should ensure they are mitigating these attack methods. Useful mitigation resources on initial compromise attack methods are listed below:

• Improve security of vulnerable devices.
  • Selecting and Hardening Remote Access VPN Solutions (CISA, NSA) 
  • Vulnerability Scanning Tools and Services (NCSC-UK)
• Protect internet-facing services.
  • Protecting internet-facing services on public service Critical National Infrastructure (CNI) (NCSC-UK)
  • Strategies for protecting web application systems against credential stuffing attacks (CCCS)
• Defend against brute force and password spraying.
  • Microsoft update on brute force and password spraying activity (NCSC-UK)
  • Russian GRU Conducting Global Brute Force Campaign to Compromise Enterprise and Cloud Environments (NSA, CISA, FBI, NCSC-UK)
• Defend against phishing.
  • Phishing attacks: defending your organisation (NCSC-UK)
  • Spotting malicious email messages (CCCS)

Enable/improve monitoring and logging processes. 

It can be months before incidents are detected, so UK, Australian, Canadian, New Zealand, and U.S. cybersecurity authorities recommend all organizations store their most important logs for at least six months. Whether through a comprehensive security information and event management (SIEM) solution or discrete logging tools, implement and maintain a segregated logging regime to detect threats to networks. Organizations can refer to the following NCSC-UK guidance on the appropriate data to collect for security purposes and when to use it: What exactly should we be logging? Additionally, all organizations—whether through contractual arrangements with an MSP or on their own—should implement endpoint detection and network defense monitoring capabilities in addition to using application allowlisting/denylisting. 

• MSPs should log the delivery infrastructure activities used to provide services to the customer. MSPs should also log both internal and customer network activity, as appropriate and contractually agreed upon. 
• Customers should enable effective monitoring and logging of their systems. If customers choose to engage an MSP to perform monitoring and logging, they should ensure that their contractual arrangements require their MSP to:
  • Implement comprehensive security event management that enables appropriate monitoring and logging of provider-managed customer systems; 
  • Provide visibility—as specified in the contractual arrangement—to customers of logging activities, including provider’s presence, activities, and connections to the customer networks (Note: customers should ensure that MSP accounts are properly monitored and audited.); and
  • Notify customer of confirmed or suspected security events and incidents occurring on the provider’s infrastructure and administrative networks, and send these to a security operations center (SOC) for analysis and triage. 

Enforce multifactor authentication (MFA). 

Organizations should secure remote access applications and enforce MFA where possible to harden the infrastructure that enables access to networks and systems.[9],[10] Note: Russian state-sponsored APT actors have recently demonstrated the ability to exploit default MFA protocols; organizations should review configuration policies to protect against “fail open” and re-enrollment scenarios.[11] 

• MSPs should recommend the adoption of MFA across all customer services and products. Note: MSPs should also implement MFA on all accounts that have access to customer environments and should treat those accounts as privileged.
• Customers should ensure that their contractual arrangements mandate the use of MFA on the services and products they receive. Contracts should also require MFA to be enforced on all MSP accounts used to access customer environments.

Manage internal architecture risks and segregate internal networks. 

Organizations should understand their environment and segregate their networks. Identify, group, and isolate critical business systems and apply appropriate network security controls to them to reduce the impact of a compromise across the organization.[12],[13]

• MSPs should review and verify all connections between internal systems, customer systems, and other networks. Segregate customer data sets (and services, where applicable) from each other—as well as from internal company networks—to limit the impact of a single vector of attack. Do not reuse admin credentials across multiple customers. 
• Customers should review and verify all connections between internal systems, MSP systems, and other networks. Ensure management of identity providers and trusts between the different environments. Use a dedicated virtual private network (VPN) or alternative secure access method, to connect to MSP infrastructure and limit all network traffic to and from the MSP to that dedicated secure connection. Verify that the networks used for trust relationships with MSPs are suitably segregated from the rest of their networks. Ensure contractual agreements specify that MSPs will not reuse admin credentials across multiple customers.

Apply the principle of least privilege. 

Organizations should apply the principle of least privilege throughout their network environment and immediate update privileges upon changes in administrative roles. Use a tiering model for administrative accounts so that these accounts do not have any unnecessary access or privileges. Only use accounts with full privileges across an enterprise when strictly necessary and consider the use of time-based privileges to further restrict their use. Identify high-risk devices, services and users to minimize their accesses.[14]

• MSPs should apply this principle to both internal and customer environments, avoiding default administrative privileges. 
• Customers should ensure that their MSP applies this principle to both provider and customer network environments. Note: customers with contractual arrangements that provide them with administration of MSP accounts within their environment should ensure that the MSP accounts only have access to the services/resources being managed by the MSP.

Deprecate obsolete accounts and infrastructure. 

Both MSPs and customers should periodically review their internet attack surface and take steps to limit it, such as disabling user accounts when personnel transition.[15] (Note: although sharing accounts is not recommended, should an organization require this, passwords to shared account should be reset when personnel transition.) Organizations should also audit their network infrastructure—paying particular attention to those on the MSP-customer boundary—to identify and disable unused systems and services. Port scanning tools and automated system inventories can assist organizations in confirming the roles and responsibilities of systems.

• Customers should be sure to disable MSP accounts that are no longer managing infrastructure. Note: disabling MSP accounts can be overlooked when a contract terminates.

Apply updates. 

Organizations should update software, including operating systems, applications, and firmware. Prioritize applying security updates to software containing known exploited vulnerabilities. Note: organizations should prioritize patching vulnerabilities included in CISA’s catalogue of known exploited vulnerabilities (KEV) as opposed to only those with high Common Vulnerability Scoring System (CVSS) scores that have not been exploited and may never be exploited.[16],[17],[18],[19]

• MSPs should implement updates on internal networks as quickly as possible.
• Customers should ensure that they understand their MSP’s policy on software updates and request that comprehensive and timely updates are delivered as an ongoing service.

Backup systems and data. 

Organizations should regularly update and test backups—including “gold images” of critical systems in the event these need to be rebuilt (Note: organizations should base the frequency of backups on their recovery point objective [20]). Store backups separately and isolate them from network connections that could enable the spread of ransomware; many ransomware variants attempt to find and encrypt/delete accessible backups. Isolating backups enables restoration of systems/data to their previous state should they be encrypted with ransomware. Note: best practices include storing backups separately, such as on external media.[21],[22],[23] 

• MSPs should regularly backup internal data as well as customer data (where contractually appropriate) and maintain offline backups encrypted with separate, offline encryption keys. Providers should encourage customers to create secure, offsite backups and exercise recovery capabilities.
• Customers should ensure that their contractual arrangements include backup services that meet their resilience and disaster recovery requirements. Specifically, customers should require their MSP to implement a backup solution that automatically and continuously backs up critical data and system configurations and store backups in an easily retrievable location, e.g., a cloud-based solution or a location that is air-gapped from the organizational network.

Develop and exercise incident response and recovery plans. 

Incident response and recovery plans should include roles and responsibilities for all organizational stakeholders, including executives, technical leads, and procurement officers. Organizations should maintain up-to-date hard copies of plans to ensure responders can access them should the network be inaccessible (e.g., due to a ransomware attack).[24]

• MSPs should develop and regularly exercise internal incident response and recovery plans and encourage customers to do the same.
• Customers should ensure that their contractual arrangements include incident response and recovery plans that meet their resilience and disaster recovery requirements. Customers should ensure these plans are tested at regular intervals.

Understand and proactively manage supply chain risk. 

All organizations should proactively manage ICT supply chain risk across security, legal, and procurement groups, using risk assessments to identify and prioritize the allocation of resources.[25],[26]

• MSPs should understand their own supply chain risk and manage the cascading risks it poses to customers.
• Customers should understand the supply chain risk associated with their MSP, including risk associated with third-party vendors or subcontractors. Customers should also set clear network security expectations with their MSPs and understand the access their MSP has to their network and the data it houses. Each customer should ensure their contractual arrangements meet their specific security requirements and that their contract specifies whether the MSP or the customer owns specific responsibilities, such as hardening, detection, and incident response.[27]

Promote transparency. 

Both MSPs and their customers will benefit from contractual arrangements that clearly define responsibilities. 

• MSPs, when negotiating the terms of a contract with their customer, should provide clear explanations of the services the customer is purchasing, services the customer is not purchasing, and all contingencies for incident response and recovery.
• Customers should ensure that they have a thorough understanding of the security services their MSP is providing via the contractual arrangement and address any security requirements that fall outside the scope of the contract. Note: contracts should detail how and when MSPs notify the customer of an incident affecting the customer’s environment.

Manage account authentication and authorization. 

All organizations should adhere to best practices for password and permission management. [28],[29],[30] Organizations should review logs for unexplained failed authentication attempts—failed authentication attempts directly following an account password change could indicate that the account had been compromised. Note: network defenders can proactively search for such “intrusion canaries” by reviewing logs after performing password changes—using off-network communications to inform users of the changes—across all sensitive accounts. (See the ACSC publication, Windows Event Logging and Forwarding as well as Microsoft’s documentation, 4625(F): An account failed to log on, for additional guidance.) 

• MSPs should verify that the customer restricts MSP account access to systems managed by the MSP.
• Customers should ensure MSP accounts are not assigned to internal administrator groups; instead, restrict MSP accounts to systems managed by the MSP. Grant access and administrative permissions on a need-to-know basis, using the principle of least privilege. Verify, via audits, that MSP accounts are being used for appropriate purposes and activities, and that these accounts are disabled when not actively being used. 

Purpose

This advisory was developed by UK, Australian, Canadian, New Zealand, and U.S. cybersecurity authorities in furtherance their respective cybersecurity missions, including their responsibilities to develop and issue cybersecurity specifications and mitigations.

Acknowledgements

The UK, Australian, Canadian, New Zealand, and U.S. cybersecurity authorities would like to thank Secureworks for their contributions to this CSA.

Disclaimer

The information in this report is being provided “as is” for informational purposes only. NCSC-UK, ACSC, CCCS, NCSC-NZ, CISA, NSA, and FBI do not endorse any commercial product or service, including any subjects of analysis. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply endorsement, recommendation, or favouring.

Contact Information

United Kingdom organizations: report a significant cyber security incident: ncsc.gov.uk/report-an-incident (monitored 24 hours) or, for urgent assistance, call 03000 200 973. Australian organizations: visit cyber.gov.au or call 1300 292 371 (1300 CYBER 1) to report cybersecurity incidents and access alerts and advisories. Canadian organizations: report incidents by emailing CCCS at contact@cyber.gc.ca. New Zealand organizations: report cyber security incidents to incidents@ncsc.govt.nz or call 04 498 7654. U.S. organizations: all organizations should report incidents and anomalous activity to CISA 24/7 Operations Center at report@cisa.gov or (888) 282-0870 and/or to the FBI via your local FBI field office or the FBI’s 24/7 CyWatch at (855) 292-3937 or CyWatch@fbi.gov. When available, please include the following information regarding the incident: date, time, and location of the incident; type of activity; number of people affected; type of equipment used for the activity; the name of the submitting company or organization; and a designated point of contact. For NSA client requirements or general cybersecurity inquiries, contact Cybersecurity_Requests@nsa.gov. 

Resources

In addition to the guidance referenced above, see the following resources:

• Joint CSA: Technical Approaches to Uncovering and Remediating Malicious Activity
• Joint CSA: 2021 Trends Show Increased Globalized Threat of Ransomware
• ACSC’s Managed Service Providers: How to manage risk to customer networks 
• CCCS:
• Cyber Security Considerations for Consumers of Managed Services 
• Baseline Cyber Security Controls for Small and Medium Organizations
• Top 10 IT Security Action Items to Protect Internet Connected Networks and Information
• CCCS’s Alert: Malicious Cyber Activity Targeting Managed Service Providers 
• CISA:
  • CISA Cybersecurity Alert: APT Activity Exploiting MSPs (2018)
  • CISA Cyber Essentials and CISA Cyber Resource Hub 
• FBI Internet Crime Complaint Center alerts on malicious and criminal cyber activity
• National Institute of Standards and Technology (NIST) National Cybersecurity Center of Excellence (NCCoE): Improving Cybersecurity of Managed Service Providers 

References

[1] State of the Market: The New Threat Landscape, Pushing MSP security to the next level (N-able) 
[2] Global targeting of enterprises via managed service providers (NCSC-UK)
[3] Guidance for MSPs and Small- and Mid-sized Businesses (CISA)
[4] Kaseya Ransomware Attack: Guidance for Affected MSPs and their Customers (CISA) 
[5] APTs Targeting IT Service Provider Customers (CISA)
[6] MSP Investigation Report (ACSC)
[7] How to Manage Your Security When Engaging a Managed Service Provider
[8] Supply Chain Cyber Security: In Safe Hands (NCSC-NZ)
[9] Multi-factor authentication for online services (NCSC-UK)
[10] Zero trust architecture design principles: MFA (NCSC-UK)
[11] Joint CISA-FBI CSA: Russian State-Sponsored Cyber Actors Gain Network Access by Exploiting Default MFA Protocols and “PrintNightmare” Vulnerability
[12] Security architecture anti-patterns (NCSC-UK)
[13] Preventing Lateral Movement (NCSC-UK)
[14] Preventing Lateral Movement: Apply the principle of least privilege (NCSC-UK)
[15] Device Security Guidance: Obsolete products (NCSC-UK)
[16] Known Exploited Vulnerabilities Catalog (CISA)
[17] The problems with patching (NCSC-UK)
[18] Security principles for cross domain solutions: Patching (NCSC-UK)
[19] Joint CSA: 2021 Top Routinely Exploited Vulnerabilities
[20] Protecting Data from Ransomware and Other Data Loss Events: A Guide for Managed Service Providers to Conduct, Maintain, and Test Backup Files (NIST)
[21] Stop Ransomware website (CISA)
[22] Offline backups in an online world (NCSC-UK)
[23] Mitigating malware and ransomware attacks (NCSC-UK)
[24] Effective steps to cyber exercise creation (NCSC-UK)
[25] Supply chain security guidance (NCSC-UK)
[26] ICT Supply Chain Resource Library (CISA)
[27] Risk Considerations for Managed Service Provider Customers (CISA)
[28] Device Security Guidance: Enterprise authentication policy (NCSC-UK)
[29] Preventing Lateral Movement: Apply the principle of least privilege (NCSC-UK)
[30] Implementing Strong Authentication (CISA)

Appendix

This advisory’s definition of MSPs aligns with the following definitions.

The definition of MSP from Gartner’s Information Technology Glossary—which is also referenced by NIST in Improving Cybersecurity of Managed Service Providers—is:

A managed service provider (MSP) delivers services, such as network, application, infrastructure and security, via ongoing and regular support and active administration on customers’ premises, in their MSP’s data center (hosting), or in a third-party data center.

MSPs may deliver their own native services in conjunction with other providers’ services (for example, a security MSP providing sys admin on top of a third-party cloud IaaS). Pure-play MSPs focus on one vendor or technology, usually their own core offerings. Many MSPs include services from other types of providers. The term MSP traditionally was applied to infrastructure or device-centric types of services but has expanded to include any continuous, regular management, maintenance and support.

The United Kingdom’s Department of Digital, Culture, Media, and Sport (DCMS) recently published the following definition of MSP, which includes examples: 

Managed Service Provider – A supplier that delivers a portfolio of IT services to business customers via ongoing support and active administration, all of which are typically underpinned by a Service Level Agreement. A Managed Service Provider may provide their own Managed Services or offer their own services in conjunction with other IT providers’ services. The Managed Services might include:

• Cloud computing services (resale of cloud services, or an in-house public and private cloud services, built and provided by the Managed Service Providers)
• Workplace services
• Managed Network
• Consulting
• Security services
• Outsourcing
• Service Integration and Management
• Software Resale
• Software Engineering
• Analytics and Artificial Intelligence (AI)
• Business Continuity and Disaster Recovery services

The Managed Services might be delivered from customer premises, from customer data centres, from Managed Service Providers’ own data centres or from 3rd party facilities (co-location facilities, public cloud data centres or network Points of Presence (PoPs)).

Revisions

• May 11, 2022: Initial version