Use AWS Transit Gateway & Direct Connect to Centralize and Streamline Your Network Connectivity

This post was originally published on this site

Last year I showed you how to Use an AWS Transit Gateway to Simplify Your Network Architecture. As I said at the time:

You can connect your existing VPCs, data centers, remote offices, and remote gateways to a managed Transit Gateway, with full control over network routing and security, even if your VPCs, Active Directories, shared services, and other resources span multiple AWS accounts. You can simplify your overall network architecture, reduce operational overhead, and gain the ability to centrally manage crucial aspects of your external connectivity, including security. Last but not least, you can use Transit Gateways to consolidate your existing edge connectivity and route it through a single ingress/egress point.

In that post I also promised you support for AWS Direct Connect, and I’m happy to announce that this support is available today for use in the US East (N. Virginia), US East (Ohio), US West (N. California), and US West (Oregon) Regions. The applications that you run in the AWS Cloud can now communicate with each other, and with your on-premises applications, at speeds of up to 10 Gbps per Direct Connect connection. You can set it up in minutes (assuming that you already have a private or hosted connection running at 1 Gbps or more) and start using it right away.

Putting it all together, you get a lot of important benefits from today’s launch:

Simplification – You can simplify your network architecture and your network management overhead by creating a hub-and-spoke model that spans multiple VPCs, regions, and AWS accounts. If you go this route, you may also be in a position to cut down on the number of AWS VPN connections that you use.

Consolidation – You have the opportunity to reduce the number of private or hosted connections, saving money and avoiding complexity in the process. You can consolidate your connectivity so that it all flows across the same BGP session.

Connectivity – You can reach your Transit Gateway using your connections from any of the 90+ AWS Direct Connect locations (except from AWS Direct Connect locations in China).

Using Transit Gateway & Direct Connect
I will use the freshly updated Direct Connect Console to set up my Transit Gateway for use with Direct Connect. The menu on the left lets me view and create the resources that I will need:

My AWS account already has access to a 1 Gbps connection (MyConnection) to TierPoint in Seattle:

I create a Direct Connect Gateway (MyDCGateway):

I create a Virtual Interface (VIF) with type Transit:

I reference my Direct Connect connection (MyConnection) and my Direct Connect Gateway (MyDCGateway) and click Create virtual interface:

When the state of my new VIF switches from pending to down I am ready to proceed:

Now I am ready to create my transit gateway (MyTransitGW). This is a VPC component; clicking on Transit gateways takes me to the VPC console. I enter a name, description, and ASN (which must be distinct from the one that I used for the Direct Connect Gateway), leave the other values as-is, and click Create Transit Gateway:

The state starts out as pending, and transitions to available:

With all of the resources ready, I am ready to connect them! I return to the Direct Connect Console, find my Transit Gateway, and click Associate Direct Connect gateway:

I associate the Transit Gateway with a Direct Connect Gateway in my account (using another account requires the ID of the gateway and the corresponding AWS account number), and list the network prefixes that I want to advertise to the other side of the Direct Connect connection. Then I click Associate Direct Connect gateway to make it so:

The state starts out as associating and transitions to associated. This can take some time, so I will take Luna for a walk:

By the time we return, the Direct Connect Gateway is associated with the Transit Gateway, and we are good to go!

In a real-world situation you would spend more time planning your network topology and addressing, and you would probably use multiple AWS accounts.

Available Now
You can use this new feature today to interface with your Transit Gateways hosted in four AWS regions.

Jeff;

New – Amazon Managed Blockchain – Create & Manage Scalable Blockchain Networks

This post was originally published on this site

Trust is a wonderful thing, and is the basis for almost every business and personal relationship or transaction. In some cases, trust is built up over an extended period of time, reinforced with each successful transaction and seen as an integral part of the relationship. In other situations, there’s no time to accumulate trust and other mechanisms must be used instead. The parties must find a way to successfully complete the transaction in the absence of trust. Today, emerging blockchain technologies such as Hyperledger Fabric and Ethereum fill this important need, allowing parties to come to consensus regarding the validity of a proposed transaction and create an unalterable digital record (commonly known as a ledger) of each transaction in the absence of trust.

Amazon Managed Blockchain
We announced Amazon Managed Blockchain at AWS re:Invent 2018 and invited you to sign up for a preview. I am happy to announce that the preview is complete and that Amazon Managed Blockchain is now available for production use in the US East (N. Virginia) Region. You can use it to create scalable blockchain networks that use the Hyperledger Fabric open source framework, with Ethereum in the works. As you will see in a minute, you can create your network in minutes. Once created, you can easily manage and maintain your blockchain network. You can manage certificates, invite new members, and scale out peer node capacity in order to process transactions more quickly.

The blockchain networks that you create with Amazon Managed Blockchain can span multiple AWS accounts so that a group of members can execute transactions and share data without a central authority. New members can easily launch and configure peer nodes that process transaction requests and store a copy of the ledger.

Using Amazon Managed Blockchain
I can create my own scalable blockchain network from the AWS Management Console, AWS Command Line Interface (CLI) (aws managedblockchain create-network), or API (CreateNetwork). To get started, I open the Amazon Managed Blockchain Console and click Create a network:

I need to choose the edition (Starter or Standard) for my network. The Starter Edition is designed for test networks and small production networks, with a maximum of 5 members per network and 2 peer nodes per member. The Standard Edition is designed for scalable production use, with up to 14 members per network and 3 peer nodes per member (check out the Amazon Managed Blockchain Pricing to learn more about both editions). I also enter a name and a description for my network:

Then I establish the voting policy for my network, and click Next to move ahead (read Work with Proposals to learn more about creating and voting on proposals):

Now, I need to create the first member of my network. Each member is a distinct identity within the network, and is visible within the network. I also set up a user name and password for my certificate authority, and click Next:

I review my choices, and click Create network and member:

My network enters the Creating status, and I take a quick break to walk my dog! When I return, my network is Available:

Inviting Members
Now that my network is available, I can invite members by clicking the Members tab:

I can see the current members of my network, both those I own and those owned by others. I click on Propose invitation to invite a new member:

Then I enter the AWS account number of the proposed member and click Create:

This creates a proposal (visible to me and to the other members of the network). I click on the ID to proceed:

I review the proposal, select my identity (block-wizard), and then click Yes to vote:

After enough Yes votes have been received to pass the threshold that I specified when I created the network, the invitation will be extended to the new member, and will be visible in the Invitations section:

If you are building a blockchain network for testing purposes and don’t have access to multiple AWS accounts, you can even invite your own account. After you do this (and vote to let yourself in), you will end up with multiple members in the same account.

Using the Network
Now that the network is running, and has some members, the next step is to create an endpoint in the Virtual Private Cloud (VPC) where I will run my blockchain applications (this feature is powered by AWS PrivateLink). Starting from the detail page for my network, I click Create VPC endpoint:

I choose the desired VPC and the subnets within it, pick a security group, and click Create:

My applications can use the VPC endpoint to communicate with my blockchain network:

The next step is to build applications that make use of the blockchain. To learn how to do this, read Build and deploy an application for Hyperledger Fabric on Amazon Managed Blockchain. You can also read Get Started Creating a Hyperledger Fabric Blockchain Network Using Amazon Managed Blockchain.

Things to Know
As usual, we have a healthy roadmap for this new service. Stay tuned to learn more!

Jeff;

PS – Check out the AWS Blockchain Pub to see a novel use for Amazon Managed Blockchain and AWS DeepLens.

 

WARNING: LRO: 977: cannot aggr pkt from port 0x5000002 as lro session port is 0x5000004

This post was originally published on this site

We have started receiving the warning: “WARNING: LRO: 977: cannot aggr pkt from port 0x5000002 as lro session port is 0x5000004″.  The esx hosts are running 6.5 build 10884925.  I searched through VMware’s knowledge base, without success.  AS of yet I do not see any indications of a problem.  Any information around the error would greatly be appreciated.  We are using a nimble array with HP Proliant DL 380 G10, usiing

 

 

Thanks,

ShineKnox

The AWS DeepRacer League Virtual Circuit is Now Open – Train Your Model Today!

This post was originally published on this site

AWS DeepRacer is a 1/18th scale four-wheel drive car with a considerable amount of onboard hardware and software. Starting at re:Invent 2018 and continuing with the AWS Global Summits, you have the opportunity to get hands-on experience with a DeepRacer. At these events, you can train a model using reinforcement learning, and then race it around a track. The fastest racers and their laptimes for each summit are shown on our leaderboards.

New DeepRacer League Virtual Circuit
Today we are launching the AWS DeepRacer League Virtual Circuit. You can build, train, and evaluate your reinforcement learning models online and compete online for some amazing prizes, all from the comfort of the DeepRacer Console!

We’ll add a new track each month, taking inspiration from famous race tracks around the globe, so that you can refine your models and broaden your skill set. The top entrant in the leaderboard each month will win an expenses-paid package to AWS re:Invent 2019, where they will take part in the League Knockout Rounds, with a chance to win the Championship Cup!

New DeepRacer Console
We are making the DeepRacer Console available today in the US East (N. Virginia) Region. You can use it to build and train your DeepRacer models and to compete in the Virtual Circuit, while gaining practical, hands-on experience with Reinforcement Learning. Following the steps in the DeepRacer Lab that is used at the hands-on DeepRacer workshops, I open the console and click Get started:

The console provides me with an overview of the model training process, and then asks to create the AWS resources needed to train and evaluate my models. I review the info and click Create resources to proceed:

The resources are created in minutes (I can click Learn RL to learn more about reinforcement learning while this is happening). I click Create model to move ahead:

I enter a name and a description for for my model:

Then I pick a track (more tracks will appear throughout the duration of the Virtual League):

Now I define the Action space for my model. This is a set of discrete actions that my model can perform. Choosing values that increase the number of options will generally enhance the quality of my model, at the cost of additional training time:

Next, I define the reward function for my model. This function evaluates the current state of the vehicle throughout the training process and returns a reward value to indicate how well the model is performing (higher rewards signify better performance). I can use one of three predefined models (available by clicking Reward function examples) as-is, customize them, or build one from scratch. I’ll use Prevent zig-zag, a sample reward function that penalizes zig-zap behavior, to get started:

The reward function is written in Python 3, and has access to parameters (track_width, distance_from_center, all_wheels_on_track, and many more) that describe the position and state of the car, and also provide information about the track.

I also control a set of hyperparameters that affect the overall training performance. Since I don’t understand any of these (just being honest here), I will accept all of the defaults:

To learn more about hyperparameters, read Systematically Tune Hyperparameters.

Finally, I specify a time limit for my training job, and click Start training. In general, simple models will converge in 90 to 120 minutes, but this is highly dependent on the maximum speed and the reward function.

The training job is initialized (this takes about 6 minutes), and I can track progress in the console:

The training job makes use of AWS RoboMaker so I can also monitor it from the RoboMaker Console. For example, I can open the Gazebo window, see my car, and watch the training process in real time:

One note of caution: changing the training environment (by directly manipulating Gazebo) will adversely affect the training run, most likely rendering it useless.

As the training progresses, the Reward graph will go up and to the right (as we often say at Amazon) if the car is learning how to stay on the track:

If the graph flattens out or declines precipitously and stays there, your reward function is not rewarding the desired behavior or some other setting is getting in the way. However, patience is a virtue, and there will be the occasional regression on the way to the top. After the training is complete, there’s a short pause while the new model is finalized and stored, and then it is time for me to evaluate my model by running it in a simulation. I click Start evaluation to do this:

I can evaluate the model on any of the available tracks. Using one track for training and a different one for evaluation is a good way to make sure that the model is general, and has not been overfit so that it works on just one track. However, using the same track for training and testing is a good way to get started, and that’s what I will do. I select the Oval Track and 3 trials, and click Start evaluation:

The RoboMaker simulator launches, with an hourly cost for the evaluation, as noted above. The results (lap times) are displayed when the simulation is complete:

At this point I can evaluate my model on another track, step back and refine my model and evaluate it again, or submit my model to the current month’s Virtual Circuit track to take part in the DeepRacer League. To do that, I click Submit to virtual race, enter my racer name, choose a model, agree to the Ts and C’s, and click Submit model:

After I submit, my model will be evaluated on the pre-season track and my lap time will be used to place me in the Virtual Circuit Leaderboard.

Things to Know
Here are a couple of things to know about the AWS DeepRacer and the AWS DeepRacer League:

AWS ResourcesAmazon SageMaker is used to train models, which are then stored in Amazon Simple Storage Service (S3). AWS RoboMaker provides the virtual track environment, which is used for training and evaluation. An AWS CloudFormation stack is used to create a Amazon Virtual Private Cloud, complete with subnets, routing tables, an Elastic IP Address, and a NAT Gateway.

Costs – You can use the DeepRacer console at no charge. As soon as you start training your first model, you will get service credits for SageMaker and RoboMaker to give you 10 hours of free training on these services. The credits are applied at the end of the month and are available for 30 days, as part of the AWS Free Tier. The DeepRacer architecture uses a NAT Gateway that carries an availability charge. Your account will automatically receive service credits to offset this charge, showing net zero on your account.

DeepRacer Cars – You can preorder your DeepRacer car now! Deliveries to addresses in the United States will begin in July 2019.

Jeff;

Now Available – Elastic Fabric Adapter (EFA) for Tightly-Coupled HPC Workloads

This post was originally published on this site

We announced Elastic Fabric Adapter (EFA) at re:Invent 2018 and made it available in preview form at the time. During the preview, AWS customers put EFA through its paces on a variety of tightly-coupled HPC workloads, providing us with valuable feedback and helping us to fine-tune the final product.

Now Available
Today I am happy to announce that EFA is now ready for production use in multiple AWS regions. It is ready to support demanding HPC workloads that need lower and more consistent network latency, along with higher throughput, than is possible with traditional TCP communication. This launch lets you apply the scale, flexibility, and elasticity of the AWS Cloud to tightly-coupled HPC apps and I can’t wait to hear what you do with it. You can, for example, scale up to thousands of compute nodes without having to reserve the hardware or the network ahead of time.

All About EFA
An Elastic Fabric Adapter is an AWS Elastic Network Adapter (ENA) with added capabilities (read my post, Elastic Network Adapter – High Performance Network Interface for Amazon EC2, to learn more about ENA). An EFA can still handle IP traffic, but also supports an important access model commonly called OS bypass. This model allows the application (most commonly through some user-space middleware) access the network interface without having to get the operating system involved with each message. Doing so reduces overhead and allows the application to run more efficiently. Here’s what this looks like (source):

The MPI Implementation and libfabric layers of this cake play crucial roles:

MPI – Short for Message Passing Interface, MPI is a long-established communication protocol that is designed to support parallel programming. It provides functions that allow processes running on a tightly-coupled set of computers to communicate in a language-independent way.

libfabric – This library fits in between several different types of network fabric providers (including EFA) and higher-level libraries such as MPI. EFA supports the standard RDM (reliable datagram) and DGRM (unreliable datagram) endpoint types; to learn more, check out the libfabric Programmer’s Manual. EFA also supports a new protocol that we call Scalable Reliable Datagram; this protocol was designed to work within the AWS network and is implemented as part of our Nitro chip.

Working together, these two layers (and others that can be slotted in instead of MPI), allow you to bring your existing HPC code to AWS and run it with little or no change.

You can use EFA today on c5n.18xlarge and p3dn.24xlarge instances in all AWS regions where those instances are available. The instances can use EFA to communicate within a VPC subnet, and the security group must have ingress and egress rules that allow all traffic within the security group to flow. Each instance can have a single EFA, which can be attached when an instance is started or while it is stopped.

You will also need the following software components:

EFA Kernel Module – The EFA Driver is in the Amazon GitHub repo, and in the Amazon Linux & Amazon Linux 2 AMIs. We are working to add it to AMIs for other Linux distributions.

Libfabric Network Stack – You will need to use an AWS-custom version (already present in the Amazon Linux and Amazon Linux 2 AMIs) for now. We are working to get our changes into the next release (1.8) of libfabric.

MPI or NCCL Implementation – You can use Open MPI 3.1.3 (or later) or NCCL (2.3.8 or later) plus the OFI driver for NCCL. We also also working on support for the Intel MPI library.

You can launch an instance and attach an EFA using the CLI, API, or the EC2 Console, with CloudFormation support coming in a couple of weeks. If you are using the CLI, you need to include the subnet ID and ask for an EFA, like this (be sure to include the appropriate security group):

$ aws ec2 run-instances ... 
  --network-interfaces DeleteOnTermination=true,DeviceIndex=0,SubnetId=SUBNET,InterfaceType=efa

After your instance has launched, run lspci | grep efa0 to verify that the EFA device is attached. You can (but don’t have to) launch your instances in a Cluster Placement Group in order to benefit from physical adjacency when every light-foot counts. When used in this way, EFA can provide one-way MPI latency of 15.5 microseconds.

You can also create a Launch Template and use it to launch EC2 instances (either directly or as part of an EC2 Auto Scaling Group) in On-Demand or Spot Form, launch Spot Fleets, and to run compute jobs on AWS Batch.

Learn More
To learn more about EFA, and to see some additional benchmarks, be sure to watch this re:Invent video (Scaling HPC Applications on EC2 w/ Elastic Fabric Adapter):

 

 

AWS Customer CFD Direct maintains the popular OpenFOAM platform for Computational Fluid Dynamics (CFD) and also produces CFD Direct From the Cloud (CFDDC), an AWS Marketplace offering that makes it easy for you to run OpenFOAM on AWS. They have been testing and benchmarking EFA and recently shared their measurements in a blog post titled OpenFOAM HPC with AWS EFA. In the post, they report on a pair of simulations:

External Aerodynamics Around a Car – This simulation scales extra-linearly to over 200 cores, gradually declining to linear scaling at 1000 cores (about 100K simulation cells per core).

Flow Over a Weir with Hydraulic Jump – This simulation (1000 cores and 100M cells) scales at between 67% and 72.6%, depending on a “data write” setting.

Read the full post to learn more and to see some graphs and visualizations.

In the Works
We plan to add EFA support to additional EC2 instance types over time. In general, we plan to provide EFA support for the two largest sizes of “n” instances of any given type, and also for bare metal instances.

Jeff;

 

Now Open – AWS Asia Pacific (Hong Kong) Region

This post was originally published on this site

The AWS Region in Hong Kong SAR is now open and you can start using it today. The official name is Asia Pacific (Hong Kong) and the API name is ap-east-1. The AWS Asia Pacific (Hong Kong) Region is the eighth active AWS Region in Asia Pacific and mainland China along with Beijing, Mumbai, Ningxia, Seoul, Singapore, Sydney, and, Tokyo. With this launch, AWS now spans 64 Availability Zones within 21 geographic regions around the world. We have also announced plans for 12 more Availability Zones and four more AWS Regions in Bahrain, Cape Town, Jakarta, and Milan.

Instances and Services
Applications running in this 3-AZ region can use C5, C5d, D2, I3, M5, M5d, R5, R5d, and T3 instances, and can make use of a long list of AWS services including Amazon API Gateway, Application Auto Scaling, AWS Certificate Manager (ACM), AWS Artifact, AWS CloudFormation, Amazon CloudFront, AWS CloudTrail, Amazon CloudWatch, CloudWatch Events, Amazon CloudWatch Logs, AWS CodeDeploy, AWS Config, AWS Config Rules, AWS Database Migration Service, AWS Direct Connect, Amazon DynamoDB, EC2 Auto Scaling, EC2 Dedicated Hosts, Amazon Elastic Block Store (EBS), Amazon Elastic Compute Cloud (EC2), Elastic Container Registry, Amazon ECS, Application Load Balancers (Classic, Network, and Application), Amazon EMR, Amazon ElastiCache, Amazon Elasticsearch Service, Amazon Glacier, AWS Identity and Access Management (IAM), Amazon Kinesis Data Streams, AWS Key Management Service (KMS), AWS Lambda, AWS Marketplace, AWS Organizations, AWS Personal Health Dashboard, AWS Resource Groups, Amazon Redshift, Amazon Relational Database Service (RDS), Amazon Aurora, Amazon Route 53 (including Private DNS for VPCs), AWS Shield, AWS Server Migration Service, AWS Snowball, AWS Snowball Edge, Amazon Simple Notification Service (SNS), Amazon Simple Queue Service (SQS), Amazon Simple Storage Service (S3), Amazon Simple Workflow Service (SWF), AWS Step Functions, AWS Support API, Amazon EC2 Systems Manager (SSM), AWS Trusted Advisor, Amazon Virtual Private Cloud, and VM Import/Export.

AWS Elastic Beanstalk, Amazon Elastic Container Service for Kubernetes, and AWS X-Ray are scheduled for deployment next month, with other services to follow. We are also working to enable cross-region delivery from SNS topics hosted in other regions to SQS queues hosted in the new region.

Using the Asia Pacific (Hong Kong) Region
As we announced last month, you need to explicitly enable this region for your AWS account in order to be able to create and manage resources within it. Enabling or disabling a region requires the account:EnableRegion, account:DisableRegion, and account:ListRegions permissions. Here’s a sample IAM policy that grants these permissions for the new region:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "aws-portal:ViewAccount",
            "account:ListRegions"
         ],
         "Resource":"*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "account:EnableRegion",
            "account:DisableRegion"
         ],
         "Resource":"*",
         "Condition":{
            "StringEquals":{
               "account:TargetRegion":"ap-east-1"
            }
         }
      }
   ]
}

Log in to the AWS Management Console as a user that has these appropriate permissions and click My Account:

Scroll down to the AWS Regions section, find the new region, and click Enable:

Then confirm your action by clicking Enable region:

The region is enabled immediately, and will be ready for use shortly thereafter.

You can also enable the region by selecting it from the menu:

And then confirming your action:

Connectivity, Edge Locations, and Latency
Hong Kong SAR is already home to three Amazon CloudFront edge locations (the first one opened way back in 2008). There are also more than thirty other edge locations and eleven regional edge caches in Asia; see the Amazon CloudFront Infrastructure page for a full list.

The region offers low-latency connections to other cities and AWS regions in the area. Here are the latest numbers:

There are now two Hong Kong AWS Direct Connect locations: the existing one at iAdvantage Mega-I and a new one at Equinix HK1. Both locations have direct connectivity to the Asia Pacific (Hong Kong) Region. If you already connect to AWS at iAdvantage, you can use your existing Direct Connect connection to access the new region via Direct Connect Gateway.

Investing in the Future
Before I wrap up I would like to tell you about some of work that we are doing to support startups and to educate developers:

AWS Activate – This global program provides startups with credits, training, and support so that they can build their businesses on AWS.

AWS Educate – This global program teaches students about cloud computing. It provides AWS credits to educators and students, along with discounts on training, access to curated content, personalized learning pathways, and collaboration tools. Dozens of Hong Kong universities and business schools are already participating.

AWS Academy – This global program is designed to bridge the gap between academia and industry by giving students the knowledge that they need to have in order to qualify for jobs that require cloud skills. The program is built around hands-on experience, and includes an AWS-authored curriculum, access to AWS certification, accreditation for educators.

Training and Certification – This global program helps developers to build cloud skills using digital or classroom training and to validate those skills by earning an industry-recognized credential. It includes learning paths for Cloud Practitioners, Architects, Developers, and Operations.

Jeff;

 

Now Available – AMD EPYC-Powered Amazon EC2 T3a Instances

This post was originally published on this site

The AMD EPYC-powered T3a instances that I promised you last year are available now and you can start using them today! Like the recently announced M5ad and R5ad instances, the T3a instances are built on the AWS Nitro System and give you an opportunity to balance your instance mix based on cost and performance.

T3a Instances
These instances deliver burstable, cost-effective performance and are a great fit for workloads that do not need high sustained compute power but experience temporary spikes in usage. You get a generous and assured baseline amount of processing power and the ability to transparently scale up to full core performance when you need more processing power, for as long as necessary. To learn more about the burstable compute model common to the T3 and the T3a, read New T3 Instances – Burstable, Cost-Effective Performance.

You can launch T3a instances today in seven sizes in the US East (N. Virginia), US West (Oregon), Europe (Ireland), US East (Ohio), and Asia Pacific (Singapore) Regions in On-Demand, Spot, and Reserved Instance form. Here are the specs:

Instance Name vCPUs RAM EBS-Optimized Bandwidth Network Bandwidth
t3a.nano
2 0.5 GiB Up to 1.5 Gbps Up to 5 Gbps
t3a.micro
2 1 GiB Up to 1.5 Gbps Up to 5 Gbps
t3a.small
2 2 GiB Up to 1.5 Gbps Up to 5 Gbps
t3a.medium
2 4 GiB Up to 1.5 Gbps Up to 5 Gbps
t3a.large
2 8 GiB Up to 2.1 Gbps Up to 5 Gbps
t3a.xlarge
4 16 GiB Up to 2.1 Gbps Up to 5 Gbps
t3a.2xlarge
8 32 GiB Up to 2.1 Gbps Up to 5 Gbps

The T3 and the T3a instances are available in the same sizes and can use the same AMIs, making it easy for you to try both and find the one that is the best match for you application.

Pricing is 10% lower than the equivalent existing T3 instances; see the On-Demand, Spot, and Reserved Instance pricing pages for more info.

Jeff;

Amazon SageMaker Ground Truth keeps simplifying labeling workflows

This post was originally published on this site

Launched at AWS re:Invent 2018, Amazon SageMaker Ground Truth is a capability of Amazon SageMaker that makes it easy for customers to efficiently and accurately label the datasets required for training machine learning systems.

A quick recap on Amazon SageMaker Ground Truth

Amazon SageMaker Ground Truth helps you build highly accurate training datasets for machine learning quickly. SageMaker Ground Truth offers easy access to public and private human labelers and provides them with built-in workflows and interfaces for common labeling tasks. Additionally, SageMaker Ground Truth can lower your labeling costs by up to 70% using automatic labeling, which works by training Ground Truth from data labeled by humans so that the service learns to label data independently.

Amazon SageMaker Ground Truth helps you build datasets for:

  • Text classification.
  • Image classification, i.e categorizing images in specific classes.
  • Object detection, i.e. locating objects in images with bounding boxes.
  • Semantic segmentation, i.e. locating objects in images with pixel-level precision.
  • Custom user-defined tasks, that let customers annotate literally anything.

You can choose to use your team of labelers and route labeling requests directly to them. Alternatively, if you need to scale up, options are provided directly in the Amazon SageMaker Ground Truth console to work with labelers outside of your organization. You can access a public workforce of over 500,000 labelers via integration with Amazon Mechanical Turk. Alternatively, if your data requires confidentiality or special skills, you can use professional labeling companies pre-screened by Amazon, and listed on the AWS Marketplace.

Announcing new features

Since the service was launched, we gathered plenty of customer feedback (keep it coming!), from companies such as T-Mobile, Pinterest, Change Healthcare, GumGum, Automagi and many more. We used it to define what the next iteration of the service would look like, and just a few weeks ago, we launched two highly requested features:

  • Multi-category bounding boxes, allowing you to label multiple categories within an image simultaneously.
  • Three new UI templates for your custom workflows, for a total of fifteen different templates that help you quickly build annotation workflows for images, text, and audio datasets.

Today, we’re happy to announce another set of new features that keep simplifying the process of building and running cost-effective labeling workflows. Let’s look at each one of them.

Job chaining

Customers often want to run a subsequent labeling job using the output of a previous labeling job. Basically, they want to chain together labeling jobs using the outputted labeled dataset (and outputted ML model if automated data labeling was enabled). For example, they may run an initial job where they identify if humans exist in an image, and then they may want to run a subsequent job where they get bounding boxes drawn around the humans.

If active learning was used, customers may also want to use the ML model that was produced in order to bootstrap automated data labeling in a subsequent job. Setup couldn’t be easier: you can chain labeling jobs with just one click!

Job tracking

Customers want to be able to see the status of the progress of their labeling jobs. We now provide near real-time status for labeling jobs.

Long-lived jobs

Many customers use experts as labelers, and these individuals perform labeling on a periodic basis. For example, healthcare companies often use clinicians as their expert labelers, and they can only perform labeling occasionally during downtime. In these scenarios, labeling jobs need to run longer, sometimes for weeks or months. We now support extended task timeout windows where each batch of a labeling job can run for 10 days, meaning labeling jobs can extend for months.

Dynamic custom workflows

When setting up custom workflows, customers want to insert or use additional context in addition to the source data. For example, a customer may want to display the specific weather conditions above each image in the tasks they send to labelers; this information can help labelers better perform the task at-hand. Specifically, this feature allows customers to inject output from previous labeling jobs or other custom content into the custom workflow. This information is passed into a pre-processing Lambda function using the augmented manifest file that includes the source data and additional context. The customer can also use the additional context to dynamically adjust the workflow.

New service providers and new languages

We are listing two new data labeling service providers onto the AWS Marketplace: Vivetic and SmartOne. With the addition of these two vendors, Amazon SageMaker Ground Truth will add support for data labeling in French, German, and Spanish.

Regional expansion

In addition to US-East (Virginia), US-Central (Ohio), US-West (Oregon), Europe (Ireland), and Asia Pacific (Tokyo), Amazon SageMaker Ground Truth is now available in Asia Pacific (Sydney).

Customer case study: ZipRecruiter

ZipRecruiter is helping people find great jobs, and helping employers build great companies. They’ve been using Amazon SageMaker since launch. Says ZipRecruiter CTO Craig Ogg: “ZipRecruiter’s AI-powered algorithm learns what each employer is looking for and provides a personalized, curated set of highly relevant candidates. On the other side of the marketplace, the company’s technology matches job seekers with the most pertinent jobs. And to do all that efficiently, we needed a Machine Learning model to extract relevant data automatically from uploaded resumes”.

Of course, building datasets is a critical part of the machine learning process, and it’s often expensive and extremely time-consuming. To solve both problems, ZipRecruiter turned to Ground Truth and one of our labeling partners, iMerit.

As Craig puts it: “Amazon SageMaker Ground Truth will significantly help us reduce the time and effort required to create datasets for training. Due to the confidential nature of the data, we initially considered using one of our teams but it would take time away from their regular tasks and it would take months to collect the data we needed. Using Amazon SageMaker Ground Truth, we engaged iMerit, a professional labeling company that has been pre-screened by Amazon, to assist with the custom annotation project. With their assistance we were able to collect thousands of annotations in a fraction of the time it would have taken using our own team.”

Getting started

I hope that this post was informative, and that the new features will let you build even faster. Please try Amazon SageMaker Ground Truth, let us know what you think, and help us build the next iteration of this cool service!

Julien

New – Query for AWS Regions, Endpoints, and More Using AWS Systems Manager Parameter Store

This post was originally published on this site

In response to requests from AWS customers, I have been asking our service teams to find ways to make information about our regions and services available programmatically. Today I am happy to announce that this information is available in the AWS Systems Manager Parameter Store, and that you can easily access it from your scripts and your code. You can get a full list of active regions, find out which services are available with them, and much more.

Running Queries
I’ll use the AWS Command Line Interface (CLI) for most of my examples; you can also use the AWS Tools for Windows PowerShell or any of the AWS SDKs. As is the case with all of the CLI commands, you can request output in JSON, tab-delimited text, or table format. I’ll use JSON, and will make liberal use of the jq utility to show the more relevant part of the output from each query.

Here’s how to query for the list of active regions:

$ aws ssm get-parameters-by-path 
  --path /aws/service/global-infrastructure/regions --output json | 
  jq .Parameters[].Name
"/aws/service/global-infrastructure/regions/ap-northeast-1"
"/aws/service/global-infrastructure/regions/eu-central-1"
"/aws/service/global-infrastructure/regions/eu-north-1"
"/aws/service/global-infrastructure/regions/eu-west-1"
"/aws/service/global-infrastructure/regions/eu-west-3"
"/aws/service/global-infrastructure/regions/sa-east-1"
"/aws/service/global-infrastructure/regions/us-east-2"
"/aws/service/global-infrastructure/regions/us-gov-east-1"
"/aws/service/global-infrastructure/regions/us-gov-west-1"
"/aws/service/global-infrastructure/regions/us-west-1"
"/aws/service/global-infrastructure/regions/ap-northeast-2"
"/aws/service/global-infrastructure/regions/ap-northeast-3"
"/aws/service/global-infrastructure/regions/ap-south-1"
"/aws/service/global-infrastructure/regions/ap-southeast-1"
"/aws/service/global-infrastructure/regions/ap-southeast-2"
"/aws/service/global-infrastructure/regions/ca-central-1"
"/aws/service/global-infrastructure/regions/cn-north-1"
"/aws/service/global-infrastructure/regions/cn-northwest-1"
"/aws/service/global-infrastructure/regions/eu-west-2"
"/aws/service/global-infrastructure/regions/us-west-2"
"/aws/service/global-infrastructure/regions/us-east-1"

Here’s how to display a complete list of all available AWS services, sort them into alphabetical order, and display the first 10 (out of 155, as I write this):

$ aws ssm get-parameters-by-path 
  --path /aws/service/global-infrastructure/services --output json | 
  jq .Parameters[].Name | sort | head -10
"/aws/service/global-infrastructure/services/acm"
"/aws/service/global-infrastructure/services/acm-pca"
"/aws/service/global-infrastructure/services/alexaforbusiness"
"/aws/service/global-infrastructure/services/apigateway"
"/aws/service/global-infrastructure/services/application-autoscaling"
"/aws/service/global-infrastructure/services/appmesh"
"/aws/service/global-infrastructure/services/appstream"
"/aws/service/global-infrastructure/services/appsync"
"/aws/service/global-infrastructure/services/athena"
"/aws/service/global-infrastructure/services/autoscaling"

Here’s how to get the list of services that are available in a given region (again, first 10, sorted):

$ aws ssm get-parameters-by-path 
  --path /aws/service/global-infrastructure/regions/us-east-1/services --output json | 
  jq .Parameters[].Name | sort | head -10
"/aws/service/global-infrastructure/regions/us-east-1/services/acm"
"/aws/service/global-infrastructure/regions/us-east-1/services/acm-pca"
"/aws/service/global-infrastructure/regions/us-east-1/services/alexaforbusiness"
"/aws/service/global-infrastructure/regions/us-east-1/services/apigateway"
"/aws/service/global-infrastructure/regions/us-east-1/services/application-autoscaling"
"/aws/service/global-infrastructure/regions/us-east-1/services/appmesh"
"/aws/service/global-infrastructure/regions/us-east-1/services/appstream"
"/aws/service/global-infrastructure/regions/us-east-1/services/appsync"
"/aws/service/global-infrastructure/regions/us-east-1/services/athena"
"/aws/service/global-infrastructure/regions/us-east-1/services/autoscaling"

Here’s how to get the list of regions where a service (Amazon Athena, in this case) is available:

$ aws ssm get-parameters-by-path 
  --path /aws/service/global-infrastructure/services/athena/regions --output json | 
  jq .Parameters[].Value
"ap-northeast-2"
"ap-south-1"
"ap-southeast-2"
"ca-central-1"
"eu-central-1"
"eu-west-1"
"eu-west-2"
"us-east-1"
"us-east-2"
"us-gov-west-1"
"ap-northeast-1"
"ap-southeast-1"
"us-west-2"

Here’s how to use the path to get the name of a service:

$ aws ssm get-parameters-by-path 
  --path /aws/service/global-infrastructure/services/athena --output json | 
  jq .Parameters[].Value
"Amazon Athena"

And here’s how you can find the regional endpoint for a given service, again using the path:

$ aws ssm get-parameter 
  --name /aws/service/global-infrastructure/regions/us-west-1/services/s3/endpoint 
  --output json | 
  jq .Parameter.Value
"s3.us-west-1.amazonaws.com"

Available Now
This data is available now and you can start using it today at no charge.

Jeff;

PS – Special thanks to my colleagues Blake Copenhaver and Phil Cali for their help with this post!

 

AWS re:Inforce 2019 – Security, Identity, and Compliance

This post was originally published on this site

AWS re:Inforce, our new conference dedicated to cloud security, opens in Boston on June 25th. We’re expecting about 8,000 attendees, making this bigger than the first re:Invent! Just like re:Invent, re:Inforce is a learning conference for builders.

With over 300 breakout sessions (intermediate, advanced, and expert) spanning four tracks and a virtual Capture The Flag event, attendees will walk away knowing how to use cloud-based infrastructure in a secure and compliant manner. The re:Inforce agenda also includes a healthy collection of bootcamps, chalk talks, workshops, full-day hands-on labs, builder sessions, leadership sessions, and the Security Jam.

Diving deeper into the session offerings, a wide range of services will be considered – including (to name a few) AWS WAF, AWS Firewall Manager, AWS KMS, AWS Secrets Manager, AWS Lambda, AWS Control Tower, Amazon SageMaker, Amazon GuardDuty, AWS CloudTrail, Amazon Macie, Amazon RDS, Amazon Aurora, AWS Identity and Access Management, Amazon EKS, and Amazon Inspector. You will have the opportunity to learn about a broad variety of important topics including building secure APIs, encryption, privileged access, auditing the cloud, open source, DevSecOps, building a security culture, hiring/staffing, and privacy by design as well as specific compliance regimes such as PCI, NIST, SOC, FedRAMP, and HIPAA.

To learn more about re:Inforce, read the FAQ, check out the Venue & Hotel info, and review the Code of Conduct.

Register Now & Save $100
If you register now and use code RFSAL19, you can save $100, while supplies last.

Jeff;