Run Deep Learning Frameworks with GPU Instance Types on Amazon EMR

Today, AWS is excited to announce support for Apache MXNet and new generation GPU instance types on Amazon EMR, which enables you to run distributed deep neural networks alongside your machine learning workflows and big data processing. Additionally, you can install and run custom deep learning libraries on your EMR clusters with GPU hardware. Through using deep learning frameworks, you have a new toolkit to address use cases ranging from autonomous vehicles to artificial intelligence (AI) to personalized healthcare to computer vision.

Amazon EMR provides a managed Hadoop framework that makes it easy, fast, and cost-effective to process vast amounts of data in Amazon S3 with frameworks like Apache Spark, Apache Hive, Presto, Apache HBase, and Apache Flink. You can securely and performantly address a large set of big data use cases at low cost, including log analysis, web indexing, data transformations (ETL), financial analysis, scientific simulation, real-time processing, and bioinformatics.

EMR has a long history in enabling you to run scalable machine learning workloads. In 2013, we added support for Apache Mahout to help you use Apache Hadoop MapReduce to run distributed machine learning workloads. In 2014, customers started leveraging Apache Spark (we added official support in 2015) to easily build scalable machine learning pipelines with the variety of open-source machine learning libraries available in Spark ML.

In the past 2 years, we also added support for Apache Zeppelin notebooks, easy installation for Jupyter notebooks, and Apache Livy for interactive Spark workloads to enable data scientists to easily and quickly develop, train, and move machine learning models into production. With EMR’s per-second billing and cost savings up to 80% by using Amazon EC2 Spot Instances, you can easily run your machine learning pipelines at a massive scale but low cost.

Today, we are making it easier to implement deep learning on Amazon EMR. We have added support for Apache MXNet (0.12.0), a scalable deep learning framework, and Amazon EC2 P3 and P2 instances, EC2 compute-optimized GPU instances, preloaded with the required GPU drivers. You can now quickly and easily create scalable and secure clusters with the latest GPU hardware for distributed training with a few clicks. Furthermore, you can install and use custom deep learning libraries like BigDL or CaffeOnSpark by preloading them on a custom Amazon Linux AMI or using bootstrap actions to customize your cluster. Additionally, EMR will soon add support for TensorFlow, another popular deep learning framework.

EMR makes it easy to combine both the data exploration and preprocessing phases of development with developing and training deep learning models. First, you can easily and cost-effectively use a variety of open-source big data frameworks including Apache Spark, Apache Hadoop, and Apache Hive to explore and process large datasets in S3.

Second, you can then use MXNet and Spark to make predictions or perform inference in addition to developing, training, and running deep learning models with preprocessed data stored in S3 or on-cluster HDFS. You pay per-second, and you can set your own maximum price for EC2 Spot Instances and use Auto Scaling. Then, you can shut down your cluster and stop paying for it when your workloads are complete, to further lower costs of experimentation and production.

You can quickly create an EMR cluster with one to thousands of nodes with Spark, MXNet, Ganglia monitoring, and Zeppelin notebooks with a just few clicks in the EMR console.

After your cluster has started, you can open up your Zeppelin notebook and start exploring data and building models with Spark and MXNet.

EMR makes it easy to monitor and debug your applications through any of the following:

• View the detailed Spark application history directly in the EMR console
• Use Amazon CloudWatch metrics
• View on-cluster Hadoop UIs
• Push application logs directly to S3
• Use Ganglia for resource use metrics

We plan to publish additional posts soon with examples of and best practices on leveraging MXNet and other frameworks on EMR for deep learning at scale. For more information about how to get started, see the Amazon EMR documentation.

---

About the Author

Jonathan Fritz is the Principal Product Manager for Amazon EMR. He leads product management for the team and works to make analytics and machine learning easy on vast amounts of data. In his spare time, he enjoys traveling to new cities, live music, and exploring the outdoors.

 

 

Powered by WPeMatico

The post Run Deep Learning Frameworks with GPU Instance Types on Amazon EMR appeared first on Artificial Intelligence Solutions.

Read More...

Building an Autonomous Vehicle, Part 4: Using Behavioral Cloning with Apache MXNet for Your Self-Driving Car

Read More...

Announcing ONNX Support for Apache MXNet

Today, AWS announces the availability of ONNX-MXNet, an open source Python package to import Open Neural Network Exchange (ONNX) deep learning models into Apache MXNet. MXNet is a fully featured and scalable deep learning framework that offers APIs across popular languages such as Python, Scala, and R. With ONNX format support for MXNet, developers can build and train models with other frameworks, such as PyTorch, Microsoft Cognitive Toolkit, or Caffe2, and import these models into MXNet to run them for inference using the MXNet highly optimized and scalable engine.

We’re also excited to share that AWS will be collaborating on the ONNX format. We will be working with Facebook, Microsoft, and the deep learning community to further develop ONNX, making it accessible and useful for deep learning practitioners.

What is ONNX?

ONNX is an open source format to encode deep learning models. ONNX defines the format for the neural network’s computational graph, as well as the format for an extensive list of operators used within the graph. With ONNX being supported by an increasing list of frameworks and hardware vendors, developers working on deep learning can move between frameworks easily, picking and choosing the framework that is best suited for the task at hand.

Quick Start

We’ll show how you can use ONNX-MXNet to import ONNX models into MXNet, and use the imported model for inference, benefiting from the MXNet optimized execution engine.

Step 1: Installations

First, install ONNX, following instructions on the ONNX repo.

Then, install the ONNX-MXNet package:

$ pip install onnx-mxnet

Step 2: Prepare an ONNX model to import

In this example, we will demonstrate importing a Super Resolution model, designed to increase spatial resolution of images. The model was built and trained with PyTorch, and exported into ONNX using PyTorch’s ONNX export API. More details about the model design are available in PyTorch’s example.

Download the Super Resolution ONNX model to your working directory:

$ wget http://ift.tt/2j0wpWG

Step 3: Import the ONNX model into MXNet

Now that we have an ONNX model file ready, let’s import it into MXNet using the ONNX-MXNet import API. Run the following code in a Python shell:

import onnx_mxnet
sym, params = onnx_mxnet.import_model('super_resolution.onnx')

This created two instances in the Python runtime: sym – the model’s symbolic graph, and params – the model’s weights. Importing the ONNX model is now done, and we have a standard MXNet model.

Step 4: Prepare input for inference 

Next, we will prepare an input image for inference. The following steps download an example image, resize it to the model’s expected input shape, and finally convert it into a numpy array.

From your shell console, download the example input image to your working directory:

$ wget http://ift.tt/2zOWxvA

Then, install Pillow, Python Imaging Library, so we can load and pre-process the input image:

$ pip install Pillow

Next, from your Python shell, run the code to prepare the image in the MXNet NDArray format:

import numpy as np
import mxnet as mx
from PIL import Image
img = Image.open("super_res_input.jpg").resize((224, 224))
img_ycbcr = img.convert("YCbCr")
img_y, img_cb, img_cr = img_ycbcr.split()
test_image = mx.nd.array(np.array(img_y)[np.newaxis, np.newaxis, :, :])

Step 5: Create the MXNet Module

We’ll be using the MXNet Module API to create the module, bind it and assign the loaded weights.
Note that the ONNX-MXNet import API assigns the input layer the name ‘input_0’, which we are using when initializing and binding the module.

mod = mx.mod.Module(symbol=sym, data_names=['input_0'], label_names=None)
mod.bind(for_training=False, data_shapes=[('input_0',test_image.shape)])
mod.set_params(arg_params=params, aux_params=None)

Step 6: Run inference

Now that we have an MXNet Module loaded, bound, and with trained weights, we’re ready to run inference. We’ll prepare a single input batch, and feed forward through the network:

from collections import namedtuple
Batch = namedtuple('Batch', ['data'])
mod.forward(Batch([test_image]))
output = mod.get_outputs()[0][0][0]

Step 7: Examine the results

Now let’s examine the results we received running inference on the Super Resolution image:

img_out_y = Image.fromarray(np.uint8((output.asnumpy().clip(0, 255)), mode='L'))
result_img = Image.merge(
"YCbCr", [
                img_out_y,
                img_cb.resize(img_out_y.size, Image.BICUBIC),
                img_cr.resize(img_out_y.size, Image.BICUBIC)
]).convert("RGB")
result_img.save("super_res_output.jpg")

Here’s the input image and the resulting output image. As you can see, the model was able to increase the image spatial resolution from 256 by 256 to 672 by 672.

Input image	Output image

What’s Next?

We’re working with our ONNX partners and community to further develop ONNX, adding more useful operators, extending ONNX-MXNet to include export and increased operator coverage. We will be working with the Apache MXNet community to bring ONNX into MXNet core APIs.

Want to learn more?

The example is available here, as part of the ONNX-MXNet GitHub repo.

Check out ONNX to dive into how network graphs and operators are encoded.

Contributions are welcomed!

Special thanks to the dmlc/nnvm community, whose ONNX code was used as a reference for this implementation.

Facebook Blog:
http://ift.tt/2iZYKg1

Microsoft Blog:
http://ift.tt/2zPqoUE

 

---

 

About the Authors

Hagay Lupesko is an Engineering Manager for AWS Deep Learning. He focuses on building Deep Learning tools that enable developers and scientists to build intelligent applications. In his spare time he enjoys reading, hiking and spending time with his family.

 

 

 

Roshani Nagmote is a Software Developer for AWS Deep Learning. She is working on innovative tools to make Deep Learning accessible for all. In her spare time, she loves to play with her adorable nephew and is a huge dog lover.

 

 

 

 

 

Powered by WPeMatico

The post Announcing ONNX Support for Apache MXNet appeared first on Artificial Intelligence Solutions.

Read More...

Amazon Polly Adds 9 AWS Regions, Korean Language Support, and a New Indian English Voice

Read More...

New AWS Deep Learning AMIs for Machine Learning Practitioners

We’re excited to announce the availability of two new versions of the AWS Deep Learning AMI: a Conda-based AMI with separate Python environments for deep learning frameworks created using Conda—a popular open source package and environment management tool; and a Base AMI with GPU drivers and libraries to deploy your own customized deep learning models.

Deep learning technology is evolving at a rapid pace—everything from frameworks and algorithms to new methods and theories from academia and industry. All of this causes complexity for developers who need tools for quickly and securely testing algorithms, optimizing for specific versions of frameworks, running tests and benchmarks, or collaborating on projects starting with a blank canvas. Virtual environments provide the freedom and flexibility to do all this, which is why we’re adding it to the AWS Deep Learning AMIs today. We’ve also set up new developer resources to help you learn more about the AMIs, choose the right AMI for your project and dive into hands-on tutorials.

New Conda-based Deep Learning AMI

The Conda-based AMI comes pre-installed with Python environments for deep learning created using Conda. Each Conda-based Python environment is configured to include the official pip package of a popular deep learning framework, and its dependencies. Think of it as a fully baked virtual environment ready to run your deep learning code, for example, to train a neural network model. Our step-by-step guide provides instructions on how to activate an environment with the deep learning framework of your choice or swap between environments using simple one-line commands.

But the benefits of the AMI don’t stop there. The environments on the AMI operate as mutually-isolated, self-contained sandboxes. This means when you run your deep learning code inside the sandbox, you get full visibility and control of its run-time environment. You can install a new software package, upgrade an existing package or change an environment variable—all without worrying about interrupting other deep learning environments on the AMI.  This level of flexibility and fine-grained control over your execution environment also means you can now run tests, and benchmark the performance of your deep learning models in a manner that is consistent and reproducible over time.

Finally, the AMI provides a visual interface that plugs straight into your Jupyter notebooks so you can switch in and out of environments, launch a notebook in an environment of your choice, and even reconfigure your environment—all with a single click, right from your Jupyter notebook browser. Our step-by-step guide walks you through these integrations and other Jupyter notebooks and tutorials.

The new Conda-based Deep Learning AMI comes packaged with the latest official releases of the following deep learning frameworks:

• Apache MXNet 0.12 with Gluon
• TensorFlow 1.4
• Caffe2 0.8.1
• PyTorch 0.2
• CNTK 2.2
• Theano 0.9
• Keras 1.2.2 and Keras 2.0.9

This AMI also includes the following libraries and drivers for GPU acceleration on the cloud:

• CUDA 8 and 9
• cuDNN 6 and 7
• NCCL 2.0.5 libraries
• NVidia Driver 384.81

New Deep Learning Base AMI

The Base AMI comes pre-installed with the foundational building blocks for deep learning. This includes NVIDIA CUDA libraries, GPU drivers, and system libraries to speed up and scale machine learning on Amazon Elastic Compute Cloud (EC2) instances. Think of the Base AMI as a clean slate to deploy your customized deep learning set up.

For example, for developers contributing to open source deep learning framework enhancements or even building a new deep learning engine, the Base AMI provides the foundation to install your own custom configurations and code repositories to test out new framework features. The Base AMI comes with the CUDA 9 environment installed by default, however you can also switch to a CUDA 8 environment using simple one-line commands given in our step-by-step user guide.

The Base AMI provides the following GPU drivers and libraries:

• CUDA 8 and 9
• CuBLAS 8 and 9
• CuDNN 6 and 7
• glibc 2.18
• OpenCV 3.2.0
• NVIDIA driver 384.81
• NCCL 2.0.5
• Python 2 and 3

Deep Learning AMIs with source code

In addition to the two new AMIs available today, we continue to support the AMIs that install all the popular deep learning frameworks from source in a unified Python environment, and include their source code on the AMI. This AMI is great if you want to try out and compare multiple frameworks in a shared base environment or you need quick access to source code on the AMI itself to recompile a framework with your custom set of build options.

The AMI comes in CUDA 8 and CUDA 9 versions to meet your specific needs of the AWS EC2 instance you want to use for deep learning. 

Deep Learning AMIs cheat sheet

We now have three types of AWS Deep Learning AMIs available in the AWS Marketplace to support the various needs of machine learning practitioners. Don’t forget to check out our AMI selection guide, simple tutorials, and more deep learning resources in our developer guide!

Conda-based AMI	Base AMI	AMIs with source code
For developers who want pre-installed pip packages of deep learning frameworks in separate virtual environments	For developers who want a clean slate to set up private deep learning engine repositories or custom builds of deep learning engines	For developers who want pre-installed deep learning frameworks and their source code in a shared python environment
Deep Learning AMI (Ubuntu) Deep Learning AMI (Amazon Linux)	Deep Learning Base AMI (Ubuntu) Deep Learning Base AMI (Amazon Linux)	For P3 instances: Deep Learning AMI with Source Code (CUDA 9, Ubuntu) Deep Learning AMI with Source Code (CUDA 9, Amazon Linux)
		For P2 instances: Deep Learning AMI with Source Code (CUDA 8, Ubuntu) Deep Learning AMI with Source Code (CUDA 8, Amazon Linux)

 

---

 

About the Author

Cynthya Peranandam is a Principal Marketing Manager for AWS artificial intelligence solutions, helping customers use deep learning to provide business value. In her spare time she likes to run and listen to music.

 

 

 

Powered by WPeMatico

The post New AWS Deep Learning AMIs for Machine Learning Practitioners appeared first on Artificial Intelligence Solutions.

Read More...

Getting Started with the AWS Deep Learning Conda and Base AMIs

Today AWS announced the availability of two new versions of the AWS Deep Learning AMI: a Conda-based AMI and a Base AMI. This post will walk you through the instructions and additional resources for making the most of your new AMIs.

New Deep Learning AMI with Conda-managed environments

The new Deep Learning AMIs for Amazon Linux and Ubuntu come pre-installed with Python environments for deep learning created using Conda – a popular open source package and environment management tool. The Conda-managed Python environments are pre-configured for popular deep learning frameworks including Apache MXNet, TensorFlow, Caffe2, PyTorch, Keras, CNTK, and Theano. In addition, each Python environment comes in two flavors – Python 2 and Python 3. After you log in to your AWS EC2 instance using the AWS Management Console, you will be greeted with a console message that lists all the Conda environments.

You can also get this list by running the following command:

conda env list

Next, to activate a Python environment for a deep learning framework of your choice, say MXNet, run the following:

For Python 2

source activate mxnet_p27

For Python 3

source activate mxnet_p36

After you are inside the Python environment, you can view the list of installed packages by running the following command:

conda list

Running your deep learning code inside the Python environment is simple. First start the python shell:

python

Then import a deep learning framework, or run your deep learning Python code like you normally would:

import mxnet

Now let’s switch to another deep learning framework, say TensorFlow. First exit the python shell

exit()

Then deactivate the current environment for MXNet

source deactivate

Then switch to a Python environment like you did before, but this times you’ll activate TensorFlow:

For Python 2

source activate tensorflow_p27

For Python 3

source activate tensorflow_p36

To read more about managing Conda environments, you can get the Conda commands cheat sheet and additional learning resources in the getting started guide for Conda. You can also go to the new AWS Deep Learning AMI doc site and explore the introductory tutorials that you can run directly from the command line.

Manage Conda from your Jupyter notebook Interface

You can also manage the Conda environments straight from your Jupyter notebook browser interface. You can launch a Jupyter notebook server on the Conda-based AMI using instructions on our doc site. Conda supports close integrations to the Jupyter notebook with following features:

Choosing your deep learning environment

First access the Jupyter server on your internet browser. On the main Files page you can choose a Conda environment with the deep learning framework of your choice from the drop-down list, as shown in the following screen shot. You can then continue starting a new notebook. It will automatically be linked to the Python environment that you selected.

You can also use the drop-down list on this page to switch to another environment with a different deep learning framework. To help you get started with your first notebook, the Conda-based AMI comes bundled with several Jupyter notebooks and ready to launch tutorials.

Managing environments

Open the Conda tab, and you’ll see a dedicated page for managing the Conda environments on the AMI:

On this page, you can browse the list of all the pre-installed Conda environments, the software packages installed inside an environment, and even reconfigure an environment by upgrading packages or uninstalling them.

Configuring the new Deep Leaning Base AMI

The Base AMI for Amazon Linux and Ubuntu comes with a foundational platform of GPU drivers and acceleration libraries you can use to deploy your own customized deep learning environment. By default, the AMI is configured with an NVidia CUDA 9 environment. However, you can also switch to a CUDA 8 environment by reconfiguring the environment variable LD_LIBRARY_PATH. You simply need to replace the CUDA 9 portion of the environment variable string with its CUDA 8 equivalent.

CUDA 9 portion of the LD_LIBRARY_PATH string (installed by default)

…:/usr/local/cuda-9.0/http://lib64:/usr/local/cuda-9.0/extras/CUPTI/lib64:/lib/nccl/cuda-9:…… rest of LD_LIBRARY_PATH value

Replace with CUDA 8

…:/usr/local/cuda-8.0/http://lib64:/usr/local/cuda-8.0/extras/CUPTI/lib64:/lib/nccl/cuda-8:…… rest of LD_LIBRARY_PATH value

Getting Started

Getting started with the Deep Learning AMI is easy. You can follow our step-by-step blog or visit our new AWS Deep Learning AMI doc site to get started with how-to guides and useful resources.

---

About the Author

 

Sumit Thakur is a Senior Product Manager for AWS Deep Learning. He works on products that make it easy for customers to get started with deep learning on cloud, with a specific focus on making it easy to use engines on Deep Learning AMI. In his spare time, he likes connecting with nature and watching sci-fi TV series.

 

 

 

Powered by WPeMatico

The post Getting Started with the AWS Deep Learning Conda and Base AMIs appeared first on Artificial Intelligence Solutions.

Read More...

Building an Autonomous Vehicle Part 3: Connecting Your Autonomous Vehicle

In the first blog post in our autonomous vehicle series, you built your Donkey vehicle and deployed your pilot server onto an Amazon EC2 instance. In the second blog, you learned to drive the Donkey car, and the Donkey car learned to drive itself. In this blog post, we’ll cover the process of streaming telemetry from the Donkey vehicle into AWS. We’ll use the AWS IoT service because it provides a scalable, reliable, and feature-rich set of services for all kinds of connected devices, including our connected vehicles.

---

1) Build an Autonomous Vehicle on AWS and Race It at the re:Invent Robocar Rally
2) Build an Autonomous Vehicle Part 2: Driving Your Vehicle
3) Building an Autonomous Vehicle Part 3: Connecting Your Autonomous Vehicle
4) Coming soon

---

AWS IoT setup

The autonomous car is going to generate a constant stream of telemetry while it is driving. When the vehicle is not driving, there is no telemetry to gather, so we don’t want to utilize any resources because it would be wasteful. In order to accommodate our workload, we are going to rely on serverless technologies to power our entire architecture. To start, we are going to use AWS IoT to design a fleet monitoring service. It can service any amount of vehicles using the same basic architecture. The following diagram illustrates the architecture for the fleet monitoring service.

The components of this solution are color-coded by logical functionality (and AWS services) to demonstrate how each component of the solution is secure, scalable, and entirely usage based. This kind of operational model lends itself to many business models and is useful for customers of all sizes. Whether it’s the weekend autonomous vehicle hobbyist who wants to track and compare lap times, or an automobile manufacturer that wants to develop their own connected vehicle platform, AWS IoT provides a secure, scalable, and usage-based cost structure.

This solution begins with the Donkey car, which is shaded in green. Data then passes through the IoT service to the pink section which is for short term data storage in DynamoDB, and then to the blue section for longer term storage in S3. Additionally, data in an AWS IoT topic can also be queried in real-time, and as show below will be used to drive a dashboard.

The Donkey car is already internet-connected, which means that it is capable of streaming telemetry to a centralized location, in this case AWS IoT. To ensure the security of the telemetry, we’ll generate certificates from AWS IoT and deploy them to the Donkey vehicle. By using certificates, the Donkey car can securely communicate with AWS IoT using TLSv1.2 over MQTT. MQTT is an extremely lightweight protocol that deals well with low signal networks, so it can cope with connection reliability challenges.

The simplest way to handle security is to use the one-click certificate creation method. To do this, open the AWS IoT console. In the navigation pane at the left, choose Security. Next, choose the Create button.

Create a certificate

A certificate can be created using the AWS IoT console, the AWS IoT API, or as we did in our previous blog post, using the Amazon EC2 Systems Manager Run Command on Raspberry Pi to generate and deliver the certificates. By performing this process using EC2 Systems Manager, we don’t need to manually copy the certificates to the Raspberry Pi.

For the sake of simplicity, we’ll use the AWS IoT console to complete the process.

We’re going to create a new policy that will be used to grant our Donkey car specific permissions to AWS services. This allows us to set fine-grained vehicle-specific settings, such as which AWS IoT topics can be accessed, and which IoT actions can be taken.

Begin by opening your AWS Management Console. Go to the AWS IoT console and choose Security, then Policies. Then choose Create.

We’ll use this policy to allow all actions to be performed against the AWS IoT service because we don’t need to prevent the use of the full functionality of the service. This privilege includes: iot:Publish, iot:Subscribe, iot:Connect, iot: Receive, iot:UpdateThingShadow, iot:GetThingShadow, and  iot:DeleteThingShadow.

In addition to the certificate and the policy, the AWS IoT service needs to create a Thing that describes our Donkey car. To do this, in the navigation pane at the left, choose Things from Registry. Now choose Create, and provide your Thing with a name and click Create thing.

Now that the policy and thing are created, we can associate them with the certificate that we previously created. To do that, in the navigation pane on the left, choose Security, Certificates. Locate and then select the certificate that you created previously. From the Actions menu, choose Attach Policy. Then from the Actions menu, choose Attach Thing.

AWS IoT rules

Now that we have a means of generating and communicating telemetry, we can focus on building the rest of the functionality of the solution. Our solution calls for two rules, one for Amazon DynamoDB to be accessed by our dashboard, and the other for Amazon Kinesis Firehose to distribute all telemetry to Amazon S3 and Amazon Kinesis Analytics for real-time analysis of the telemetry.

DynamoDB Rule

We will start by creating the rule for DynamoDB. In the navigation pane at the left in the AWS IoT console, choose Rules. Then choose the Create button.

On the Create a rule page, give the rule a name and add a description that easily identifies the type of data and the AWS service that is consuming those records.

 

Then choose Add action, and select Split message into multiple columns of a database table (DynamoDBv2).

Next choose Configure action.

Next, you can either create a new table or use an existing one. We’ll create a new table by choosing Create a new resource.

We now follow the Create DynamoDB table wizard. We call the Table name AutonomousVehicles and set the Primary key (partition key) to be the vehicleID attribute which will be unique among all of the vehicles in the fleet. We’ll add a sort key on the attribute time which will allow the most efficient use of DynamoDB for queries. Leave the default settings. They should be more than sufficient to handle telemetry from a single vehicle.

While DynamoDB is creating the table, we can set a TTL attribute to keep the costs of running DynamoDB very low. If you keep the default of 5 Read capacity units and 5 Write capacity units the monthly bill is around $2.50. The RCU and WCU can be adjusted up and down manually, or you can choose to use Auto Scaling. To keep costs low, we’ll also allow DynamoDB to automatically expire items based on an attribute that the user defines. In our case, we are going to add a TTL timestamp to our telemetry. In our program that runs on the Donkey car, we’ll set the attribute dynamodb_ttl as the current Unix Timestamp, plus 2592000 (30 days) and then store it in DynamoDB. 30 days later DynamoDB sees that the TTL attribute has expired and deletes those items automatically.

Choose Continue to enable TTL. Return where you left off with the AWS IoT Configure action, and from Table name select your table from the drop-down list. If you do not see it, click the refresh arrows. Next, choose Create a new role and name it AutonomousVehiclesDynamoDB or something similar. Choose Update role, and then choose Add action.

Next we can review all of our choices. When you are ready, choose Create rule.

Kinesis Firehose Rule

We can now follow similar steps to create another rule to send all of the telemetry to Kinesis Firehose. In the navigation pane at the left, choose Rules. Then choose the Create button.

On the Create a rule page, give the rule a name with a description that easily identifies the type of data and the AWS service that is consuming those records.

Then choose Add action, and select Send messages to an Amazon Kinesis Firehose stream:

Then choose Configure action.

Now there is a choice: Create a new stream or use an existing one. We’ll create a new stream by choosing Create a new resource.

We now follow the Create delivery stream wizard and provide a name for the Delivery stream. Choose Next.

Choose Next.

Ensure that Amazon S3 is selected. Choose an appropriate S3 bucket and Prefix where you want the telemetry to be stored.

Choose Create new, or Choose for the IAM role.  From the drop-down list, select Create a new IAM Role and provide it with a Role Name. Choose Allow. Then choose Next.

Finally, confirm everything is correct and choose Create delivery stream.

Return to the Configure action screen and select the new stream that was created.

Then choose Create a new role to be used by AWS IoT to access the Amazon Kinesis Firehose stream. After naming the role, choose Update role and then Add action.

Finally, confirm all of the choices, and then choose Create rule when ready.

The AWS IoT service is now ready and waiting for telemetry to stream in. Since AWS IoT is serverless and usage based, there is no cost incurred with deploying it and leaving it on. Amazon Kinesis Firehose and Amazon S3 are also priced based on the volume of data ingested. We can now generate telemetry, and then use the AWS IoT console to check that it is being sent to the proper topic.

Alternatively, we could build a dashboard that connects directly to AWS IoT so that it could consume this telemetry in near real time. Here is an example of a dashboard that is hosted on Amazon S3, which means it is completely serverless and requires no management of underlying webservers. An example of how you can visualize IoT telemetry can be found in this tutorial:

Deploy an End-to–End IoT Application (pdf)

And in this blog post:

Build a Visualization and Monitoring Dashboard for IoT Data with Amazon Kinesis Analytics and Amazon QuickSight

In our next blog post we will re-cap what we’ve done so far and talk about what’s next for autonomous vehicles.

Powered by WPeMatico

The post Building an Autonomous Vehicle Part 3: Connecting Your Autonomous Vehicle appeared first on Artificial Intelligence Solutions.

Read More...