April 2024 – Operating, Maintaining, and Securing IoT Networks

25Apr

Monitoring ingested data – Working with Data and Analytics

by Keitra Marche AWS Certification Exam, Monitoring for security purposes

Monitoring is an important component of analyzing IoT data, and we can do this as detailed next:

You can check messages that have been ingested into your channel through the AWS IoT Analytics console. Within the console, on the left pane, click on Channel and choose the name of the channel that we created.

On the page, scroll down to the Monitoring section. Adjust the time frame that you currently want to be displayed as needed by choosing one of the existing time frame indicators. You will then see a graph line that shows the number of messages that were ingested into the channel during that period.

You can also check for pipeline activity executions. Follow the same workflow process you went through by clicking Pipelines, followed by the name of the pipeline that was created on the console, and adjust the time frame indicators. You will see a graph line that shows the number of pipeline activity execution errors in that period.

Now that we can monitor the data, let’s look at creating a dataset from our data.

Creating a dataset from the data

We can now look at creating a dataset from the data we have. Proceed as follows:

Navigate to Datasets from the sidebar and click on the Create button.

Name your dataset mydataset.

For the action, use a SQL expression such as select * from mydatastore.

Complete the dataset creation by clicking Create.

To generate dataset content, select the dataset and click on the Run Now button.

To view the content, click on the dataset name and navigate to its content. You should see the results and can even download them if they’re available.

With that, we have been able to create an end-to-end pipeline! Note that much of this infrastructure is still based solely on AWS. In real-world deployments, we would see more interconnectivity between AWS and on-premises equipment and see real-time data analysis being performed based on the data ingested.

As always, feel free to look at the documentation for all the hardware and software involved in this practical. We encourage you to explore further data analysis options, particularly in the transformation process when creating an ETL Glue job.

Summary

In this chapter, we have covered the fundamentals of data analytics within IoT workloads, discussing how different services within AWS can handle analysis and storage loads that are required as part of our IoT cloud workflow. We then learned more about how we can design and develop the implementation of our architecture according to our use case and learned how to practically use the offerings from AWS IoT Analytics to provision an end-to-end data pipeline.

In the next chapter, we will be discussing security and privacy within IoT, which is an imperative topic to talk about, given how it has become even more prevalent as more and more people have shifted their workloads into the cloud.

Archives April 2024