☁️Cloudy with a chance of IoT Data
Disclaimer: I am a consultant who works for an analytics company with a Microsoft and Amazon AWS partnership. I was in no part solicited or paid for this review. The views and opinions expressed in this article are those of the author and do not necessarily reflect those of any other related party.
For those of you who know me, I love sensors: they give me data, allow me to learn how things work, and show how information changes over time. So in my spare time, I am exploring the field. (And for those of you who are new to IoT, check out the Particle Photon. It's a fantastic wifi-enabled development board for prototyping IoT solutions.)
In my day-job, I am an analytics consultant leading teams, projects, and clients through the work needed to move their data into the cloud and visualize it in Tableau or other tools. The goal is to empower the client to use their data to gain information about their business to help them earn more, spend less, do more, etc. I wanted to do a similar thing with my hobbyist IoT data: move the data to the cloud and visualize it in a leading analytics tool. Armed with my personal email, I created free-trial accounts on all 3 major cloud vendors: Amazon AWS, Google, and Microsoft Azure. This post is a brief recap of my findings, the surprises, and the results. Enjoy!
Cloud 1: Google Cloud
I started with Google because Particle.io and Google have a partner relationship. And I've used BigQuery in other projects, so I had some familiarity with the platform. Using Google cloud, I managed to correctly set up the integration and webhook to Pub/Sub (with some help from the tutorials).
However, I couldn't get a Node.js server spun up correctly to transfer the Pub/Sub topic to BigQuery. And when I looked into DataStream, all of the sample code was out of date and constantly threw validation errors. And I couldn't find any helpful documentation online or on forums or on Stack Exchange. So I abandoned Google Cloud.
Cloud 2: Amazon AWS
I figured I would try AWS next as I am most familiar with it from previous experience as a consultant. After 2-3 days of tinkering with AWS, I was not able to get any data elements to flow end-to-end. Lambda would talk to DynamoDB. API Gateway could receive the webhook from Particle. However the API gateway / Lambda integration just wasn't working well.
Despite several rounds of Node.js code and Python code in Lambda, I could never fully resolve the issues. I finally abandoned the setup when I ran into an authentication error somewhere in the stack of API Gateway/Lambda/Webhook. And all of the examples I could find online were just too old to be helpful -- the API Gateway functionality has changed quite a bit in AWS. So I abandoned AWS.
Cloud 3: Microsoft Azure
I have to admit, after AWS and Google didn't work out, I was skeptical of Azure. I had the least familiarity with the platform from the start. Nevertheless, I created an account and was pleasantly surprised. Microsoft had an excellent set of tutorials for IoT, they were well documented, up-to-date, and simple! After one evening, I manged to get the Particle Photon(s) connected to Azure and saving the data into "Azure Storage" (the Azure equivalent to S3 from AWS or Cloud Buckets from GCP.) In a second evening, I was able to route the data into a SQL server instance for connecting to Tableau. Woohoo!
(Note: Microsoft also makes Power BI available for visualizations and they have a good tutorial here. For speed of dashboard setup, I went with Tableau because I already have done a lot of work with it and I could publish a version to Tableau Public to share with all of you. Maybe a future post should explore Power BI? 😊)
I was pleasantly surprised by the ease of use of the Microsoft Azure platform for this specific use case. And I was even more surprised that the documentation was easy to use and up-to-date. Thank you Microsoft! 😊
Here is my Tableau dashboard. You can see an interactive version in Tableau Public.
- Interestingly enough, I have two sets of hardware running the same code and same sensor hardware, but I found the two humidity sensors are nearly 10 percentage points off from each other. Never seen that before because I never combined two sensor's data together! 😊
- Jumps in the Wifi RSSI are often changes in the router configuration or moving the IoT hardware. In this case, I moved one of the sensors farther away from my router at home so the signal quality dropped (so the RSSI gets a larger negative value.)
- The State of Charge variable shows the two sensors diverging in value: one plugged into the wall powered by USB, the second hooked up to a Solar Panel and running on a LiPoly battery.
In short, I was pleasantly surprised the Microsoft Azure Cloud. If you have an IoT project on the horizon, whether personal or professional, check out Azure. Oh, and tell them Gordon Strodel from Slalom sent you! 😉
- Great set of IoT Tutorials from Microsoft: here
- To learn more about IoT and get started with a easy-to-use platform, check out Particle.io
- Ubidots is also a great platform for quickly visualizing IoT data. Check out my dashboard there too!
- Lastly, for those of you interested in an end-to-end tutorial of IoT, Azure, and making beer, check out this tutorial!
- You can check out my tinkering in IoT here on my personal site