Scraping and providing data for IoT (in this case ESP32/ESP8622)

This article continued the previous post about scraping coronavirus cases, from the previous post we have got coronavirus cases data by requests.get from infocorona.baliprov.go.id, and thank you for the amazing module “BeautifulSoup”.

There are three steps that we will do:

1. Scraping the data from the original source (we have done with the previous post).
2. Providing the data to IoT (ESP32).
3. Get and display the data from IoT.

We will skip the first step, and continue to step two, there are many alternatives for providing data, such as create a REST API (in this case from complex API to simpler once), create a simple JSON file, use MQTT broker, etc. While my previous post was written, I choose the MQTT broker as a solution but after inspired by https://medium.com/hyperjump-tech/layanan-api-untuk-covid-19-tapi-berasa-statik-89116355ab5 by Ariya Hidayat, for providing data I will use Github and Netlify, in a simple scenario I put the codes on Github than Netlify running it by a trigger.

I put scrapper script to Github covid_cases_bali.py

This file will create three files, there are index.html, corona.html, and JSON file, we will use this JSON file for providing data to ESP32. This script will automatically build and deploy on Netlify which trigger by webhook every 60 minutes (every 60 minutes this script will scrapping data from our sources infocorona.baliprov.go.id), good news for our admin.

Netlify Dashboard

Our third step is to display data to LCD/etc from ESP32. For this case, I use ESP32 which flashed with Micropython firmware. I create a python script to do this job.

covid_case_esp32.py

The result was shown as below.

Final result

The last task is to print this to LCD/OLED/E-paper which attached to ESP32, maybe my next post.