Hello,
I’ve noticed a couple of people asking lately, how to collect data from a HAT or pHAT and log it to a local file.
The routine below should be a good jumping off point for people to start with and does the following:
- Uses python to collect data from a sensor (in my case an Enviro pHAT).
- Writes the python output to a comma separated file.
- Uses gnuplot to turn slices of the
.csvfile in to graphs, stored locally. - Instructs a web server on a different machine to pull those files into the
/var/wwwfolder.
A web page then easily serves these, as they always have the same name.
So, putting all of this together:
The Enviro pHAT sits on top of a pi Zero WH (named phats). It uses ssh keys for authentication.
The pi is connected to the internal network (and therefore the web server) and has a known IP address.
The webserver is named sheevaplug, it has a single user with no rights allocated. The user has to su to get rights.
On the pi there are a few files:
- The python file that collects data from the Enviro pHAT (enviroplot.py).
- The
.csvfile that the python file writes comma seperated values to (enviroplot.csv). - The gnuplot script(s) that interpret the
.csvrows and produce the.pnggraphs (temp_range.plot, temp_recent.plot). - The
shfile that wraps the python file and the gnuplot scripts together (enviroplot.sh). - A
cronjob then executes theshfile every 15 minutes.
enviroplot.py
#!/usr/bin/env python
import csv
from datetime import datetime
from envirophat import weatherdt = datetime.now()
full_date = dt.strftime(‘%Y-%m-%d’)
full_time = dt.strftime(‘%H:%M:%S’)
full_day = dt.strftime(‘%A’)
year = dt.year
month = dt.month
date = dt.day
hour = dt.hourtemp = round(weather.temperature(), 2)
baro = round(weather.pressure(unit = ‘hPa’), 2)csvWrite = full_date, full_time, full_day, year, month, date, hour, temp, baro
csvFile = open(‘/home/foo/Dev/Raspi/enviroplot/enviroplot.csv’, ‘a’)with csvFile:
writer = csv.writer(csvFile)
writer.writerow(csvWrite)
The important bits here are:
round(xxxx, 2)rounds to two decimal places.- The
'a'at the end of thecsvFileline means append. The.csvfile must exist first, even if it is empty to start with.
enviroplot.csv (Example of data collected)
full_date, full_time, full_day, year, month, date, hour, temp, baro
2019-01-23,22:15:01,Wednesday,2019,1,23,22,23.12,998.17
2019-01-23,22:30:01,Wednesday,2019,1,23,22,22.89,998.35
2019-01-23,22:45:02,Wednesday,2019,1,23,22,22.54,998.41
2019-01-23,23:00:02,Wednesday,2019,1,23,23,22.2,998.44
2019-01-23,23:15:01,Wednesday,2019,1,23,23,21.9,998.46
2019-01-23,23:30:01,Wednesday,2019,1,23,23,21.73,998.82
2019-01-23,23:45:01,Wednesday,2019,1,23,23,21.47,999.24
2019-01-24,00:00:01,Thursday,2019,1,24,0,21.18,999.26
2019-01-24,00:15:01,Thursday,2019,1,24,0,20.96,999.3
2019-01-24,00:30:02,Thursday,2019,1,24,0,20.92,999.32
2019-01-24,00:45:01,Thursday,2019,1,24,0,20.87,999.64
2019-01-24,01:00:01,Thursday,2019,1,24,1,20.63,999.82
I have deliberately collected the full date, full time, day, year, month, date, hour separately as I may want to analyse these elements later and don’t want to have to dissect the .csv string.
(The final two values are temperature and barometric pressure.)
The following two .plot examples are the gnuplot scripts.
Gnuplot is an incredibly versatile open source plotting piece of software and is available in the repos.
sudo apt install gnuplotIt is capable of creating 2D and 3D graphs and outputting to a variety of formats.
You can even plot ASCII graphs in the terminal with it!
You can create a graph with as little as one line, specifying simply where the input data is.
I’m still learning, so you may have better looking outputs with more practice.
temp_range.plot
reset
unset key
set key off
set terminal png
set datafile separator ","
set xdata time
set timefmt "%Y-%m-%dT%H:%M:%S"
set format x "%d/%m"
set xlabel "Date (day/month)"
set ylabel "Temp"
set style data points
set title "Temp Range"
set grid
plot '/home/foo/Dev/Raspi/enviroplot/enviroplot.csv' using 1:8The important bits here are:
- ‘
plot’ i.e. the location of the.csvfile containing data. - ‘
Using 1:8’ i.e. use data columns 1 & 8
temp_recent.plot
reset
unset key
set key off
set terminal png
set datafile separator ","
set xdata time
set timefmt "%Y-%m-%dT%H:%M:%S"
set format x "%d/%m"
set xlabel "Date (day/month)"
set ylabel "Temp"
set style data line
set title "Recent Temp"
set grid
plot '< tail -n 672 /home/foo/Dev/Raspi/enviroplot/enviroplot.csv' using 1&2:8The important bits here are:
- ‘
tail -n 672’ i.e. take the last 672 data points ( 1 x week at 15 minute intervals). - ‘
using 1&2:8’ i.e. concatenate columns 1&2 as one data column and column 8 as the other.
enviroplot.sh
#!/bin/sh
# This is the script that the cron job will execute
# Run the python script to collect the variables
python /home/foo/Dev/Raspi/enviroplot/enviroplot.py
#### Create the graph images from gnuplot
# 'Sleep' is there to prevent a conflict
gnuplot /home/foo/Dev/Raspi/enviroplot/temp_range.plot > /home/foo/Dev/Raspi/enviroplot/temp_range.png
sleep 5
gnuplot /home/foo/Dev/Raspi/enviroplot/temp_recent.plot > /home/foo/Dev/Raspi/enviroplot/temp_recent.pngThe important bits here are:
- Runs the python script that writes the Enviro pHAT data to the
.csvfile - Calls the gnuplot file(s) and outputs it to a named
.pngfile i.e. creates the graph image file(s). 'Sleep'is there ascronwill execute both commands at the same time; better to have a slight delay and wait for the first command to finish.
phats crontab
# m h dom mon dow command
# This is the script that runs the bash script to collect data and create graphs
*/15 * * * * /home/foo/Dev/Raspi/enviroplot/enviroplot.shThe cron job that runs every 15 minutes to execute the above enviroplot.sh script.
The above will get you a working pair of graphs that are refreshed every 15 minutes.
But they will be on the pi which may be headless or unattached to any sort of display.
Much better to put the images on the interweb where you can watch your house getting hotter when it’s burning.
The next two files are on the web server and comprise of:
- A
shscript to pull the files from the pi over the local network. - A
cronjob to do that every 15 minutes.
I have to pull the files into the web server, rather than push the files from the pi because, as I said earlier the server only has a single user with no rights and I couldn’t find a way to connect to the server using ssh, then pass a password to get rights. (Probably is a way, but too late now.)
/var/www/get_graphs.sh
#!/bin/sh
# This is the script that the cron job will execute
# Collect the graphs from the 'phats' machine
# 'Sleep' is there to prevent a conflict
scp -i ~/.ssh/id_rsa_phats foo@phats:/home/foo/Dev/Raspi/enviroplot/temp_range.png /var/www/html/
sleep 5
scp -i ~/.ssh/id_rsa_phats foo@phats:/home/foo/Dev/Raspi/enviroplot/temp_recent.png /var/www/html/The important bits here are:
scpis secure copy and allows use ofsshkeys.- I’m using
sshkeys for authentication, this prevents having to type my password repeatedly and offers secure copying and logins. 'Sleep'is there ascronwill execute both commands at the same time; better to have a slight delay and wait for the first command to finish.
sheevaplug crontab
# m h dom mon dow command
# This is the script that runs the bash script to collect the graphs from the 'phats' machine
*/15 * * * * sh /var/www/html/get_graphs.shThe cron job that runs every 15 minutes to execute the above get_graphs.sh script.
And Finally!
This is what the output looks like, snipped from a web page.
Yes, I know the axis on the first graph looks a bit odd, as I said I’m just starting out with gnuplot.
I hope the above helps someone !
Simon.