Data Collection, Logging, Plotting & Serving - A Simple Guide


#1

Hello,

I’ve noticed a couple of people asking lately, how to collect data from a HAT or pHAT and log it to a local file.

The routine below should be a good jumping off point for people to start with and does the following:

  • Uses python to collect data from a sensor (in my case an Enviro pHAT).
  • Writes the python output to a comma separated file.
  • Uses gnuplot to turn slices of the .csv file in to graphs, stored locally.
  • Instructs a web server on a different machine to pull those files into the /var/www folder.

A web page then easily serves these, as they always have the same name.

So, putting all of this together:

The Enviro pHAT sits on top of a pi Zero WH (named phats). It uses ssh keys for authentication.

The pi is connected to the internal network (and therefore the web server) and has a known IP address.

The webserver is named sheevaplug, it has a single user with no rights allocated. The user has to su to get rights.

On the pi there are a few files:

  • The python file that collects data from the Enviro pHAT (enviroplot.py).
  • The .csv file that the python file writes comma seperated values to (enviroplot.csv).
  • The gnuplot script(s) that interpret the .csv rows and produce the .png graphs (temp_range.plot, temp_recent.plot).
  • The sh file that wraps the python file and the gnuplot scripts together (enviroplot.sh).
  • A cron job then executes the sh file every 15 minutes.

enviroplot.py

#!/usr/bin/env python

import csv
from datetime import datetime
from envirophat import weather

dt = datetime.now()

full_date = dt.strftime(’%Y-%m-%d’)
full_time = dt.strftime(’%H:%M:%S’)
full_day = dt.strftime(’%A’)
year = dt.year
month = dt.month
date = dt.day
hour = dt.hour

temp = round(weather.temperature(), 2)
baro = round(weather.pressure(unit = ‘hPa’), 2)

csvWrite = full_date, full_time, full_day, year, month, date, hour, temp, baro
csvFile = open(’/home/foo/Dev/Raspi/enviroplot/enviroplot.csv’, ‘a’)

with csvFile:
writer = csv.writer(csvFile)
writer.writerow(csvWrite)

The important bits here are:

  • round(xxxx, 2) rounds to two decimal places.
  • The 'a' at the end of the csvFile line means append. The .csv file must exist first, even if it is empty to start with.

enviroplot.csv (Example of data collected)

full_date, full_time, full_day, year, month, date, hour, temp, baro
2019-01-23,22:15:01,Wednesday,2019,1,23,22,23.12,998.17
2019-01-23,22:30:01,Wednesday,2019,1,23,22,22.89,998.35
2019-01-23,22:45:02,Wednesday,2019,1,23,22,22.54,998.41
2019-01-23,23:00:02,Wednesday,2019,1,23,23,22.2,998.44
2019-01-23,23:15:01,Wednesday,2019,1,23,23,21.9,998.46
2019-01-23,23:30:01,Wednesday,2019,1,23,23,21.73,998.82
2019-01-23,23:45:01,Wednesday,2019,1,23,23,21.47,999.24
2019-01-24,00:00:01,Thursday,2019,1,24,0,21.18,999.26
2019-01-24,00:15:01,Thursday,2019,1,24,0,20.96,999.3
2019-01-24,00:30:02,Thursday,2019,1,24,0,20.92,999.32
2019-01-24,00:45:01,Thursday,2019,1,24,0,20.87,999.64
2019-01-24,01:00:01,Thursday,2019,1,24,1,20.63,999.82

I have deliberately collected the full date, full time, day, year, month, date, hour separately as I may want to analyse these elements later and don’t want to have to dissect the .csv string.
(The final two values are temperature and barometric pressure.)

The following two .plot examples are the gnuplot scripts.

Gnuplot is an incredibly versatile open source plotting piece of software and is available in the repos.

sudo apt install gnuplot

It is capable of creating 2D and 3D graphs and outputting to a variety of formats.

You can even plot ASCII graphs in the terminal with it!

You can create a graph with as little as one line, specifying simply where the input data is.

I’m still learning, so you may have better looking outputs with more practice.

temp_range.plot

reset

unset key
set key off

set terminal png

set datafile separator ","

set xdata time
set timefmt "%Y-%m-%dT%H:%M:%S"
set format x "%d/%m"

set xlabel "Date (day/month)"
set ylabel "Temp"

set style data points

set title "Temp Range"
set grid

plot '/home/foo/Dev/Raspi/enviroplot/enviroplot.csv' using 1:8

The important bits here are:

  • plot’ i.e. the location of the .csv file containing data.
  • Using 1:8’ i.e. use data columns 1 & 8

temp_recent.plot

reset

unset key
set key off

set terminal png

set datafile separator ","

set xdata time
set timefmt "%Y-%m-%dT%H:%M:%S"
set format x "%d/%m"

set xlabel "Date (day/month)"
set ylabel "Temp"

set style data line

set title "Recent Temp"
set grid

plot '< tail -n 672 /home/foo/Dev/Raspi/enviroplot/enviroplot.csv' using 1&2:8

The important bits here are:

  • tail -n 672’ i.e. take the last 672 data points ( 1 x week at 15 minute intervals).
  • using 1&2:8’ i.e. concatenate columns 1&2 as one data column and column 8 as the other.

enviroplot.sh

#!/bin/sh
# This is the script that the cron job will execute

# Run the python script to collect the variables
python /home/foo/Dev/Raspi/enviroplot/enviroplot.py

#### Create the graph images from gnuplot
# 'Sleep' is there to prevent a conflict
gnuplot /home/foo/Dev/Raspi/enviroplot/temp_range.plot > /home/foo/Dev/Raspi/enviroplot/temp_range.png
sleep 5
gnuplot /home/foo/Dev/Raspi/enviroplot/temp_recent.plot > /home/foo/Dev/Raspi/enviroplot/temp_recent.png

The important bits here are:

  • Runs the python script that writes the Enviro pHAT data to the .csv file
  • Calls the gnuplot file(s) and outputs it to a named .png file i.e. creates the graph image file(s).
  • 'Sleep' is there as cron will execute both commands at the same time; better to have a slight delay and wait for the first command to finish.

phats crontab

# m h  dom mon dow   command
# This is the script that runs the bash script to collect data and create graphs
*/15 * * * * /home/foo/Dev/Raspi/enviroplot/enviroplot.sh

The cron job that runs every 15 minutes to execute the above enviroplot.sh script.

The above will get you a working pair of graphs that are refreshed every 15 minutes.

But they will be on the pi which may be headless or unattached to any sort of display.

Much better to put the images on the interweb where you can watch your house getting hotter when it’s burning.

The next two files are on the web server and comprise of:

  • A sh script to pull the files from the pi over the local network.
  • A cron job to do that every 15 minutes.

I have to pull the files into the web server, rather than push the files from the pi because, as I said earlier the server only has a single user with no rights and I couldn’t find a way to connect to the server using ssh, then pass a password to get rights. (Probably is a way, but too late now.)

/var/www/get_graphs.sh

#!/bin/sh
# This is the script that the cron job will execute

# Collect the graphs from the 'phats' machine
# 'Sleep' is there to prevent a conflict

scp -i ~/.ssh/id_rsa_phats foo@phats:/home/foo/Dev/Raspi/enviroplot/temp_range.png /var/www/html/
sleep 5
scp -i ~/.ssh/id_rsa_phats foo@phats:/home/foo/Dev/Raspi/enviroplot/temp_recent.png /var/www/html/

The important bits here are:

  • scp is secure copy and allows use of ssh keys.
  • I’m using ssh keys for authentication, this prevents having to type my password repeatedly and offers secure copying and logins.
  • 'Sleep' is there as cron will execute both commands at the same time; better to have a slight delay and wait for the first command to finish.

sheevaplug crontab

# m h  dom mon dow   command
# This is the script that runs the bash script to collect the graphs from the 'phats' machine
*/15 * * * * sh /var/www/html/get_graphs.sh

The cron job that runs every 15 minutes to execute the above get_graphs.sh script.

And Finally!

This is what the output looks like, snipped from a web page.

Yes, I know the axis on the first graph looks a bit odd, as I said I’m just starting out with gnuplot.

I hope the above helps someone !

Simon.


#2

Thanks for that, I for one am interested in doing something similar. Will have to sit down and take a god long look at what you did when I get some free time. =)


#3

To be honest, the explanation is a lot longer than the actual code !

It boils down to:

  • Collect data from device (obviously the python will vary per sensor).
  • Write the collected data to a file.
  • Read the file using gnuplot (arguably easier to understand than python, but that may be me).
  • Make a pretty graph and put it somewhere.

Anyway, hope it gets you started.

Shout if you need a hand and I’ll try to assist.

Cheers,

Simon.