Basic intro for uploading to Luftdaten / Air quality project

New user here and hope to add some content that helps out other newbies. Heads-up, other than common sense and monkey sees / monkey does, I have no experience with either Python, nor linux, so Google was my friend last days ;-) Not sure if I should post this as a ‘Project’, but here we go.

I bought the Enviro+ and the PMS5003 Particulate Matter Sensor and want to contribute to the citizen scient project in collecting particulate matter data. A great tutorial was already provided via learn.pimoroni.com.

This excellent tutorial gave me (and will give you) a great start to start playing around with this. The provided example works, but I have added some features I would like to share. For me these were my first steps in the wondrous world of python and communicating with the hardware and I have already learned a lot.

Biggest changes I made were:

  1. extended info shown on LCD, it shows SSID, IP & last upload date/time as well now
  2. changed the output to console when you run it there, shows more information, also including the data that is not sent to Luftdaten
  3. created two variables so you can easily set/change the frequency for the console updates and the uploads to Luftdaten

#1 was handy for me, since it sometimes switches to another wifi here at home, so I can see on the LCD as well how to SSH to it.
#3 is just easy during testing. On line 36/37 you will find below variables:
update_CLI = 30
update_LuftDaten = 150
update_CLI is the console update in seconds, change that to 5 and it will, independently from uploading update more. Sending data to Luftdaten is set to 150, so around 2.5 minutes.

I am sure I’ll make more changes, but this might help others preventing to re-invent the wheel!

Still on my personal to-do list before I go truly live:

  • waiting on an extension cable to get the enviro+ away from the RPI Zero W. The heat of the CPU seriously messes up temperature & humidity readings. The first you can compensate somewhat, but for RH it’s useless.
  • make a proper casing to mount it outside
  • find out how to log ALL available date for personal use and make that human readable through a website. Thinking about uploading small xml-files using sFTP, that would also cover data-loss of wifi drops.

Kind regards,
Remco

Feel free to use at your own risk and let me know if it’s helpful!

  • save below file to your /examples/ folder: /home/pi/enviroplus-python/examples/ with a name like rmb_luftdaten.py (or anything you recognize)
  • start as any script, ie: python /home/pi/enviroplus-python/examples/rmb_luftdaten.py
  • there was some trial and error involved, open to code optimization! :-)

Tried to attach it as .py, but that didn’t work:

#!/usr/bin/env python

# RMB Changes V2:
# - clear console in between updates (import OS)
# - show date/time when started (import datetime)
# - formatted console output
# - added extra info to console output (IP, SSID, etc)
# - added extra info to LCD output (SSID / IP)
# RMB Changes V3:
# - created extra variables to use independent output frequency to console
#   and to Luftdaten (change vars update_CLI & update_LuftDaten)
# - show all Enviro+ data, also what's not sent to LuftDaten
# - added tracking of last transmitted values
# - show last upload date/time on both console & LCD

import os
import requests
import ST7735
import sys
import time
from bme280 import BME280
from datetime import datetime
from enviroplus import gas
from PIL import Image, ImageDraw, ImageFont
from pms5003 import PMS5003, ReadTimeoutError
from subprocess import PIPE, Popen, check_output

try:
    from smbus2 import SMBus
except ImportError:
    from smbus import SMBus

# RMB - extra values for LCD &  console output + extra variables for update frequency
# values update_CLI (console) and update_LuftDate is in seconds
update_CLI = 30
update_LuftDaten = 150
last_update = '> xx-xx xx:xx'
last_P1 = 'n/a'
last_P2 = 'n/a'
last_Temp = 'n/a'
last_Humi = 'n/a'
last_Pres = 'n/a'
started = datetime.now().strftime("%d-%m-%Y %H:%M:%S")
ssid = check_output(['iwgetid']).split('"')[1]
hostname = check_output(['hostname', '-I']).rstrip()

# RMB clear console before we start!
os.system('clear')

# RMB original text updated
print("""Adaptation on luftdaten.py by Remco Brand

Reads temperature, pressure, humidity, PM2.5, and PM10 from Enviro plus and
sends data to Luftdaten Citizen Science project.

Note: you need to register at: https://meine.luftdaten.info/ and enter your
Raspberry Pi serial number that's displayed on the Enviro plus LCD along
with the other details before the data appears on the Luftdaten map.

Press Ctrl+C to exit!
""")

bus = SMBus(1)

# BME280 temperature/pressure/humidity sensor
bme280 = BME280()

# PMS5003 particulate sensor
pms5003 = PMS5003()

# Create ST7735 LCD display class
disp = ST7735.ST7735(
    port=0,
    cs=1,
    dc=9,
    backlight=12,
    rotation=270,
    spi_speed_hz=10000000
)

# Check for Wi-Fi connection
def check_wifi():
    if check_output(['hostname', '-I']):
        return True
    else:
        return False

# Initialize display
disp.begin()

WIDTH = disp.width
HEIGHT = disp.height

# Read values from BME280 and PMS5003 and return as dict
def read_values():
    values = {}
    cpu_temp = get_cpu_temperature()
    raw_temp = bme280.get_temperature()
    comp_temp = raw_temp - ((cpu_temp - raw_temp) / comp_factor)
    values["temperature"] = "{:.2f}".format(comp_temp)
    values["pressure"] = "{:.2f}".format(bme280.get_pressure() * 100)
    values["humidity"] = "{:.2f}".format(bme280.get_humidity())
    try:
        pm_values = pms5003.read()
        values["P2"] = str(pm_values.pm_ug_per_m3(2.5))
        values["P1"] = str(pm_values.pm_ug_per_m3(10))
    except ReadTimeoutError:
        pms5003.reset()
        pm_values = pms5003.read()
        values["P2"] = str(pm_values.pm_ug_per_m3(2.5))
        values["P1"] = str(pm_values.pm_ug_per_m3(10))
    return values

def read_valuesCL():
    valuesCL = {}
    cpu_temp = get_cpu_temperature()
    raw_temp = bme280.get_temperature()
    comp_temp = raw_temp - ((cpu_temp - raw_temp) / comp_factor)
    valuesCL["tempCPU"]             = "{:.2f}".format(cpu_temp)
    valuesCL["tempRaw"]             = "{:.2f}".format(raw_temp)
    valuesCL["tempComp"]            = "{:.2f}".format(comp_temp)
    valuesCL["pressure"]            = "{:.2f}".format(bme280.get_pressure())
    valuesCL["humidity"]            = "{:.2f}".format(bme280.get_humidity())
    gas_values = gas.read_all()
    valuesCL["oxidising"]       = "{:.2f}".format(gas_values.oxidising / 1000)
    valuesCL["reducing"]        = "{:.2f}".format(gas_values.reducing / 1000)
    valuesCL["nh3"]             = "{:.2f}".format(gas_values.nh3 / 1000)
    try:
        pm_values = pms5003.read()
        valuesCL["P01"]             = str(pm_values.pm_ug_per_m3(1.0))
        valuesCL["P25"]             = str(pm_values.pm_ug_per_m3(2.5))
        valuesCL["P10"]             = str(pm_values.pm_ug_per_m3(10))
    except ReadTimeoutError:
        pms5003.reset()
        pm_values = pms5003.read()
        valuesCL["P01"] = str(pm_values.pm_ug_per_m3(1.0))
        valuesCL["P25"] = str(pm_values.pm_ug_per_m3(2.5))
        valuesCL["P10"] = str(pm_values.pm_ug_per_m3(10))
    return valuesCL

# Get CPU temperature to use for compensation
def get_cpu_temperature():
    process = Popen(['vcgencmd', 'measure_temp'], stdout=PIPE, universal_newlines=True)
    output, _error = process.communicate()
    output = output.decode()
    return float(output[output.index('=') + 1:output.rindex("'")])

# Get Raspberry Pi serial number to use as ID
def get_serial_number():
    with open('/proc/cpuinfo', 'r') as f:
        for line in f:
            if line[0:6] == 'Serial':
                return line.split(":")[1].strip()

# Display Raspberry Pi serial and Wi-Fi status on LCD
def display_status():
    wifi_status = "OK" if check_wifi() else "disconnected"
    text_colour = (170, 170, 170)
    back_colour = (0, 0, 0) if check_wifi() else (85, 15, 15)
    id = get_serial_number()
# RMB included SSID, IP-address & last update as well on the LCD
    message = "S#: {}\nLast upload: {}\nSSID: {}\nIP: {}".format(id,last_update,ssid,hostname)
    img = Image.new('RGB', (WIDTH, HEIGHT), color=(0, 0, 0))
    draw = ImageDraw.Draw(img)
    size_x, size_y = draw.textsize(message, font)
    x = (WIDTH - size_x) / 2
    y = (HEIGHT / 2) - (size_y / 2)
    draw.rectangle((0, 0, 160, 80), back_colour)
    draw.text((x, y), message, font=font, fill=text_colour)
    disp.display(img)

def send_to_luftdaten(values, id):
    pm_values = dict(i for i in values.items() if i[0].startswith("P"))
    temp_values = dict(i for i in values.items() if not i[0].startswith("P"))

    resp_1 = requests.post("https://api.luftdaten.info/v1/push-sensor-data/",
             json={
                 "software_version": "enviro-plus 0.0.1",
                 "sensordatavalues": [{"value_type": key, "value": val} for
                                      key, val in pm_values.items()]
             },
             headers={
                 "X-PIN":    "1",
                 "X-Sensor": id,
                 "Content-Type": "application/json",
                 "cache-control": "no-cache"
             }
    )

    resp_2 = requests.post("https://api.luftdaten.info/v1/push-sensor-data/",
             json={
                 "software_version": "enviro-plus 0.0.1",
                 "sensordatavalues": [{"value_type": key, "value": val} for
                                      key, val in temp_values.items()]
             },
             headers={
                 "X-PIN":    "11",
                 "X-Sensor": id,
                 "Content-Type": "application/json",
                 "cache-control": "no-cache"
             }
    )

    if resp_1.ok and resp_2.ok:
        return True
    else:
        return False

# Compensation factor for temperature
comp_factor = 1.35

# Raspberry Pi ID to send to Luftdaten
id = "raspi-" + get_serial_number()

# Width and height to calculate text position
WIDTH = disp.width
HEIGHT = disp.height

# Text settings
font_size = 14
#font = ImageFont.truetype("fonts/Asap/Asap-Bold.ttf", font_size)
# RMB changed relative to full path
font = ImageFont.truetype("/home/pi/enviroplus-python/examples/fonts/Asap/Asap-Bold.ttf", font_size)

# Display Raspberry Pi serial and Wi-Fi status
# RMB added extra data here so it shows SSID / hostname too
print("RPI serial:          {}".format(get_serial_number()))
print("Started:             " + started)
print("Wi-Fi:               {}".format("Connected" if check_wifi() else "Disconnected"))
print("SSID / IP-address:   " + ssid + " / " + hostname)

time_since_update = 0
update_time = time.time()

# RMB added extra values for secondary loop to console
CLI_time_since_update = 0
CLI_update_time = time.time()

# Main loop to read data, display, and send to Luftdaten
# RMB biggest changes here. Added a lot more data and made the update frequency for
# console & upload independent from each other.
while True:
    try:
        time_since_update = time.time() - update_time
        CLI_time_since_update = time.time() - CLI_update_time
        values = read_values()
        valuesCL = read_valuesCL()

# Update to screen
        if CLI_time_since_update > update_CLI-1:
            ssid = check_output(['iwgetid']).split('"')[1]
            hostname = check_output(['hostname', '-I']).rstrip()
            cpu_temp = get_cpu_temperature()
            os.system('clear')
            print ("RPI serial: {}".format(get_serial_number()) + " | Script started:     " + started)
            print ("Wi-Fi:      {}".format("Connection OK" if check_wifi() else "NO Connection") + "    | SSID / IP-address:  " + ssid + " / " + hostname)
            print ('\nRefresh rates: console = {} seconds / upload to LuftDaten.info =  {} seconds'.format(update_CLI,update_LuftDaten))
            print ("\nData for " + datetime.now().strftime("%d-%m %H:%M:%S"))
            print ("Particulates 1.0/2.5/10um:    {} / {} / {} ug/m3".format(valuesCL.get('P01'),valuesCL.get('P25'),valuesCL.get('P10')) )
            print ("Temperature CPU:              {} C".format(valuesCL.get('tempCPU')) )
            print ("Temperature Compensated:      {} C (correction factor: {})".format(valuesCL.get('tempComp'),comp_factor) )
            print ("Atmospheric pressure:         {} mBar".format(valuesCL.get('pressure')) )
            print ("Relative humidity:            {} %".format(valuesCL.get('humidity')) )
            print ("NH3 / Reduced / Oxidized:     {} / {} / {} kOhm".format(valuesCL.get('nh3'),valuesCL.get('reducing'),valuesCL.get('oxidising')) )
            print ("\nBelow data was uploaded to Luftdaten {}".format(last_update) )
            print ("Particulates 2.5 / 10 um:     " + last_P2 + " ug/m3 / " + last_P1 + " ug/m3")
            print ("Temp / Humidity / Pressure:   {}C / {}% RH / {} Pa".format(last_Temp,last_Humi,last_Pres))
            CLI_update_time = time.time()

# Update to luftdaten
        if time_since_update > update_LuftDaten-1:
            resp = send_to_luftdaten(values, id)
            update_time = time.time()
            last_update = '> ' + datetime.now().strftime("%d-%m %H:%M")
            last_P1 = values.get('P1')
            last_P2 = values.get('P2')
            last_Temp = values.get('temperature')
            last_Humi = values.get('humidity')
            last_Pres = values.get('pressure')
            print("\nResponse: {}".format("OK - data sent" if resp else "failed"))
        display_status()

    except Exception as e:
        print(e)

Welcome to the forum and thanks for the information. I’m going to play with it when I can.