← Back to Lectures
HalesAir Logo
HalesAir · Session 04

Data Logging &
Transmission

Halesowen College · T Level Data Analytics

CSV file logging to Pico flash MicroPython file I/O Wi-Fi connection & NTP time sync HTTP POST to remote server

Where We Are

Building the complete data pipeline — one step at a time

BME680
Sensor
→
MicroPython
Pico W
→
CSV File
(today: step 1)
→
Wi-Fi / NTP
(today: step 2)
→
Server / API
(today: step 3)
→
Dashboard
(Session 06)
Sessions 01–02

Hardware, sensors, wiring, I2C. You built the node.

Session 03

MicroPython basics. You read live data and printed it.

Session 04 (today)

Save that data locally, sync the clock, send it to the internet.

💾

Part 1
Logging to CSV

Structured data · Persistent storage · Import into Excel/Python

CSV — Comma-Separated Values

The format

  • Plain text — one row per reading, values separated by commas
  • First row is usually a header describing each column
  • Universally compatible — Excel, Python pandas, R, Google Sheets
  • The standard format for environmental monitoring data

CSV is intentionally simple. Its strength is that every piece of software on the planet can read it. We'll analyse it in Python in Session 05.

Example CSV output
timestamp temp_c hum_pct pres_hpa gas_ohm
2026-03-10 09:00:0218.3621013.248200
2026-03-10 09:00:1218.4611013.249100
2026-03-10 09:00:2218.5611013.349800
Each row = 10 seconds of data. At one row per 10 s → ~8,640 rows per day

Writing to a File in MicroPython

# Write header once (first run only) with open("log.csv", "w") as f: f.write("timestamp,temp_c,hum_pct," "pres_hpa,gas_ohm\n") # Append readings (use "a" mode) while True: temp = sensor.temperature hum = sensor.humidity pres = sensor.pressure gas = sensor.gas ts = get_timestamp() # see next slide row = f"{ts},{temp:.2f},{hum:.1f}," f"{pres:.1f},{gas}\n" with open("log.csv", "a") as f: f.write(row) time.sleep(10)

File modes

"w" — write

Creates the file (or overwrites if it exists). Use once for the header row.

"a" — append

Adds to the end of the file without deleting existing content. Use every loop.

"r" — read

Opens to read only. We'll use this in Session 05 to analyse the collected data.

Flash limit: The Pico has 2 MB of flash. At ~60 bytes/row × 8640 rows/day, that's ~500 KB/day — roughly 4 days before you need to download and clear the file.

📡

Part 2
Wi-Fi & Time Sync

Connecting to the network · NTP · Real timestamps

Connecting to Wi-Fi

import network, time SSID = "YourHotspot" PASSWORD = "YourPassword" def connect_wifi(): wlan = network.WLAN(network.STA_IF) wlan.active(True) if not wlan.isconnected(): print("Connecting…") wlan.connect(SSID, PASSWORD) while not wlan.isconnected(): time.sleep(0.5) print("Connected:", wlan.ifconfig()) return wlan connect_wifi()

Security note

Never hardcode passwords in code you share or commit to Git. Store credentials in a separate secrets.py file — import it, don't paste it.

# secrets.py (keep private!) SSID = "YourHotspot" PASSWORD = "YourPassword" # main.py from secrets import SSID, PASSWORD

In the classroom: Use your phone's mobile hotspot, not the college Wi-Fi — enterprise 802.1x authentication is not supported by MicroPython's network library.

NTP — Getting Accurate Time

Network Time Protocol — used by every computer on the internet to stay synchronised

import ntptime, time, machine def sync_time(): ntptime.settime() # sets RTC from NTP print("Time synced") def get_timestamp(): t = time.localtime() return (f"{t[0]}-{t[1]:02d}-{t[2]:02d} " f"{t[3]:02d}:{t[4]:02d}:{t[5]:02d}") connect_wifi() sync_time() print(get_timestamp()) # → 2026-03-10 09:00:02

How NTP works

  • Pico contacts an NTP server (default: pool.ntp.org)
  • Server returns the current UTC time to millisecond precision
  • MicroPython's RTC is updated — time.localtime() is now accurate
  • One sync per session is enough — RTC stays accurate for hours

UTC vs local: NTP gives you UTC. For UK data during BST (summer), add 1 hour. During GMT (winter), add 0. We'll handle this in the analysis scripts.

Source: IETF RFC 5905 — NTPv4. Pool: pool.ntp.org
🌐

Part 3
Sending Data to the Internet

HTTP POST · REST APIs · Real-time pipeline

How HTTP Works (briefly)

The request/response model

  • The internet runs on HTTP — the same protocol as web browsing
  • GET — ask the server for data (like loading a webpage)
  • POST — send data TO the server (like submitting a form)
  • We use POST to send each sensor reading to our API endpoint

When you visit a website, your browser sends a GET request. When you log in, it sends a POST request with your credentials. Our sensor does the same — but with air quality readings.

REST API basics

  • An API endpoint is a URL that accepts data
  • Data is typically sent as JSON — a text format similar to Python dicts
  • Server responds with a status code: 200 OK = success
  • 401 = auth error · 500 = server error
// JSON payload we will POST: { "unit_id": "sensor_A", "temp_c": 18.4, "hum_pct": 61.0, "pres_hpa": 1013.2, "gas_ohm": 49100 }

HTTP POST in MicroPython

import urequests, ujson ENDPOINT = "https://api.halesair.site/readings" API_KEY = "your-secret-api-key" def post_reading(temp, hum, pres, gas): payload = { "unit_id": "sensor_A", "temp_c": round(temp, 2), "hum_pct": round(hum, 1), "pres_hpa": round(pres, 1), "gas_ohm": gas, } headers = { "Content-Type": "application/json", "X-API-Key": API_KEY, } r = urequests.post( ENDPOINT, data=ujson.dumps(payload), headers=headers ) print(f"Status: {r.status_code}") r.close()

Key points

  • urequests — lightweight HTTP for MicroPython (install via Thonny)
  • ujson.dumps() — converts Python dict to JSON string
  • API key in the header — proves the request comes from our sensor
  • Always r.close() to free the socket

Error handling: Wrap in try/except — if Wi-Fi drops, fall back to local CSV logging rather than crashing the sensor.

Session 04 uses a test endpoint — we'll confirm data arrives on the server before going live.

Putting It All Together

The complete main.py flow for a deployed sensor node

# 1. Boot → connect Wi-Fi → sync time connect_wifi() sync_time() # 2. Write CSV header (once) write_header() # 3. Main sensor loop while True: temp, hum, pres, gas = read_sensor(sensor) ts = get_timestamp() append_csv(ts, temp, hum, pres, gas) # local backup try: post_reading(temp, hum, pres, gas) # remote send except Exception as e: print(f"Send failed: {e} — saved locally") time.sleep(10)

Resilience pattern: Always write to local CSV first, then attempt remote send. If the network is down, data is safe. Next session: we'll add re-transmission of any missed readings.

Activity: Build the Full Pipeline

⏱ 30 minutes

By the end you have a working data pipeline: sensor → CSV → server.

1

(8 min) Add CSV logging to your Session 03 script. Run it for 60 seconds, then download log.csv via Thonny's file manager. Open in Excel — confirm you see tidy columns.

2

(7 min) Add the connect_wifi() and sync_time() functions. Connect to your phone's hotspot. Verify that get_timestamp() now returns the correct date and time.

3

(10 min) Add post_reading() using the test endpoint URL provided. Confirm you see Status: 200 in Thonny. Check the live dashboard — your data should appear within seconds.

4

(5 min) Wrap the POST in a try/except block. Disconnect from Wi-Fi (turn off hotspot), run the loop, confirm it still writes to CSV and prints the fallback message.

💡 If you get OSError: -2 on connect, check your SSID spelling is exact (case-sensitive). If you get an empty CSV, check you opened with "a" not "w" in the loop.

Coming Up — Session 05

Date TBC · Data Cleaning & Analysis

What we'll cover

Loading your CSV into Python pandas, cleaning anomalous readings, computing summary statistics, and visualising temperature and IAQ trends as your first research outputs.

Before Session 05: Leave your sensor running overnight. Bring at least 24 hours of CSV data to the session.

Data target: We need at least 500 rows per site for meaningful statistical analysis. Keep your sensor online!

Questions? Get in touch:

jwilliams.science · HalesAir Project