Unnamed home automation software


background

For the last few years, I’ve had a recurring obsession with automating my house. It started with overpriced ZigBee based lights and has continued with an eclectic selection of software, hardware, and labour hours.

The first thing I wanted was a central automation system - a system that makes decisions based on the state of sensors related to the house.

Being a proud and stubborn man, I wanted to do everything myself.

I wrote my own home automation and management system in Golang, but I over-complicated how sensor changes were interpreted. I tried to implement a complicated presence guessing system to figure out where people are in the house. The server only spoke a weird protocol (not HTTP) and after starting on a website for it, I started to get tired of writing software and thinking hard without seeing results.

Midway through last year, I gave up in this post.

home assistant

I installed HomeAssistant on a Raspberry Pi 4.

The web UI is pretty, usable and can be customised easily.

The system remembers sensor states and can manage automations alongside the dashboard functionality. It supports many different devices.

why not…

The other large open source movement was OpenHAB, but I didn’t like the 90s feel of the web UI, and couldn’t really configure anything without opening a text editor.

HomeAssistant gets regular patches, lots of integrations and good history of announcing security issues.

automations

Here’s how HomeAssistant thinks of home automation.

The state of things in your house can be changed with automations. Based on sensor states (on, off, transition from 900 to 700, below 900), you can take certain actions. For example, I have an automation that turns on the hall light when someone walks past a motion sensor, but only when it’s dark outside. This is done with a trigger, a condition and an action. There’s another automation that turns the hall light off 10 seconds after the motion sensor reports ‘no activity’. This means a person in the hallway will continually trigger the motion sensor, keeping the light on until they leave, but only if it’s dark outside.

That’s a lot to take in, but it’s the core principle behind most automation systems, so it’s good to understand.

I looked at their documentation for sending sensor information. Their HTTP API was too boring and verbose for my liking, especially since I wanted to work with resource constrained microprocessors.

MQTT

I thought back to the time I tried to reverse engineer my Chinese robot vacuum cleaner, it talked over MQTT.

I installed and firewalled an MQTT server on the same Pi 4. MQTT is a very simple messaging protocol, designed for use in cheap, low power devices. MQTT packets have a topic and a message (there are other bits, but let’s ignore them for now). You can subscribe to a topic and publish to a topic. Topics have paths like this “home/binary_sensor/pir/state” and subscriptions can contain wildcards like this “home/#”. HomeAssistant supports MQTT, and while it’s possible to mangle MQTT messages and topics from existing devices, you can follow their MQTT topic and message format to integrate quicker.

The final product

In the following sections, I go into the specifics of the features I implemented on what timeline. The most interesting part is knowing what the system can actually do, so here’s an overview of how our home automation is currently configured. The planter is currently offline because I need to do some maintenance. These are the capabilities and devices I use to obtain them.

Study (Pi, Arduino x2, ESP32)

  • Light turns on and off based on movement near a microwave sensor in the last 2 minutes and darkness.
  • Provides per minute temperature and relative humidity data
  • Provides house-wide ‘outside light intensity’ which is transformed into a darkness on/off switch
  • ESP32 measures ultrasonic sensors to detect my posture and graph how my posture does throughout the day
  • Has an LED cube that is updated based on house state changes.

Hall (ESP32)

  • Light turns on and off based on movement near a PIR sensor in the last 10 seconds and darkness.
  • Provides per minute temperature and relative humidity data

Entrance (ESP32+LiPo battery)

  • Light turns on and off based on movement near a PIR sensor in the last 30 seconds. It’s always dark in the entrance.
  • Has an extended low power mode

Mushroom box (ESP32)

  • Has no control over fans permanently on
  • Provides per minute temperature and relative humidity data
  • Provides per minute volitile organic compound and estimated CO2 data

Planter (ESP32+LiPo battery)

  • Provides data every 30 minutes
  • Multiple capacitive soil sensors
  • Multiple temperature and humidity sensors
  • No power control for the above, causing power loss/drain when ESP32 is asleep
  • Has an extended low power mode

Bedroom (Pi)

  • Light turns on and off based on movement near a PIR sensor in the last 45 seconds and darkness.
  • Pressure sensitive resistors and weight scales were used to guess if we were in the bed
  • The above doesn’t function properly

Printer (Pi)

  • Light turns on and off based on movement near a PIR sensor in the last 45 seconds and darkness.
  • Provides per minute temperature and relative humidity data
  • Provides control over two mains RF switches - one for a light and the other for my 3D printer (giving me a remote emergency kill switch)
  • Provides 3D printer control via OctoPrint

This is pretty cool, but my next house will be more comprehensive. Now I have a taste for automation, our next garden and mushroom tent project will be heavily automated, providing great statistics, graphs and predictive data to increase our harvests in any way possible. Our house will likely have more nodes in it, and hopefully I’ll solve some of the power issues, so I won’t need to replace batteries every two weeks (as my current configuration demands).

device specifics and implementation details

raspberry pi (zer0)

I already had a few Raspberry Pi Zero devices stuck to our windows to watch for motion outside our apartment. This is handy for monitoring our 3D printed bird feeder stuck on the window with an IKEA toilet roll holder. One day I might add image recognition to it, so I can tell who is at the front door based on the last recognised entity to pass the camera.

I got to work writing a Golang MQTT announcer for sensor data. Instead of using configuration files, I built different applications for each computer’s sensors and copied them over SSH+wifi.

This received a few features since April 2020. Let’s go over them from the Git history, which is easier than trying to remember what I did.

April 25-26

  • Wrote the base application, with support for MQTT publishing and HomeAssistant device configuration
  • Implemented a temperature and humidity sensor over MQTT
  • Implemented a GPIO watcher, letting me send events when a pin on the Pi switches between high/low
  • Motion sensing pins emit motion sensor events over MQTT

May 3-6, 16

  • Implemented MQTT subscriber to listen for commands
  • Implemented power socket control over 443MHz
  • Transformed MQTT events into LED cube commands and sent over serial to an Arduino for 433Mhz broadcast

June 17

  • Added support for MCP23017

August 14, 19

Sept 3

  • Added support for requesting lumens from an Arduino with a light sensitive resistor soldered to an analog pin and VOut

Sept 17 (and a few days before)

  • Implemented HomeAssistant light commands
  • Implemented a Bluetooth 4.2 Light Strip for use in my kitchen

October 5-12

  • Finding and fixing some recurring bugs
  • Set MQTT configuration to mark the device as unavailable after 24h of no activity

Not a bad set of features, in my opinion. The bluetooth light strip took some reverse engineering, I had to look at the Android source code for the app in Ghidra to extract the Bluetooth commands. After that, it was fairly easy to control the lights after pairing with them.

ESP32

As I installed these Raspberry Pi Zero computers, I started to get concerned about their price, longevity and power cost versus my utilisation. I didn’t really need a lot of these higher powered, high memory devices. My goals are relatively simple - sensors in rooms that can communicate to MQTT to trigger devices. The other challenge I was beginning to consider were rooms without power, where I need devices that could last on battery power for a few days.

With this criteria in mind, I set out looking for simple, lower power devices that are very easy to flash/program, very easy to integrate with sensors via GPIO. I didn’t have any operating system system or memory requirements. I was certain I wanted a device that had WiFi or RF support built in to the chip. I preferred the broad support of existing protocols like Bluetooth or WiFi over other wireless protocols, like LoRa.

My search very quickly led me to the ESP32.

  • Cheap: These SoC devices are cheaper than most components at around 1.5-2GBP per device.
  • RF: They also support a lot of WiFi and Bluetooth 4.2 features, including less common WiFi features, like raw frames
  • Size: Smaller device footprint than a Raspberry Pi Zero
  • Flash: 4MB of onboard flash is neat - ‘partitions’ are used to separate areas of flash storage
  • CPU: Power hungry but fast, ESP32’s dual core handles wireless and core tasks with one processor and user tasks on the other
  • Memory: 520KB of SRAM (shared Code/Data storage)
  • Peripheral features: RF, CAN bus, PWM, hall effect sensor, Ethernet MAC, SD Card, UART, SPI, I2S, AES, SHA-2, RSA, ECC, HW-RNG

While these devices are memory constrained, you can write efficent C programs that will fit on them. Applications can get up to 1.9MB, if you’re careful about how you store certain values.

Anyway, I started an Arduino project and got to work. Let’s take a quick trip down my commit history.

July 18-20

  • My code is recorded as “initial commit”, “fiddling about”, and “i did it”
  • The first ‘node’ can send ESPNOW packets with sensor information, specifically analog measurement on a pin, and temp, humidity
  • The base stations exist
    • ‘serial’ listens to ESPNOW packets, translates them to MQTT messages and forwards them over serial
    • ‘bridge’ listens to serial and forwards that to MQTT over regular WiFi
  • Using ESPNOW avoids the expense of WiFi handshakes but provides easy and encrypted 250byte messages

July 26

  • I rewrote the planter node into more reusable code
  • I buffered messages for ESPNOW instead of sending an ESPNOW message for each sensor value emitted (there are a few bugs)

July 29, 31

  • Fixing bugs, config I alluded to before

September 2-8, 11

  • Audio sensor added but doesn’t work with my deep sleep code so had to give up
  • Improved deep sleep and fixed receiver bugs

September 22

  • Committed VOC sensor improvements (volitile organic compound, these require calibration and permanent storage of their baselines, which was implemented here)

October 31

  • Fixed more bugs, added support for weight sensors

November 2

  • Got builds from the terminal working

March 6-7 2021

  • Committed my posture detection via ultrasonic sensors from a few months ago
  • ‘bridge’ can update over the air
  • Wrote a lazy build script which builds, updates an HTTP server and publishes an MQTT message to a firmware update topic
  • Added versions in semvar format (1.0.0.0), other improvements

March 8

  • ‘serial’ can update over the air
  • Build script automatically increases build version
  • Rewrote ‘serial’ code, it’s neater
  • ‘node’ listens to ota messages

March 9

  • changed from 1.0.0.0 to 1 versioning (increasing uint)
  • ‘node’ version 18 or higher can update over the air
  • the above means i only have to visit all my nodes with a laptop once to update them… then they can update themselves!

March 10

  • ‘node’ battery improvements to reduce time spent waiting for version announcement to return
  • ‘node’ will wait for version response after first boot, so rebooting a node will update firmware immediately
  • Build script waits for ota update complete messages via MQTT subscribe
  • ‘serial’ can emit stats and build script can request them in a loop

Pretty much all you needed for a functional system was written in July, but improvements over time really helped with battery conservation and performance.

Investigating an issue where a battery powered ESP32 waking up from deep sleep causes an (assumed) voltage dip which triggers my motion sensor, causing the ESP32 to stay awake for a long time every 240 seconds. A digital oscilioscope is coming in the mail with hopefully enough resolution to detect the issue, if it’s that.

My next post will likely involve more details about setting up my automation inside a new home for the first time in a while.