Files

Fredrik Wahlberg c1519b3eb5 Update ROADMAP: Document Phase 6 lessons learned

Key discoveries during systemd service implementation:
- AmbientCapabilities doesn't work in user services
- NoNewPrivileges prevents file capabilities
- Must use setcap with readlink -f on actual binary

2025-12-27 14:18:28 +01:00

18 KiB

Raw Blame History

ROADMAP: Modernizing Sensorpajen

Overview

This roadmap outlines the migration from the current tmux/cron-based system to a modern systemd service running on Raspberry Pi.

Migration Date: Started December 27, 2025
Target Completion: TBD

Current State

What We Have

LYWSD03MMC.py: Main Bluetooth sensor reader
temperatur_koksfonstret.py: DHT11 sensor reader (to be removed)
bluetooth_utils.py: Bluetooth utility functions
sensorer.ini: MAC address to sensor name mapping
sendToMQTT.sh: MQTT publishing callback (hardcoded credentials)
startup.sh/sensorer.sh: tmux-based startup scripts
Cron jobs for scheduling

Known Issues

MQTT credentials hardcoded in shell scripts
Legacy pirate_audio references in startup.sh
Manual tmux orchestration
Mixed configuration sources
DHT11 functionality to be removed

Target Architecture

Final Structure

sensorpajen/
├── src/
│   └── sensorpajen/
│       ├── __init__.py
│       ├── main.py              # Entry point
│       ├── config.py            # Configuration management
│       ├── sensor_reader.py     # Bluetooth sensor logic
│       ├── mqtt_publisher.py    # MQTT publishing
│       └── utils.py             # Utilities (from bluetooth_utils.py)
├── config/                      # Configuration directory (relative)
│   ├── sensors.json.example     # Sensor mapping template
│   ├── sensorpajen.env.example  # Environment file template
│   ├── sensors.json             # Actual sensor mapping (not in git)
│   └── sensorpajen.env          # Actual environment file (not in git)
├── debian/                      # APT package files
│   ├── control
│   ├── rules
│   ├── changelog
│   └── ...                      # Other Debian package files
├── pyproject.toml               # Project metadata and dependencies
├── requirements.txt             # Dependencies (bluepy, paho-mqtt)
├── README.md                    # Updated documentation
├── AGENTS.md                    # Agent guidelines
├── ROADMAP.md                   # This file
├── legacy/                      # Legacy scripts (moved here temporarily)
│   ├── LYWSD03MMC.py
│   ├── temperatur_koksfonstret.py
│   ├── sendToMQTT.sh
│   ├── startup.sh
│   ├── sensorer.sh
│   └── sensorer.ini
└── systemd/
    ├── sensorpajen.service      # Systemd service unit
    └── README.md                # Systemd installation instructions

Configuration Strategy

Using relative paths for portability across systems:

Sensor Mapping: config/sensors.json (relative to project root)
- Maps MAC addresses to sensor names
- JSON format for Python ease
- Not committed to git (use sensors.json.example as template)
MQTT Credentials: config/sensorpajen.env (relative to project root)
- Contains sensitive MQTT configuration
- Permissions: 0600 (owner read/write only)
- Not committed to git (use sensorpajen.env.example as template)

Environment Variables (via systemd EnvironmentFile):

MQTT_HOST=192.168.0.114
MQTT_USER=hasse
MQTT_PASSWORD=casablanca
MQTT_CLIENT_ID=mibridge
SENSOR_CONFIG_FILE=config/sensors.json

Git Ignore: Add to .gitignore:

config/sensors.json
config/sensorpajen.env

Migration Phases

Phase 1: Preparation & Cleanup ✅ DONE (2025-12-27)

Goal: Reorganize repository without breaking existing functionality

Notes:

Created modern Python package structure with src/ layout
Converted INI sensor config to JSON format (sensors.json.example)
Environment-based configuration instead of hardcoded values
DHT11 sensor functionality removed as planned
Legacy scripts preserved in legacy/ folder

Tasks:

✅ Create new directory structure
✅ Create pyproject.toml with dependencies
✅ Remove DHT11 functionality
✅ Move legacy scripts to legacy/ folder
✅ Create config file templates (sensors.json.example, sensorpajen.env.example)
✅ Preserve requirements.txt for backward compatibility

Phase 2: Python Package Structure ✅ DONE (2025-12-27)

Goal: Create modern Python package with proper entry point

Notes:

Used src/ layout for better packaging practices
Direct Python MQTT integration (no shell script callbacks)
ATC firmware BLE advertisement reading (passive scanning)
Watchdog thread for BLE connection recovery
Clean separation of concerns (config, MQTT, sensors, main)

Tasks:

✅ Created src/sensorpajen/init.py with version info
✅ Created src/sensorpajen/config.py
- Environment variable loading with validation
- SensorConfig class for JSON sensor mapping
- Relative path resolution (PROJECT_ROOT)
- Configuration validation and logging
✅ Created src/sensorpajen/utils.py
- Ported bluetooth_utils.py (MIT licensed, Colin GUYON)
- BLE scanning and advertisement parsing
✅ Created src/sensorpajen/mqtt_publisher.py
- MQTTPublisher class with connection management
- Direct publishing (replaces sendToMQTT.sh)
- Automatic reconnection support
- Battery data publishing (optional)
✅ Created src/sensorpajen/sensor_reader.py
- SensorReader class for BLE scanning
- ATC packet parsing
- Duplicate packet filtering
- Watchdog for BLE recovery
- Measurement dataclass
✅ Created src/sensorpajen/main.py
- Application entry point
- Signal handling (SIGTERM, SIGINT)
- Graceful shutdown
- Logging to stdout for journald

Phase 3: Configuration Migration ✅ DONE (2025-12-27)

Goal: Replace .ini file with JSON and environment variables

Notes: Templates created in Phase 1, successfully tested on Raspberry Pi

Tasks:

Create sensor mapping converter script

Read sensorer.ini
Output to sensors.json

{
  "sensors": [
    {
      "mac": "A4:C1:38:98:7B:B6",
      "name": "mi_temp_1"
    },
    {
      "mac": "A4:C1:38:29:03:0D",
      "name": "mi_temp_2"
    }
  ]
}

configuration file templates

config/sensorpajen.env.example

# MQTT Configuration
MQTT_HOST=192.168.0.114
MQTT_PORT=1883
MQTT_USER=hasse
MQTT_PASSWORD=casablanca
MQTT_CLIENT_ID=mibridge

# Sensor Configuration (relative to project root)
SENSOR_CONFIG_FILE=config/sensors.json

# Application Settings
WATCHDOG_TIMEOUT=5
ENABLE_BATTERY=true
LOG_LEVEL=INFO

config/sensors.json.example

{
  "sensors": [
    {
      "mac": "A4:C1:38:98:7B:B6",
      "name": "mi_temp_1",
      "comment": "Example sensor"
    }
  ]
}

Copy templates to actual config files (not in git):

cp config/sensorpajen.env.example config/sensorpajen.env
cp config/sensors.json.example config/sensors.json
chmod 600 config/sensorpajen.env
# Edit both files with your actual configurationnsorpajen/sensorpajen.env
chmod 600 /home/fredrik/.config/sensorpajen/sensorpajen.env

Document all configuration variables in README

config/sensorpajen.env config/sensors.json .deb debian/.debhelper/ debian/sensorpajen/ debian/files debian/.log debian/*.substvars

Phase 4: Virtual Environment & Dependencies ✅ DONE (2025-12-27)

Goal: Set up isolated Python environment

Notes: Tested on Raspberry Pi, paho-mqtt v2.x compatibility fixed

Tasks:

Create virtual environment:
```
python3 -m venv .venv
```

Update .gitignore:

.venv/
__pycache__/
*.pyc
.env
sensorpajen.env

Install dependencies:

source .venv/bin/activate
pip install --upgrade pip
pip install bluepy paho-mqtt
pip install -e .  # Install package in development mode

Document virtual environment usage in README

---✅ DONE (2025-12-27) Goal: Allow non-root user to access Bluetooth

Notes: Tested on Raspberry Pi with setcap on actual Python binary

Phase 5: Bluetooth Permissions ✅ DONE (2025-12-27)

Goal: Allow non-root user to access Bluetooth

Notes: Tested on Raspberry Pi with setcap on actual Python binary

Tasks:

✅ Bluetooth capabilities set with setcap
✅ Documented in SETUP_ON_PI.md with correct readlink -f usage
✅ Tested successfully on Raspberry Pi

Phase 6: Systemd Service Creation ✅ DONE (2025-12-27)

Goal: Create and configure systemd user service

Notes:

User service for easier management (no sudo required)
Service ready for installation on Raspberry Pi
Comprehensive documentation provided
Important discoveries:
- AmbientCapabilities does NOT work in user services (only system services)
- Must use setcap on the Python binary instead
- NoNewPrivileges=true prevents file capabilities from working - must be disabled
- Capabilities must be set on actual binary, not symlinks: setcap ... $(readlink -f python3)

Tasks:

✅ Created systemd/sensorpajen.service
✅ Created systemd/README.md with full documentation
✅ Service management and troubleshooting guides included
✅ Tested and verified working on Raspberry Pi

Phase 7: Testing & Validation ✓ TODO

Goal: Verify new service works before removing legacy

Tasks:

Stop legacy cron/tmux processes:

crontab -e  # Comment out sensorpajen entries
tmux kill-session -t sensorer

Start new service:
```
systemctl --user start sensorpajen
```

Monitor logs:

journalctl --user -u sensorpajen -f
```APT Package Creation ✓ TODO

Goal: Create Debian package for easy installation on Raspberry Pi

Tasks:

Create debian/ directory structure:
```
mkdir -p debian
```
Create debian/control: ``APT package installation instructions
- Development installation instructions
- Configuration guide (relative paths)
- Service management commands
- Troubleshooting section
- Remove DHT11 references
- Remove pirate_audio references
Create INSTALL.md:
- APT package installation steps
- Manual installation steps
- Configuration examples
- First-time setup guide
- Raspberry Pi specific instructionsds}, ${misc:Depends}, python3-bluepy, python3-paho-mqtt, bluetooth, bluez Description: Bluetooth temperature sensor monitor Monitors Xiaomi Mijia LYWSD03MMC Bluetooth temperature sensors and publishes data to MQTT broker.

Create debian/rules:

#!/usr/bin/make -f

%:
	dh $@ --with python3 --buildsystem=pybuild

override_dh_auto_install:
	pytOption 1: APT Package (Recommended for Raspberry Pi)

1. Download and install the .deb package:
```bash
sudo dpkg -i sensorpajen_1.0.0_all.deb
sudo apt-get install -f  # Fix any dependencies

Configure:

mkdir -p ~/sensorpajen/config
cp /usr/share/doc/sensorpajen/examples/sensorpajen.env.example ~/sensorpajen/config/sensorpajen.env
cp /usr/share/doc/sensorpajen/examples/sensors.json.example ~/sensorpajen/config/sensors.json
# Edit both files
nano ~/sensorpajen/config/sensorpajen.env
nano ~/sensorpajen/config/sensors.json
chmod 600 ~/sensorpajen/config/sensorpajen.env

Enable and start service:

systemctl --user enable sensorpajen
systemctl --user start sensorpajen

Option 2: Development Installation

Clone Repository

git clone <repo> ~/sensorpajen
cd ~/sensorpajen

Create Virtual Environment

python3 -m venv .venv
source .venv/bin/activate
pip install -e .

Relative Paths (For Portability)

Project root: ~/sensorpajen/ (or wherever you clone/install)
Application config: ~/sensorpajen/config/
Environment file: ~/sensorpajen/config/sensorpajen.env (0600)
Sensor mapping: ~/sensorpajen/config/sensors.json (0644)
Service file: ~/.config/systemd/user/sensorpajen.service

Advantages of Relative Paths

Works on any system (development, production, multiple Raspberry Pis)
Easy to backup/restore entire directory
No hardcoded paths in code
Simple to deploy via git pull or package installation
User service runs without sudo

APT Package Installation

When installed via .deb package:

Python package: /usr/lib/python3/dist-packages/sensorpajen/
Service file: /lib/systemd/user/sensorpajen.service
Config templates: /usr/share/doc/sensorpajen/examples/

User config: ~/sensorpajen/config/ (created by user)sensorpajen


5. Verify
```bash
systemctl --user status sensorpajen
journalctl --user -u sensorpajen -f
```uetooth access
if [ "$1" = "configure" ]; then
    PYTHON_PATH=$(readlink -f /usr/bin/python3)
    setcap 'cap_net_raw,cap_net_admin+eip' "$PYTHON_PATH" || true
fi

#DEBHELPER#

Create debian/README.Debian:
- Installation instructions
- Configuration guide
- Service management
Build the package:
```
dpkg-buildpackage -us -uc -b
```

Test installation on Raspberry Pi:

sudo dpkg -i ../sensorpajen_1.0.0_all.deb
sudo apt-get install -f  # Fix dependencies if needed

Create installation documentation:
- Package installation instructions
- Configuration setup after installation
- Service enablement

Phase 9:

Verify MQTT messages:

mosquitto_sub -h 192.168.0.114 -u hasse -P casablanca -t "MiTemperature2/#" -v

Test service restart:
```
systemctl --user restart sensorpajen
```
Test crash recovery (kill process, verify auto-restart)

Test boot behavior:

systemctl --user enable sensorpajen
sudo reboot
# After reboot, verify service is running
systemctl --user status sensorpajen

Phase 8: Cleanup & Document

Publish APT package to personal repository
Create automated build pipeline for .deb packages
Add support for multiple MQTT brokers
Implement configuration validation toolation ✓ TODO Goal: Remove legacy code and finalize documentation

Tasks:

Once new service is stable (run for 1-2 weeks):
- Delete legacy/ folder
- Remove cron jobs completely
- Remove tmux session references
Update README.md:
- Installation instructions
- Configuration guide
- Service management commands
- Troubleshooting section
- Remove DHT11 references
- Remove pirate_audio references
Create INSTALL.md:
- Fresh installation steps
- Configuration examples
- First-time setup guide

Document in README:

## Installation

### 1. Clone Repository
git clone <repo> /home/fredrik/dev/sensorpajen
cd /home/fredrik/dev/sensorpajen

### 2. Create Virtual Environment
python3 -m venv .venv
source .venv/bin/activate
pip install -e .

### 3. Configure
mkdir -p ~/.config/sensorpajen
cp systemd/sensorpajen.env.example ~/.config/sensorpajen/sensorpajen.env
# Edit configuration
nano ~/.config/sensorpajen/sensorpajen.env
chmod 600 ~/.config/sensorpajen/sensorpajen.env

### 4. Convert Sensor Configuration
# Create sensors.json from your sensor list

### 5. Install Service
cp systemd/sensorpajen.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable sensorpajen
systemctl --user start sensorpajen

### 6. Verify
systemctl --user status sensorpajen
journalctl --user -u sensorpajen -f

Add troubleshooting section:
- Bluetooth permission issues
- MQTT connection problems
- Service won't start
- Log locations

Configuration File Locations (Linux Best Practices)

User Service Configuration

Service files: ~/.config/systemd/user/
Application config: ~/.config/sensorpajen/
Environment file: ~/.config/sensorpajen/sensorpajen.env (0600)
Sensor mapping: ~/.config/sensorpajen/sensors.json (0644)

System Service (Alternative - Not Recommended)

If running as system service (not user service):

Service file: /etc/systemd/system/sensorpajen.service
Config directory: /etc/sensorpajen/
Environment file: /etc/sensorpajen/sensorpajen.env (0600)

Recommendation: Use user service (current approach) since:

No sudo required for service management
Easier permission management
Better security isolation
Simpler Bluetooth access

Success Criteria

The migration is complete when:

✅ Service starts automatically on boot
✅ All 8 Bluetooth sensors are being read
✅ MQTT messages are published correctly
✅ Service recovers automatically from crashes
✅ No hardcoded credentials in code
✅ Logs are visible via journalctl
✅ DHT11 functionality completely removed
✅ Legacy scripts removed
✅ Documentation is complete and accurate
✅ Service runs as user (not root)
✅ Virtual environment is working

Rollback Plan

If issues arise during migration:

Stop new service:

systemctl --user stop sensorpajen
systemctl --user disable sensorpajen

Restore legacy scripts from legacy/ folder:
```
cp legacy/* .
```

Restore cron jobs:

crontab -e
# Uncomment:
# @reboot /home/fredrik/dev/sensorpajen/sensorer.sh

Reboot or manually start tmux session

Future Enhancements

After successful migration, consider:

Add Prometheus metrics endpoint
Add systemd watchdog support
Implement graceful sensor failure handling
Add MQTT TLS support
Create web dashboard for sensor status
Add sensor calibration configuration
Implement sensor auto-discovery
Add health check endpoint

Notes

Keep legacy scripts during migration for safety
Test thoroughly before removing cron jobs
Monitor for at least 1-2 weeks before final cleanup
Document any issues encountered during migration
Take notes of actual MAC addresses and sensor names during conversion

References

systemd user services: man systemd.service
XDG Base Directory: ~/.config/ for user configuration
Bluetooth capabilities: man capabilities
journalctl: man journalctl
Python logging: https://docs.python.org/3/library/logging.html

18 KiB Raw Blame History

ROADMAP: Modernizing Sensorpajen

Overview

Current State

What We Have

Known Issues

Target Architecture

Final Structure

Configuration Strategy

Migration Phases

Phase 1: Preparation & Cleanup ✅ DONE (2025-12-27)

Tasks:

Phase 2: Python Package Structure ✅ DONE (2025-12-27)

Tasks:

Phase 3: Configuration Migration ✅ DONE (2025-12-27)

Tasks:

Phase 4: Virtual Environment & Dependencies ✅ DONE (2025-12-27)

Tasks:

Phase 5: Bluetooth Permissions ✅ DONE (2025-12-27)

Tasks:

Phase 6: Systemd Service Creation ✅ DONE (2025-12-27)

Tasks:

Phase 7: Testing & Validation ✓ TODO

Tasks:

Tasks:

Option 2: Development Installation

Relative Paths (For Portability)

Advantages of Relative Paths

APT Package Installation

Phase 9:

Phase 8: Cleanup & Document

Tasks:

Configuration File Locations (Linux Best Practices)

User Service Configuration

System Service (Alternative - Not Recommended)

Success Criteria

Rollback Plan

Future Enhancements

Notes

References

18 KiB

Raw Blame History