fredrik/sensorpajen
1
0
Fork 0
Code Issues Pull Requests Projects Releases 2 Wiki Activity
Files
fbcd007e64bb3fd9030fd15e5581ca6e6cfd0d50
sensorpajen/ROADMAP-v2.md

17 KiB
Raw Blame History

ROADMAP: Modernizing Sensorpajen

Overview

This roadmap outlines the migration from the current tmux/cron-based system to a modern systemd service running on Raspberry Pi.

Migration Date: Started December 27, 2025
Target Completion: TBD

Current State

What We Have

  • LYWSD03MMC.py: Main Bluetooth sensor reader
  • temperatur_koksfonstret.py: DHT11 sensor reader (to be removed)
  • bluetooth_utils.py: Bluetooth utility functions
  • sensorer.ini: MAC address to sensor name mapping
  • sendToMQTT.sh: MQTT publishing callback (hardcoded credentials)
  • startup.sh/sensorer.sh: tmux-based startup scripts
  • Cron jobs for scheduling

Known Issues

  • MQTT credentials hardcoded in shell scripts
  • Legacy pirate_audio references in startup.sh
  • Manual tmux orchestration
  • Mixed configuration sources
  • DHT11 functionality to be removed

Target Architecture

Final Structure

sensorpajen/
├── src/
│   └── sensorpajen/
│       ├── __init__.py
│       ├── main.py              # Entry point
│       ├── config.py            # Configuration management
│       ├── sensor_reader.py     # Bluetooth sensor logic
│       ├── mqtt_publisher.py    # MQTT publishing
│       └── utils.py             # Utilities (from bluetooth_utils.py)
├── config/                      # Configuration directory (relative)
│   ├── sensors.json.example     # Sensor mapping template
│   ├── sensorpajen.env.example  # Environment file template
│   ├── sensors.json             # Actual sensor mapping (not in git)
│   └── sensorpajen.env          # Actual environment file (not in git)
├── debian/                      # APT package files
│   ├── control
│   ├── rules
│   ├── changelog
│   └── ...                      # Other Debian package files
├── pyproject.toml               # Project metadata and dependencies
├── requirements.txt             # Dependencies (bluepy, paho-mqtt)
├── README.md                    # Updated documentation
├── AGENTS.md                    # Agent guidelines
├── ROADMAP.md                   # This file
├── legacy/                      # Legacy scripts (moved here temporarily)
│   ├── LYWSD03MMC.py
│   ├── temperatur_koksfonstret.py
│   ├── sendToMQTT.sh
│   ├── startup.sh
│   ├── sensorer.sh
│   └── sensorer.ini
└── systemd/
    ├── sensorpajen.service      # Systemd service unit
    └── README.md                # Systemd installation instructions

Configuration Strategy

Using relative paths for portability across systems:

  1. Sensor Mapping: config/sensors.json (relative to project root)

    • Maps MAC addresses to sensor names
    • JSON format for Python ease
    • Not committed to git (use sensors.json.example as template)

  2. MQTT Credentials: config/sensorpajen.env (relative to project root)

    • Contains sensitive MQTT configuration
    • Permissions: 0600 (owner read/write only)
    • Not committed to git (use sensorpajen.env.example as template)

  3. Environment Variables (via systemd EnvironmentFile):

    MQTT_HOST=192.168.0.114
MQTT_USER=hasse
MQTT_PASSWORD=casablanca
MQTT_CLIENT_ID=mibridge
SENSOR_CONFIG_FILE=config/sensors.json

  4. Git Ignore: Add to .gitignore:

    config/sensors.json
config/sensorpajen.env

Migration Phases

Phase 1: Preparation & Cleanup ✅ DONE (2025-12-27)

Goal: Reorganize repository without breaking existing functionality

Notes:

  • Created modern Python package structure with src/ layout
  • Converted INI sensor config to JSON format (sensors.json.example)
  • Environment-based configuration instead of hardcoded values
  • DHT11 sensor functionality removed as planned
  • Legacy scripts preserved in legacy/ folder

Tasks:

  • ✅ Create new directory structure
  • ✅ Create pyproject.toml with dependencies
  • ✅ Remove DHT11 functionality
  • ✅ Move legacy scripts to legacy/ folder
  • ✅ Create config file templates (sensors.json.example, sensorpajen.env.example)
  • ✅ Preserve requirements.txt for backward compatibility

Phase 2: Python Package Structure ✅ DONE (2025-12-27)

Goal: Create modern Python package with proper entry point

Notes:

  • Used src/ layout for better packaging practices
  • Direct Python MQTT integration (no shell script callbacks)
  • ATC firmware BLE advertisement reading (passive scanning)
  • Watchdog thread for BLE connection recovery
  • Clean separation of concerns (config, MQTT, sensors, main)

Tasks:

  • ✅ Created src/sensorpajen/init.py with version info
  • ✅ Created src/sensorpajen/config.py
    • Environment variable loading with validation
    • SensorConfig class for JSON sensor mapping
    • Relative path resolution (PROJECT_ROOT)
    • Configuration validation and logging
  • ✅ Created src/sensorpajen/utils.py
    • Ported bluetooth_utils.py (MIT licensed, Colin GUYON)
    • BLE scanning and advertisement parsing
  • ✅ Created src/sensorpajen/mqtt_publisher.py
    • MQTTPublisher class with connection management
    • Direct publishing (replaces sendToMQTT.sh)
    • Automatic reconnection support
    • Battery data publishing (optional)
  • ✅ Created src/sensorpajen/sensor_reader.py
    • SensorReader class for BLE scanning
    • ATC packet parsing
    • Duplicate packet filtering
    • Watchdog for BLE recovery
    • Measurement dataclass
  • ✅ Created src/sensorpajen/main.py
    • Application entry point
    • Signal handling (SIGTERM, SIGINT)
    • Graceful shutdown
    • Logging to stdout for journald

Phase 3: Configuration Migration ✅ DONE (2025-12-27)

Goal: Replace .ini file with JSON and environment variables

Notes: Templates created in Phase 1, successfully tested on Raspberry Pi

Tasks:

  1. Create sensor mapping converter script
    • Read sensorer.ini
    • Output to sensors.json
    {
  "sensors": [
    {
      "mac": "A4:C1:38:98:7B:B6",
      "name": "mi_temp_1"
    },
    {
      "mac": "A4:C1:38:29:03:0D",
      "name": "mi_temp_2"
    }
  ]
}

configuration file templates

  • config/sensorpajen.env.example
# MQTT Configuration
MQTT_HOST=192.168.0.114
MQTT_PORT=1883
MQTT_USER=hasse
MQTT_PASSWORD=casablanca
MQTT_CLIENT_ID=mibridge

# Sensor Configuration (relative to project root)
SENSOR_CONFIG_FILE=config/sensors.json

# Application Settings
WATCHDOG_TIMEOUT=5
ENABLE_BATTERY=true
LOG_LEVEL=INFO
  • config/sensors.json.example
{
  "sensors": [
    {
      "mac": "A4:C1:38:98:7B:B6",
      "name": "mi_temp_1",
      "comment": "Example sensor"
    }
  ]
}

  1. Copy templates to actual config files (not in git):

    cp config/sensorpajen.env.example config/sensorpajen.env
cp config/sensors.json.example config/sensors.json
chmod 600 config/sensorpajen.env
# Edit both files with your actual configurationnsorpajen/sensorpajen.env
chmod 600 /home/fredrik/.config/sensorpajen/sensorpajen.env

  2. Document all configuration variables in README

config/sensorpajen.env config/sensors.json .deb debian/.debhelper/ debian/sensorpajen/ debian/files debian/.log debian/*.substvars

Phase 4: Virtual Environment & Dependencies ✅ DONE (2025-12-27)

Goal: Set up isolated Python environment

Notes: Tested on Raspberry Pi, paho-mqtt v2.x compatibility fixed

Tasks:

  1. Create virtual environment:

    python3 -m venv .venv

  2. Update .gitignore:

    .venv/
__pycache__/
*.pyc
.env
sensorpajen.env

  3. Install dependencies:

    source .venv/bin/activate
pip install --upgrade pip
pip install bluepy paho-mqtt
pip install -e .  # Install package in development mode

  4. Document virtual environment usage in README

---✅ DONE (2025-12-27) Goal: Allow non-root user to access Bluetooth

Notes: Tested on Raspberry Pi with setcap on actual Python binary

Phase 5: Bluetooth Permissions ✅ DONE (2025-12-27)

Goal: Allow non-root user to access Bluetooth

Notes: Tested on Raspberry Pi with setcap on actual Python binary

Tasks:

  • ✅ Bluetooth capabilities set with setcap
  • ✅ Documented in SETUP_ON_PI.md with correct readlink -f usage
  • ✅ Tested successfully on Raspberry Pi

Phase 6: Systemd Service Creation ✅ DONE (2025-12-27)

Goal: Create and configure systemd user service

Notes:

  • User service for easier management (no sudo required)
  • Service ready for installation on Raspberry Pi
  • Comprehensive documentation provided
  • Important discoveries:
    • AmbientCapabilities does NOT work in user services (only system services)
    • Must use setcap on the Python binary instead
    • NoNewPrivileges=true prevents file capabilities from working - must be disabled
    • Capabilities must be set on actual binary, not symlinks: setcap ... $(readlink -f python3)

Tasks:

  • ✅ Created systemd/sensorpajen.service
  • ✅ Created systemd/README.md with full documentation
  • ✅ Service management and troubleshooting guides included
  • ✅ Tested and verified working on Raspberry Pi

Phase 7: Testing & Validation ✅ DONE (2025-12-27)

Goal: Verify new service works before removing legacy

Notes:

  • Service tested and running successfully
  • Legacy cron/tmux system stopped
  • All sensors reporting correctly via systemd service

Tasks:

  • ✅ Stopped legacy cron/tmux processes
  • ✅ Started new systemd service
  • ✅ Monitored logs - no errors
  • ✅ Verified all 8 sensors reporting
  • ✅ Confirmed MQTT publishing working
  • ✅ Tested service restart and auto-recovery

Phase 8: APT Package Creation ✅ DONE (2025-12-27)

Goal: Create Debian package for easy installation on Raspberry Pi

Notes:

  • Complete debian/ directory structure created
  • System-wide installation to /opt/sensorpajen
  • Configuration in /etc/sensorpajen
  • Dedicated sensorpajen system user
  • Automatic venv creation in postinst
  • Bluetooth capabilities set automatically
  • Config preserved on remove/purge for safety
  • Dual-mode support: system installation and development
  • config.py auto-detects installation type

Files Created:

  • ✅ debian/control - Package metadata and dependencies
  • ✅ debian/compat - Debhelper compatibility level
  • ✅ debian/changelog - Package version history
  • ✅ debian/rules - Build instructions
  • ✅ debian/install - File installation mappings
  • ✅ debian/postinst - Post-installation script (user, venv, setcap)
  • ✅ debian/prerm - Pre-removal script (stop service)
  • ✅ debian/postrm - Post-removal script (cleanup)
  • ✅ debian/sensorpajen.service - System-wide systemd unit

Code Updates:

  • ✅ Updated src/sensorpajen/config.py to detect system installation
    • Checks for /opt/sensorpajen existence
    • Uses /etc/sensorpajen for config in system mode
    • Falls back to PROJECT_ROOT/config for development
  • ✅ Updated scripts/approve-sensors.sh for dual-mode operation
    • Detects system vs development installation
    • Uses correct venv and config paths
  • ✅ Created scripts/verify-deb.sh - Automated build and verification

Package Details:

  • Package name: sensorpajen
  • Version: 2.0.0-dev
  • Architecture: all
  • System paths:
    • Application: /opt/sensorpajen/
    • Configuration: /etc/sensorpajen/
    • Service file: /etc/systemd/system/sensorpajen.service
    • Examples: /usr/share/doc/sensorpajen/examples/
  • Runs as dedicated sensorpajen user (system account)
  • Auto-enables service but waits for configuration before starting

Build and Test:

# Build package
./scripts/verify-deb.sh

# Or manually:
dpkg-buildpackage -us -uc -b
lintian ../sensorpajen_*.deb

# Install on Raspberry Pi:
scp ../sensorpajen_*.deb pi@raspberrypi:~/
ssh pi@raspberrypi
sudo apt install ./sensorpajen_*.deb

# Configure:
sudo nano /etc/sensorpajen/sensorpajen.env
sudo nano /etc/sensorpajen/sensors.json

# Start:
sudo systemctl start sensorpajen
sudo journalctl -u sensorpajen -f

Phase 9: Cleanup & Documentation ✅ DONE (2025-12-27)

Goal: Remove legacy code and finalize documentation

Notes:

  • Legacy cron/tmux scripts removed
  • Documentation focused on practical usage
  • INSTALL.md created for sysadmins

Tasks:

  • ✅ Deleted legacy/ folder (old cron/tmux scripts)
  • ✅ Created INSTALL.md with concise installation guide
  • ✅ Updated README.md troubleshooting section
  • ✅ Documentation assumes sysadmin familiarity

Migration Complete! 🎉

All phases completed. The system has been successfully migrated from a legacy cron/tmux-based system to a modern systemd service with:

  • ✅ Python package structure
  • ✅ Environment-based configuration (no .ini files)
  • ✅ Systemd user service with auto-restart
  • ✅ Automatic sensor discovery with approval workflow
  • ✅ Configuration auto-reload (no restart needed)
  • ✅ ntfy notifications for new sensors
  • ✅ Comprehensive documentation

Version: 2.0.0-dev
Status: Production-ready

## Installation

### 1. Clone Repository
git clone <repo> /home/fredrik/dev/sensorpajen
cd /home/fredrik/dev/sensorpajen

### 2. Create Virtual Environment
python3 -m venv .venv
source .venv/bin/activate
pip install -e .

### 3. Configure
mkdir -p ~/.config/sensorpajen
cp systemd/sensorpajen.env.example ~/.config/sensorpajen/sensorpajen.env
# Edit configuration
nano ~/.config/sensorpajen/sensorpajen.env
chmod 600 ~/.config/sensorpajen/sensorpajen.env

### 4. Convert Sensor Configuration
# Create sensors.json from your sensor list

### 5. Install Service
cp systemd/sensorpajen.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable sensorpajen
systemctl --user start sensorpajen

### 6. Verify
systemctl --user status sensorpajen
journalctl --user -u sensorpajen -f
  1. Add troubleshooting section:
    • Bluetooth permission issues
    • MQTT connection problems
    • Service won't start
    • Log locations

Configuration File Locations (Linux Best Practices)

User Service Configuration

  • Service files: ~/.config/systemd/user/
  • Application config: ~/.config/sensorpajen/
  • Environment file: ~/.config/sensorpajen/sensorpajen.env (0600)
  • Sensor mapping: ~/.config/sensorpajen/sensors.json (0644)

System Service (Alternative - Not Recommended)

If running as system service (not user service):

  • Service file: /etc/systemd/system/sensorpajen.service
  • Config directory: /etc/sensorpajen/
  • Environment file: /etc/sensorpajen/sensorpajen.env (0600)

Recommendation: Use user service (current approach) since:

  • No sudo required for service management
  • Easier permission management
  • Better security isolation
  • Simpler Bluetooth access

Success Criteria

The migration is complete when:

  • ✅ Service starts automatically on boot
  • ✅ All 8 Bluetooth sensors are being read
  • ✅ MQTT messages are published correctly
  • ✅ Service recovers automatically from crashes
  • ✅ No hardcoded credentials in code
  • ✅ Logs are visible via journalctl
  • ✅ DHT11 functionality completely removed
  • ✅ Legacy scripts removed
  • ✅ Documentation is complete and accurate
  • ✅ Service runs as user (not root)
  • ✅ Virtual environment is working

Rollback Plan

If issues arise during migration:

  1. Stop new service:

    systemctl --user stop sensorpajen
systemctl --user disable sensorpajen

  2. Restore legacy scripts from legacy/ folder:

    cp legacy/* .

  3. Restore cron jobs:

    crontab -e
# Uncomment:
# @reboot /home/fredrik/dev/sensorpajen/sensorer.sh

  4. Reboot or manually start tmux session

Future Enhancements

After successful migration, consider:

  • Add Prometheus metrics endpoint
  • Add systemd watchdog support
  • Implement graceful sensor failure handling
  • Add MQTT TLS support
  • Create web dashboard for sensor status
  • Add sensor calibration configuration
  • Implement sensor auto-discovery
  • Add health check endpoint

Notes

  • Keep legacy scripts during migration for safety
  • Test thoroughly before removing cron jobs
  • Monitor for at least 1-2 weeks before final cleanup
  • Document any issues encountered during migration
  • Take notes of actual MAC addresses and sensor names during conversion

References

Reference in New Issue View Git Blame Copy Permalink