sensorpajen/ROADMAP.md

ROADMAP: Modernizing Sensorpajen

Overview

This roadmap outlines the migration from the current tmux/cron-based system to a modern systemd service running on Raspberry Pi.

Migration Date: Started December 27, 2025
Target Completion: TBD

---

Current State

What We Have

• LYWSD03MMC.py: Main Bluetooth sensor reader
• temperatur_koksfonstret.py: DHT11 sensor reader (to be removed)
• bluetooth_utils.py: Bluetooth utility functions
• sensorer.ini: MAC address to sensor name mapping
• sendToMQTT.sh: MQTT publishing callback (hardcoded credentials)
• startup.sh/sensorer.sh: tmux-based startup scripts
• Cron jobs for scheduling

Known Issues

• MQTT credentials hardcoded in shell scripts
• Legacy pirate_audio references in startup.sh
• Manual tmux orchestration
• Mixed configuration sources
• DHT11 functionality to be removed

---

Target Architecture

Final Structure

sensorpajen/
├── src/
│   └── sensorpajen/
│       ├── __init__.py
│       ├── main.py              # Entry point
│       ├── config.py            # Configuration management
│       ├── sensor_reader.py     # Bluetooth sensor logic
│       ├── mqtt_publisher.py    # MQTT publishing
│       └── utils.py             # Utilities (from bluetooth_utils.py)
├── config/                      # Configuration directory (relative)
│   ├── sensors.json.example     # Sensor mapping template
│   ├── sensorpajen.env.example  # Environment file template
│   ├── sensors.json             # Actual sensor mapping (not in git)
│   └── sensorpajen.env          # Actual environment file (not in git)
├── debian/                      # APT package files
│   ├── control
│   ├── rules
│   ├── changelog
│   └── ...                      # Other Debian package files
├── pyproject.toml               # Project metadata and dependencies
├── requirements.txt             # Dependencies (bluepy, paho-mqtt)
├── README.md                    # Updated documentation
├── AGENTS.md                    # Agent guidelines
├── ROADMAP.md                   # This file
├── legacy/                      # Legacy scripts (moved here temporarily)
│   ├── LYWSD03MMC.py
│   ├── temperatur_koksfonstret.py
│   ├── sendToMQTT.sh
│   ├── startup.sh
│   ├── sensorer.sh
│   └── sensorer.ini
└── systemd/
    ├── sensorpajen.service      # Systemd service unit
    └── README.md                # Systemd installation instructions

Configuration Strategy

Using relative paths for portability across systems:

1. Sensor Mapping: config/sensors.json (relative to project root)

   • Maps MAC addresses to sensor names
   • JSON format for Python ease
   • Not committed to git (use sensors.json.example as template)

2. MQTT Credentials: config/sensorpajen.env (relative to project root)

   • Contains sensitive MQTT configuration
   • Permissions: 0600 (owner read/write only)
   • Not committed to git (use sensorpajen.env.example as template)

3. Environment Variables (via systemd EnvironmentFile):

   MQTT_HOST=192.168.0.114
   MQTT_USER=hasse
   MQTT_PASSWORD=casablanca
   MQTT_CLIENT_ID=mibridge
   SENSOR_CONFIG_FILE=config/sensors.json
   

4. Git Ignore: Add to .gitignore:

   config/sensors.json
   config/sensorpajen.env
   

---

Migration Phases

Phase 1: Preparation & Cleanup ✅ DONE (2025-12-27)

Goal: Reorganize repository without breaking existing functionality

Notes:

• Created modern Python package structure with src/ layout
• Converted INI sensor config to JSON format (sensors.json.example)
• Environment-based configuration instead of hardcoded values
• DHT11 sensor functionality removed as planned
• Legacy scripts preserved in legacy/ folder

Tasks:

• ✅ Create new directory structure
• ✅ Create pyproject.toml with dependencies
• ✅ Remove DHT11 functionality
• ✅ Move legacy scripts to legacy/ folder
• ✅ Create config file templates (sensors.json.example, sensorpajen.env.example)
• ✅ Preserve requirements.txt for backward compatibility

---

Phase 2: Python Package Structure ✅ DONE (2025-12-27)

Goal: Create modern Python package with proper entry point

Notes:

• Used src/ layout for better packaging practices
• Direct Python MQTT integration (no shell script callbacks)
• ATC firmware BLE advertisement reading (passive scanning)
• Watchdog thread for BLE connection recovery
• Clean separation of concerns (config, MQTT, sensors, main)

Tasks:

• ✅ Created src/sensorpajen/init.py with version info
• ✅ Created src/sensorpajen/config.py
  • Environment variable loading with validation
  • SensorConfig class for JSON sensor mapping
  • Relative path resolution (PROJECT_ROOT)
  • Configuration validation and logging
• ✅ Created src/sensorpajen/utils.py
  • Ported bluetooth_utils.py (MIT licensed, Colin GUYON)
  • BLE scanning and advertisement parsing
• ✅ Created src/sensorpajen/mqtt_publisher.py
  • MQTTPublisher class with connection management
  • Direct publishing (replaces sendToMQTT.sh)
  • Automatic reconnection support
  • Battery data publishing (optional)
• ✅ Created src/sensorpajen/sensor_reader.py
  • SensorReader class for BLE scanning
  • ATC packet parsing
  • Duplicate packet filtering
  • Watchdog for BLE recovery
  • Measurement dataclass
• ✅ Created src/sensorpajen/main.py
  • Application entry point
  • Signal handling (SIGTERM, SIGINT)
  • Graceful shutdown
  • Logging to stdout for journald

---

Phase 3: Configuration Migration ✅ DONE (2025-12-27)

Goal: Replace .ini file with JSON and environment variables

Notes: Templates created in Phase 1, successfully tested on Raspberry Pi

Tasks:

1. Create sensor mapping converter script
   • Read sensorer.ini
   • Output to sensors.json

   {
     "sensors": [
       {
         "mac": "A4:C1:38:98:7B:B6",
         "name": "mi_temp_1"
       },
       {
         "mac": "A4:C1:38:29:03:0D",
         "name": "mi_temp_2"
       }
     ]
   }
   

configuration file templates

• config/sensorpajen.env.example

# MQTT Configuration
MQTT_HOST=192.168.0.114
MQTT_PORT=1883
MQTT_USER=hasse
MQTT_PASSWORD=casablanca
MQTT_CLIENT_ID=mibridge

# Sensor Configuration (relative to project root)
SENSOR_CONFIG_FILE=config/sensors.json

# Application Settings
WATCHDOG_TIMEOUT=5
ENABLE_BATTERY=true
LOG_LEVEL=INFO

• config/sensors.json.example

{
  "sensors": [
    {
      "mac": "A4:C1:38:98:7B:B6",
      "name": "mi_temp_1",
      "comment": "Example sensor"
    }
  ]
}

3. Copy templates to actual config files (not in git):

   cp config/sensorpajen.env.example config/sensorpajen.env
   cp config/sensors.json.example config/sensors.json
   chmod 600 config/sensorpajen.env
   # Edit both files with your actual configurationnsorpajen/sensorpajen.env
   chmod 600 /home/fredrik/.config/sensorpajen/sensorpajen.env
   

4. Document all configuration variables in README

---

config/sensorpajen.env config/sensors.json .deb debian/.debhelper/ debian/sensorpajen/ debian/files debian/.log debian/*.substvars

Phase 4: Virtual Environment & Dependencies ✅ DONE (2025-12-27)

Goal: Set up isolated Python environment

Notes: Tested on Raspberry Pi, paho-mqtt v2.x compatibility fixed

Tasks:

1. Create virtual environment:

   python3 -m venv .venv
   

2. Update .gitignore:

   .venv/
   __pycache__/
   *.pyc
   .env
   sensorpajen.env
   

3. Install dependencies:

   source .venv/bin/activate
   pip install --upgrade pip
   pip install bluepy paho-mqtt
   pip install -e .  # Install package in development mode
   

4. Document virtual environment usage in README

---✅ DONE (2025-12-27) Goal: Allow non-root user to access Bluetooth

Notes: Tested on Raspberry Pi with setcap on actual Python binary

Phase 5: Bluetooth Permissions ✅ DONE (2025-12-27)

Goal: Allow non-root user to access Bluetooth

Notes: Tested on Raspberry Pi with setcap on actual Python binary

Tasks:

• ✅ Bluetooth capabilities set with setcap
• ✅ Documented in SETUP_ON_PI.md with correct readlink -f usage
• ✅ Tested successfully on Raspberry Pi

---

Phase 6: Systemd Service Creation ✅ DONE (2025-12-27)

Goal: Create and configure systemd user service

Notes:

• User service for easier management (no sudo required)
• Service ready for installation on Raspberry Pi
• Comprehensive documentation provided
• Important discoveries:
  • AmbientCapabilities does NOT work in user services (only system services)
  • Must use setcap on the Python binary instead
  • NoNewPrivileges=true prevents file capabilities from working - must be disabled
  • Capabilities must be set on actual binary, not symlinks: setcap ... $(readlink -f python3)

Tasks:

• ✅ Created systemd/sensorpajen.service
• ✅ Created systemd/README.md with full documentation
• ✅ Service management and troubleshooting guides included
• ✅ Tested and verified working on Raspberry Pi

---

Phase 7: Testing & Validation ✅ DONE (2025-12-27)

Goal: Verify new service works before removing legacy

Notes:

• Service tested and running successfully
• Legacy cron/tmux system stopped
• All sensors reporting correctly via systemd service

Tasks:

• ✅ Stopped legacy cron/tmux processes
• ✅ Started new systemd service
• ✅ Monitored logs - no errors
• ✅ Verified all 8 sensors reporting
• ✅ Confirmed MQTT publishing working
• ✅ Tested service restart and auto-recovery

---

Phase 8: APT Package Creation ✓ TODO

Goal: Create Debian package for easy installation on Raspberry Pi

Tasks:

1. Create debian/ directory structure:

   mkdir -p debian
   

2. Create debian/control: ``APT package installation instructions

   • Development installation instructions
   • Configuration guide (relative paths)
   • Service management commands
   • Troubleshooting section
   • Remove DHT11 references
   • Remove pirate_audio references

3. Create INSTALL.md:

   • APT package installation steps
   • Manual installation steps
   • Configuration examples
   • First-time setup guide
   • Raspberry Pi specific instructionsds}, ${misc:Depends}, python3-bluepy, python3-paho-mqtt, bluetooth, bluez Description: Bluetooth temperature sensor monitor Monitors Xiaomi Mijia LYWSD03MMC Bluetooth temperature sensors and publishes data to MQTT broker.

4. Create debian/rules:

   #!/usr/bin/make -f
   
   %:
   	dh $@ --with python3 --buildsystem=pybuild
   
   override_dh_auto_install:
   	pytOption 1: APT Package (Recommended for Raspberry Pi)
   
   1. Download and install the .deb package:
   ```bash
   sudo dpkg -i sensorpajen_1.0.0_all.deb
   sudo apt-get install -f  # Fix any dependencies
   

   2. Configure:

   mkdir -p ~/sensorpajen/config
   cp /usr/share/doc/sensorpajen/examples/sensorpajen.env.example ~/sensorpajen/config/sensorpajen.env
   cp /usr/share/doc/sensorpajen/examples/sensors.json.example ~/sensorpajen/config/sensors.json
   # Edit both files
   nano ~/sensorpajen/config/sensorpajen.env
   nano ~/sensorpajen/config/sensors.json
   chmod 600 ~/sensorpajen/config/sensorpajen.env
   

   3. Enable and start service:

   systemctl --user enable sensorpajen
   systemctl --user start sensorpajen
   

   Option 2: Development Installation

   1. Clone Repository

   git clone <repo> ~/sensorpajen
   cd ~/sensorpajen
   

   2. Create Virtual Environment

   python3 -m venv .venv
   source .venv/bin/activate
   pip install -e .
   

Relative Paths (For Portability)

• Project root: ~/sensorpajen/ (or wherever you clone/install)
• Application config: ~/sensorpajen/config/
• Environment file: ~/sensorpajen/config/sensorpajen.env (0600)
• Sensor mapping: ~/sensorpajen/config/sensors.json (0644)
• Service file: ~/.config/systemd/user/sensorpajen.service

Advantages of Relative Paths

• Works on any system (development, production, multiple Raspberry Pis)
• Easy to backup/restore entire directory
• No hardcoded paths in code
• Simple to deploy via git pull or package installation
• User service runs without sudo

APT Package Installation

When installed via .deb package:

• Python package: /usr/lib/python3/dist-packages/sensorpajen/
• Service file: /lib/systemd/user/sensorpajen.service
• Config templates: /usr/share/doc/sensorpajen/examples/
• User config: ~/sensorpajen/config/ (created by user)sensorpajen

  
  5. Verify
  ```bash
  systemctl --user status sensorpajen
  journalctl --user -u sensorpajen -f
  ```uetooth access
  if [ "$1" = "configure" ]; then
      PYTHON_PATH=$(readlink -f /usr/bin/python3)
      setcap 'cap_net_raw,cap_net_admin+eip' "$PYTHON_PATH" || true
  fi
  
  #DEBHELPER#
  

7. Create debian/README.Debian:

   • Installation instructions
   • Configuration guide
   • Service management

8. Build the package:

   dpkg-buildpackage -us -uc -b
   

9. Test installation on Raspberry Pi:

   sudo dpkg -i ../sensorpajen_1.0.0_all.deb
   sudo apt-get install -f  # Fix dependencies if needed
   

10. Create installation documentation:

    • Package installation instructions
    • Configuration setup after installation
    • Service enablement

---

Phase 9: Cleanup & Documentation ✅ DONE (2025-12-27)

Goal: Remove legacy code and finalize documentation

Notes:

• Legacy cron/tmux scripts removed
• Documentation focused on practical usage
• INSTALL.md created for sysadmins

Tasks:

• ✅ Deleted legacy/ folder (old cron/tmux scripts)
• ✅ Created INSTALL.md with concise installation guide
• ✅ Updated README.md troubleshooting section
• ✅ Documentation assumes sysadmin familiarity

---

Migration Complete! 🎉

All phases completed. The system has been successfully migrated from a legacy cron/tmux-based system to a modern systemd service with:

• ✅ Python package structure
• ✅ Environment-based configuration (no .ini files)
• ✅ Systemd user service with auto-restart
• ✅ Automatic sensor discovery with approval workflow
• ✅ Configuration auto-reload (no restart needed)
• ✅ ntfy notifications for new sensors
• ✅ Comprehensive documentation

Version: 2.0.0-dev
Status: Production-ready

## Installation

### 1. Clone Repository
git clone <repo> /home/fredrik/dev/sensorpajen
cd /home/fredrik/dev/sensorpajen

### 2. Create Virtual Environment
python3 -m venv .venv
source .venv/bin/activate
pip install -e .

### 3. Configure
mkdir -p ~/.config/sensorpajen
cp systemd/sensorpajen.env.example ~/.config/sensorpajen/sensorpajen.env
# Edit configuration
nano ~/.config/sensorpajen/sensorpajen.env
chmod 600 ~/.config/sensorpajen/sensorpajen.env

### 4. Convert Sensor Configuration
# Create sensors.json from your sensor list

### 5. Install Service
cp systemd/sensorpajen.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable sensorpajen
systemctl --user start sensorpajen

### 6. Verify
systemctl --user status sensorpajen
journalctl --user -u sensorpajen -f

5. Add troubleshooting section:
   • Bluetooth permission issues
   • MQTT connection problems
   • Service won't start
   • Log locations

---

Configuration File Locations (Linux Best Practices)

User Service Configuration

• Service files: ~/.config/systemd/user/
• Application config: ~/.config/sensorpajen/
• Environment file: ~/.config/sensorpajen/sensorpajen.env (0600)
• Sensor mapping: ~/.config/sensorpajen/sensors.json (0644)

System Service (Alternative - Not Recommended)

If running as system service (not user service):

• Service file: /etc/systemd/system/sensorpajen.service
• Config directory: /etc/sensorpajen/
• Environment file: /etc/sensorpajen/sensorpajen.env (0600)

Recommendation: Use user service (current approach) since:

• No sudo required for service management
• Easier permission management
• Better security isolation
• Simpler Bluetooth access

---

Success Criteria

The migration is complete when:

• ✅ Service starts automatically on boot
• ✅ All 8 Bluetooth sensors are being read
• ✅ MQTT messages are published correctly
• ✅ Service recovers automatically from crashes
• ✅ No hardcoded credentials in code
• ✅ Logs are visible via journalctl
• ✅ DHT11 functionality completely removed
• ✅ Legacy scripts removed
• ✅ Documentation is complete and accurate
• ✅ Service runs as user (not root)
• ✅ Virtual environment is working

---

Rollback Plan

If issues arise during migration:

1. Stop new service:

   systemctl --user stop sensorpajen
   systemctl --user disable sensorpajen
   

2. Restore legacy scripts from legacy/ folder:

   cp legacy/* .
   

3. Restore cron jobs:

   crontab -e
   # Uncomment:
   # @reboot /home/fredrik/dev/sensorpajen/sensorer.sh
   

4. Reboot or manually start tmux session

---

Future Enhancements

After successful migration, consider:

• Add Prometheus metrics endpoint
• Add systemd watchdog support
• Implement graceful sensor failure handling
• Add MQTT TLS support
• Create web dashboard for sensor status
• Add sensor calibration configuration
• Implement sensor auto-discovery
• Add health check endpoint

---

Notes

• Keep legacy scripts during migration for safety
• Test thoroughly before removing cron jobs
• Monitor for at least 1-2 weeks before final cleanup
• Document any issues encountered during migration
• Take notes of actual MAC addresses and sensor names during conversion

---

References

• systemd user services: man systemd.service
• XDG Base Directory: ~/.config/ for user configuration
• Bluetooth capabilities: man capabilities
• journalctl: man journalctl
• Python logging: https://docs.python.org/3/library/logging.html