sensorpajen/ROADMAP.md

# ROADMAP: Modernizing Sensorpajen

## Overview

This roadmap outlines the migration from the current tmux/cron-based system to a modern systemd service running on Raspberry Pi.

**Migration Date**: Started December 27, 2025  
**Target Completion**: TBD

---

## Current State

### What We Have
- LYWSD03MMC.py: Main Bluetooth sensor reader
- temperatur_koksfonstret.py: DHT11 sensor reader (to be removed)
- bluetooth_utils.py: Bluetooth utility functions
- sensorer.ini: MAC address to sensor name mapping
- sendToMQTT.sh: MQTT publishing callback (hardcoded credentials)
- startup.sh/sensorer.sh: tmux-based startup scripts
- Cron jobs for scheduling

### Known Issues
- MQTT credentials hardcoded in shell scripts
- Legacy pirate_audio references in startup.sh
- Manual tmux orchestration
- Mixed configuration sources
- DHT11 functionality to be removed

---

## Target Architecture

### Final Structure
```
sensorpajen/
├── src/
│   └── sensorpajen/
│       ├── __init__.py
│       ├── main.py              # Entry point
│       ├── config.py            # Configuration management
│       ├── sensor_reader.py     # Bluetooth sensor logic
│       ├── mqtt_publisher.py    # MQTT publishing
│       └── utils.py             # Utilities (from bluetooth_utils.py)
├── config/                      # Configuration directory (relative)
│   ├── sensors.json.example     # Sensor mapping template
│   ├── sensorpajen.env.example  # Environment file template
│   ├── sensors.json             # Actual sensor mapping (not in git)
│   └── sensorpajen.env          # Actual environment file (not in git)
├── debian/                      # APT package files
│   ├── control
│   ├── rules
│   ├── changelog
│   └── ...                      # Other Debian package files
├── pyproject.toml               # Project metadata and dependencies
├── requirements.txt             # Dependencies (bluepy, paho-mqtt)
├── README.md                    # Updated documentation
├── AGENTS.md                    # Agent guidelines
├── ROADMAP.md                   # This file
├── legacy/                      # Legacy scripts (moved here temporarily)
│   ├── LYWSD03MMC.py
│   ├── temperatur_koksfonstret.py
│   ├── sendToMQTT.sh
│   ├── startup.sh
│   ├── sensorer.sh
│   └── sensorer.ini
└── systemd/
    ├── sensorpajen.service      # Systemd service unit
    └── README.md                # Systemd installation instructions
```

### Configuration Strategy

Using relative paths for portability across systems:

1. **Sensor Mapping**: `config/sensors.json` (relative to project root)
   - Maps MAC addresses to sensor names
   - JSON format for Python ease
   - Not committed to git (use sensors.json.example as template)

2. **MQTT Credentials**: `config/sensorpajen.env` (relative to project root)
   - Contains sensitive MQTT configuration
   - Permissions: 0600 (owner read/write only)
   - Not committed to git (use sensorpajen.env.example as template)

3. **Environment Variables** (via systemd EnvironmentFile):
   ```
   MQTT_HOST=192.168.0.114
   MQTT_USER=hasse
   MQTT_PASSWORD=casablanca
   MQTT_CLIENT_ID=mibridge
   SENSOR_CONFIG_FILE=config/sensors.json
   ```

4. **Git Ignore**: Add to .gitignore:
   ```
   config/sensors.json
   config/sensorpajen.env
   ```

---

## Migration Phases

### Phase 1: Preparation & Cleanup ✅ DONE (2025-12-27)
**Goal**: Reorganize repository without breaking existing functionality

**Notes**: 
- Created modern Python package structure with src/ layout
- Converted INI sensor config to JSON format (sensors.json.example)
- Environment-based configuration instead of hardcoded values
- DHT11 sensor functionality removed as planned
- Legacy scripts preserved in legacy/ folder

#### Tasks:
- ✅ Create new directory structure
- ✅ Create pyproject.toml with dependencies
- ✅ Remove DHT11 functionality
- ✅ Move legacy scripts to legacy/ folder
- ✅ Create config file templates (sensors.json.example, sensorpajen.env.example)
- ✅ Preserve requirements.txt for backward compatibility

---

### Phase 2: Python Package Structure ✅ DONE (2025-12-27)
**Goal**: Create modern Python package with proper entry point

**Notes**:
- Used src/ layout for better packaging practices
- Direct Python MQTT integration (no shell script callbacks)
- ATC firmware BLE advertisement reading (passive scanning)
- Watchdog thread for BLE connection recovery
- Clean separation of concerns (config, MQTT, sensors, main)

#### Tasks:
- ✅ Created src/sensorpajen/__init__.py with version info
- ✅ Created src/sensorpajen/config.py
  - Environment variable loading with validation
  - SensorConfig class for JSON sensor mapping
  - Relative path resolution (PROJECT_ROOT)
  - Configuration validation and logging
- ✅ Created src/sensorpajen/utils.py
  - Ported bluetooth_utils.py (MIT licensed, Colin GUYON)
  - BLE scanning and advertisement parsing
- ✅ Created src/sensorpajen/mqtt_publisher.py
  - MQTTPublisher class with connection management
  - Direct publishing (replaces sendToMQTT.sh)
  - Automatic reconnection support
  - Battery data publishing (optional)
- ✅ Created src/sensorpajen/sensor_reader.py
  - SensorReader class for BLE scanning
  - ATC packet parsing
  - Duplicate packet filtering
  - Watchdog for BLE recovery
  - Measurement dataclass
- ✅ Created src/sensorpajen/main.py
  - Application entry point
  - Signal handling (SIGTERM, SIGINT)
  - Graceful shutdown
  - Logging to stdout for journald

---

### Phase 3: Configuration Migration ✅ DONE (2025-12-27)
**Goal**: Replace .ini file with JSON and environment variables

**Notes**: Templates created in Phase 1, successfully tested on Raspberry Pi

#### Tasks:
1. Create sensor mapping converter script
   - Read sensorer.ini
   - Output to sensors.json
   ```json
   {
     "sensors": [
       {
         "mac": "A4:C1:38:98:7B:B6",
         "name": "mi_temp_1"
       },
       {
         "mac": "A4:C1:38:29:03:0D",
         "name": "mi_temp_2"
       }
     ]
   }
   ```
configuration file templates
   - `config/sensorpajen.env.example`
   ```bash
   # MQTT Configuration
   MQTT_HOST=192.168.0.114
   MQTT_PORT=1883
   MQTT_USER=hasse
   MQTT_PASSWORD=casablanca
   MQTT_CLIENT_ID=mibridge
   
   # Sensor Configuration (relative to project root)
   SENSOR_CONFIG_FILE=config/sensors.json
   
   # Application Settings
   WATCHDOG_TIMEOUT=5
   ENABLE_BATTERY=true
   LOG_LEVEL=INFO
   ```

   - `config/sensors.json.example`
   ```json
   {
     "sensors": [
       {
         "mac": "A4:C1:38:98:7B:B6",
         "name": "mi_temp_1",
         "comment": "Example sensor"
       }
     ]
   }
   ```

3. Copy templates to actual config files (not in git):
   ```bash
   cp config/sensorpajen.env.example config/sensorpajen.env
   cp config/sensors.json.example config/sensors.json
   chmod 600 config/sensorpajen.env
   # Edit both files with your actual configurationnsorpajen/sensorpajen.env
   chmod 600 /home/fredrik/.config/sensorpajen/sensorpajen.env
   ```

4. Document all configuration variables in README

---
config/sensorpajen.env
   config/sensors.json
   *.deb
   debian/.debhelper/
   debian/sensorpajen/
   debian/files
   debian/*.log
   debian/*.substvars
### Phase 4: Virtual Environment & Dependencies ✅ DONE (2025-12-27)
**Goal**: Set up isolated Python environment

**Notes**: Tested on Raspberry Pi, paho-mqtt v2.x compatibility fixed

#### Tasks:
1. Create virtual environment:
   ```bash
   python3 -m venv .venv
   ```

2. Update .gitignore:
   ```
   .venv/
   __pycache__/
   *.pyc
   .env
   sensorpajen.env
   ```

3. Install dependencies:
   ```bash
   source .venv/bin/activate
   pip install --upgrade pip
   pip install bluepy paho-mqtt
   pip install -e .  # Install package in development mode
   ```

4. Document virtual environment usage in README

---✅ DONE (2025-12-27)
**Goal**: Allow non-root user to access Bluetooth

**Notes**: Tested on Raspberry Pi with setcap on actual Python binary
### Phase 5: Bluetooth Permissions ✅ DONE (2025-12-27)
**Goal**: Allow non-root user to access Bluetooth

**Notes**: Tested on Raspberry Pi with setcap on actual Python binary

#### Tasks:
- ✅ Bluetooth capabilities set with setcap
- ✅ Documented in SETUP_ON_PI.md with correct readlink -f usage
- ✅ Tested successfully on Raspberry Pi

---

### Phase 6: Systemd Service Creation ✅ DONE (2025-12-27)
**Goal**: Create and configure systemd user service

**Notes**:
- User service for easier management (no sudo required)
- Service ready for installation on Raspberry Pi
- Comprehensive documentation provided
- **Important discoveries**:
  - `AmbientCapabilities` does NOT work in user services (only system services)
  - Must use `setcap` on the Python binary instead
  - `NoNewPrivileges=true` prevents file capabilities from working - must be disabled
  - Capabilities must be set on actual binary, not symlinks: `setcap ... $(readlink -f python3)`

#### Tasks:
- ✅ Created systemd/sensorpajen.service
- ✅ Created systemd/README.md with full documentation
- ✅ Service management and troubleshooting guides included
- ✅ Tested and verified working on Raspberry Pi

---

### Phase 7: Testing & Validation ✓ TODO
**Goal**: Verify new service works before removing legacy

#### Tasks:
1. Stop legacy cron/tmux processes:
   ```bash
   crontab -e  # Comment out sensorpajen entries
   tmux kill-session -t sensorer
   ```

2. Start new service:
   ```bash
   systemctl --user start sensorpajen
   ```

3. Monitor logs:
   ```bash
   journalctl --user -u sensorpajen -f
   ```APT Package Creation ✓ TODO
**Goal**: Create Debian package for easy installation on Raspberry Pi

#### Tasks:
1. Create debian/ directory structure:
   ```bash
   mkdir -p debian
   ```

2. Create `debian/control`:
   ``APT package installation instructions
   - Development installation instructions
   - Configuration guide (relative paths)
   - Service management commands
   - Troubleshooting section
   - Remove DHT11 references
   - Remove pirate_audio references

3. Create INSTALL.md:
   - APT package installation steps
   - Manual installation steps
   - Configuration examples
   - First-time setup guide
   - Raspberry Pi specific instructionsds}, ${misc:Depends},
            python3-bluepy,
            python3-paho-mqtt,
            bluetooth,
            bluez
   Description: Bluetooth temperature sensor monitor
    Monitors Xiaomi Mijia LYWSD03MMC Bluetooth temperature sensors
    and publishes data to MQTT broker.
   ```

3. Create `debian/rules`:
   ```makefile
   #!/usr/bin/make -f
   
   %:
   	dh $@ --with python3 --buildsystem=pybuild
   
   override_dh_auto_install:
   	pytOption 1: APT Package (Recommended for Raspberry Pi)
   
   1. Download and install the .deb package:
   ```bash
   sudo dpkg -i sensorpajen_1.0.0_all.deb
   sudo apt-get install -f  # Fix any dependencies
   ```
   
   2. Configure:
   ```bash
   mkdir -p ~/sensorpajen/config
   cp /usr/share/doc/sensorpajen/examples/sensorpajen.env.example ~/sensorpajen/config/sensorpajen.env
   cp /usr/share/doc/sensorpajen/examples/sensors.json.example ~/sensorpajen/config/sensors.json
   # Edit both files
   nano ~/sensorpajen/config/sensorpajen.env
   nano ~/sensorpajen/config/sensors.json
   chmod 600 ~/sensorpajen/config/sensorpajen.env
   ```
   
   3. Enable and start service:
   ```bash
   systemctl --user enable sensorpajen
   systemctl --user start sensorpajen
   ```
   
   ### Option 2: Development Installation
   
   1. Clone Repository
   ```bash
   git clone <repo> ~/sensorpajen
   cd ~/sensorpajen
   ```
   
   2. Create Virtual Environment
   ```bash
   python3 -m venv .venv
   source .venv/bin/activate
   pip install -e .
   ```

### Relative Paths (For Portability)
- **Project root**: `~/sensorpajen/` (or wherever you clone/install)
- **Application config**: `~/sensorpajen/config/`
- **Environment file**: `~/sensorpajen/config/sensorpajen.env` (0600)
- **Sensor mapping**: `~/sensorpajen/config/sensors.json` (0644)
- **Service file**: `~/.config/systemd/user/sensorpajen.service`

### Advantages of Relative Paths
- Works on any system (development, production, multiple Raspberry Pis)
- Easy to backup/restore entire directory
- No hardcoded paths in code
- Simple to deploy via git pull or package installation
- User service runs without sudo

### APT Package Installation
When installed via .deb package:
- **Python package**: `/usr/lib/python3/dist-packages/sensorpajen/`
- **Service file**: `/lib/systemd/user/sensorpajen.service`
- **Config templates**: `/usr/share/doc/sensorpajen/examples/`
- **User config**: `~/sensorpajen/config/` (created by user)sensorpajen
   ```
   
   5. Verify
   ```bash
   systemctl --user status sensorpajen
   journalctl --user -u sensorpajen -f
   ```uetooth access
   if [ "$1" = "configure" ]; then
       PYTHON_PATH=$(readlink -f /usr/bin/python3)
       setcap 'cap_net_raw,cap_net_admin+eip' "$PYTHON_PATH" || true
   fi
   
   #DEBHELPER#
   ```

7. Create `debian/README.Debian`:
   - Installation instructions
   - Configuration guide
   - Service management

8. Build the package:
   ```bash
   dpkg-buildpackage -us -uc -b
   ```

9. Test installation on Raspberry Pi:
   ```bash
   sudo dpkg -i ../sensorpajen_1.0.0_all.deb
   sudo apt-get install -f  # Fix dependencies if needed
   ```

10. Create installation documentation:
    - Package installation instructions
    - Configuration setup after installation
    - Service enablement

---

### Phase 9: 

4. Verify MQTT messages:
   ```bash
   mosquitto_sub -h 192.168.0.114 -u hasse -P casablanca -t "MiTemperature2/#" -v
   ```

5. Test service restart:
   ```bash
   systemctl --user restart sensorpajen
   ```

6. Test crash recovery (kill process, verify auto-restart)

7. Test boot behavior:
   ```bash
   systemctl --user enable sensorpajen
   sudo reboot
   # After reboot, verify service is running
   systemctl --user status sensorpajen
   ```

---

### Phase 8: Cleanup & Document
- [ ] Publish APT package to personal repository
- [ ] Create automated build pipeline for .deb packages
- [ ] Add support for multiple MQTT brokers
- [ ] Implement configuration validation toolation ✓ TODO
**Goal**: Remove legacy code and finalize documentation

#### Tasks:
1. Once new service is stable (run for 1-2 weeks):
   - Delete legacy/ folder
   - Remove cron jobs completely
   - Remove tmux session references

2. Update README.md:
   - Installation instructions
   - Configuration guide
   - Service management commands
   - Troubleshooting section
   - Remove DHT11 references
   - Remove pirate_audio references

3. Create INSTALL.md:
   - Fresh installation steps
   - Configuration examples
   - First-time setup guide

4. Document in README:
   ```markdown
   ## Installation
   
   ### 1. Clone Repository
   git clone <repo> /home/fredrik/dev/sensorpajen
   cd /home/fredrik/dev/sensorpajen
   
   ### 2. Create Virtual Environment
   python3 -m venv .venv
   source .venv/bin/activate
   pip install -e .
   
   ### 3. Configure
   mkdir -p ~/.config/sensorpajen
   cp systemd/sensorpajen.env.example ~/.config/sensorpajen/sensorpajen.env
   # Edit configuration
   nano ~/.config/sensorpajen/sensorpajen.env
   chmod 600 ~/.config/sensorpajen/sensorpajen.env
   
   ### 4. Convert Sensor Configuration
   # Create sensors.json from your sensor list
   
   ### 5. Install Service
   cp systemd/sensorpajen.service ~/.config/systemd/user/
   systemctl --user daemon-reload
   systemctl --user enable sensorpajen
   systemctl --user start sensorpajen
   
   ### 6. Verify
   systemctl --user status sensorpajen
   journalctl --user -u sensorpajen -f
   ```

5. Add troubleshooting section:
   - Bluetooth permission issues
   - MQTT connection problems
   - Service won't start
   - Log locations

---

## Configuration File Locations (Linux Best Practices)

### User Service Configuration
- **Service files**: `~/.config/systemd/user/`
- **Application config**: `~/.config/sensorpajen/`
- **Environment file**: `~/.config/sensorpajen/sensorpajen.env` (0600)
- **Sensor mapping**: `~/.config/sensorpajen/sensors.json` (0644)

### System Service (Alternative - Not Recommended)
If running as system service (not user service):
- **Service file**: `/etc/systemd/system/sensorpajen.service`
- **Config directory**: `/etc/sensorpajen/`
- **Environment file**: `/etc/sensorpajen/sensorpajen.env` (0600)

**Recommendation**: Use user service (current approach) since:
- No sudo required for service management
- Easier permission management
- Better security isolation
- Simpler Bluetooth access

---

## Success Criteria

The migration is complete when:

- ✅ Service starts automatically on boot
- ✅ All 8 Bluetooth sensors are being read
- ✅ MQTT messages are published correctly
- ✅ Service recovers automatically from crashes
- ✅ No hardcoded credentials in code
- ✅ Logs are visible via journalctl
- ✅ DHT11 functionality completely removed
- ✅ Legacy scripts removed
- ✅ Documentation is complete and accurate
- ✅ Service runs as user (not root)
- ✅ Virtual environment is working

---

## Rollback Plan

If issues arise during migration:

1. Stop new service:
   ```bash
   systemctl --user stop sensorpajen
   systemctl --user disable sensorpajen
   ```

2. Restore legacy scripts from legacy/ folder:
   ```bash
   cp legacy/* .
   ```

3. Restore cron jobs:
   ```bash
   crontab -e
   # Uncomment:
   # @reboot /home/fredrik/dev/sensorpajen/sensorer.sh
   ```

4. Reboot or manually start tmux session

---

## Future Enhancements

After successful migration, consider:

- [ ] Add Prometheus metrics endpoint
- [ ] Add systemd watchdog support
- [ ] Implement graceful sensor failure handling
- [ ] Add MQTT TLS support
- [ ] Create web dashboard for sensor status
- [ ] Add sensor calibration configuration
- [ ] Implement sensor auto-discovery
- [ ] Add health check endpoint

---

## Notes

- Keep legacy scripts during migration for safety
- Test thoroughly before removing cron jobs
- Monitor for at least 1-2 weeks before final cleanup
- Document any issues encountered during migration
- Take notes of actual MAC addresses and sensor names during conversion

---

## References

- systemd user services: `man systemd.service`
- XDG Base Directory: `~/.config/` for user configuration
- Bluetooth capabilities: `man capabilities`
- journalctl: `man journalctl`
- Python logging: https://docs.python.org/3/library/logging.html