sensorpajen/ROADMAP.md

# ROADMAP: Modernizing Sensorpajen

## Overview

This roadmap outlines the migration from the current tmux/cron-based system to a modern systemd service running on Raspberry Pi.

**Migration Date**: Started December 27, 2025  
**Target Completion**: TBD

---

## Current State

### What We Have
- LYWSD03MMC.py: Main Bluetooth sensor reader
- temperatur_koksfonstret.py: DHT11 sensor reader (to be removed)
- bluetooth_utils.py: Bluetooth utility functions
- sensorer.ini: MAC address to sensor name mapping
- sendToMQTT.sh: MQTT publishing callback (hardcoded credentials)
- startup.sh/sensorer.sh: tmux-based startup scripts
- Cron jobs for scheduling

### Known Issues
- MQTT credentials hardcoded in shell scripts
- Legacy pirate_audio references in startup.sh
- Manual tmux orchestration
- Mixed configuration sources
- DHT11 functionality to be removed

---

## Target Architecture

### Final Structure
```
sensorpajen/
├── src/
│   └── sensorpajen/
│       ├── __init__.py
│       ├── main.py              # Entry point
│       ├── config.py            # Configuration management
│       ├── sensor_reader.py     # Bluetooth sensor logic
│       ├── mqtt_publisher.py    # MQTT publishing
│       └── utils.py             # Utilities (from bluetooth_utils.py)
├── config/                      # Configuration directory (relative)
│   ├── sensors.json.example     # Sensor mapping template
│   ├── sensorpajen.env.example  # Environment file template
│   ├── sensors.json             # Actual sensor mapping (not in git)
│   └── sensorpajen.env          # Actual environment file (not in git)
├── debian/                      # APT package files
│   ├── control
│   ├── rules
│   ├── changelog
│   └── ...                      # Other Debian package files
├── pyproject.toml               # Project metadata and dependencies
├── requirements.txt             # Dependencies (bluepy, paho-mqtt)
├── README.md                    # Updated documentation
├── AGENTS.md                    # Agent guidelines
├── ROADMAP.md                   # This file
├── legacy/                      # Legacy scripts (moved here temporarily)
│   ├── LYWSD03MMC.py
│   ├── temperatur_koksfonstret.py
│   ├── sendToMQTT.sh
│   ├── startup.sh
│   ├── sensorer.sh
│   └── sensorer.ini
└── systemd/
    ├── sensorpajen.service      # Systemd service unit
    └── README.md                # Systemd installation instructions
```

### Configuration Strategy

Using relative paths for portability across systems:

1. **Sensor Mapping**: `config/sensors.json` (relative to project root)
   - Maps MAC addresses to sensor names
   - JSON format for Python ease
   - Not committed to git (use sensors.json.example as template)

2. **MQTT Credentials**: `config/sensorpajen.env` (relative to project root)
   - Contains sensitive MQTT configuration
   - Permissions: 0600 (owner read/write only)
   - Not committed to git (use sensorpajen.env.example as template)

3. **Environment Variables** (via systemd EnvironmentFile):
   ```
   MQTT_HOST=192.168.0.114
   MQTT_USER=hasse
   MQTT_PASSWORD=casablanca
   MQTT_CLIENT_ID=mibridge
   SENSOR_CONFIG_FILE=config/sensors.json
   ```

4. **Git Ignore**: Add to .gitignore:
   ```
   config/sensors.json
   config/sensorpajen.env
   ```

---

## Migration Phases

### Phase 1: Preparation & Cleanup ✅ DONE (2025-12-27)
**Goal**: Reorganize repository without breaking existing functionality

**Notes**: 
- Created modern Python package structure with src/ layout
- Converted INI sensor config to JSON format (sensors.json.example)
- Environment-based configuration instead of hardcoded values
- DHT11 sensor functionality removed as planned
- Legacy scripts preserved in legacy/ folder

#### Tasks:
- ✅ Create new directory structure
- ✅ Create pyproject.toml with dependencies
- ✅ Remove DHT11 functionality
- ✅ Move legacy scripts to legacy/ folder
- ✅ Create config file templates (sensors.json.example, sensorpajen.env.example)
- ✅ Preserve requirements.txt for backward compatibility

---

### Phase 2: Python Package Structure ✅ DONE (2025-12-27)
**Goal**: Create modern Python package with proper entry point

**Notes**:
- Used src/ layout for better packaging practices
- Direct Python MQTT integration (no shell script callbacks)
- ATC firmware BLE advertisement reading (passive scanning)
- Watchdog thread for BLE connection recovery
- Clean separation of concerns (config, MQTT, sensors, main)

#### Tasks:
- ✅ Created src/sensorpajen/__init__.py with version info
- ✅ Created src/sensorpajen/config.py
  - Environment variable loading with validation
  - SensorConfig class for JSON sensor mapping
  - Relative path resolution (PROJECT_ROOT)
  - Configuration validation and logging
- ✅ Created src/sensorpajen/utils.py
  - Ported bluetooth_utils.py (MIT licensed, Colin GUYON)
  - BLE scanning and advertisement parsing
- ✅ Created src/sensorpajen/mqtt_publisher.py
  - MQTTPublisher class with connection management
  - Direct publishing (replaces sendToMQTT.sh)
  - Automatic reconnection support
  - Battery data publishing (optional)
- ✅ Created src/sensorpajen/sensor_reader.py
  - SensorReader class for BLE scanning
  - ATC packet parsing
  - Duplicate packet filtering
  - Watchdog for BLE recovery
  - Measurement dataclass
- ✅ Created src/sensorpajen/main.py
  - Application entry point
  - Signal handling (SIGTERM, SIGINT)
  - Graceful shutdown
  - Logging to stdout for journald

---

### Phase 3: Configuration Migration ✅ DONE (2025-12-27)
**Goal**: Replace .ini file with JSON and environment variables

**Notes**: Templates created in Phase 1, successfully tested on Raspberry Pi

#### Tasks:
1. Create sensor mapping converter script
   - Read sensorer.ini
   - Output to sensors.json
   ```json
   {
     "sensors": [
       {
         "mac": "A4:C1:38:98:7B:B6",
         "name": "mi_temp_1"
       },
       {
         "mac": "A4:C1:38:29:03:0D",
         "name": "mi_temp_2"
       }
     ]
   }
   ```
configuration file templates
   - `config/sensorpajen.env.example`
   ```bash
   # MQTT Configuration
   MQTT_HOST=192.168.0.114
   MQTT_PORT=1883
   MQTT_USER=hasse
   MQTT_PASSWORD=casablanca
   MQTT_CLIENT_ID=mibridge
   
   # Sensor Configuration (relative to project root)
   SENSOR_CONFIG_FILE=config/sensors.json
   
   # Application Settings
   WATCHDOG_TIMEOUT=5
   ENABLE_BATTERY=true
   LOG_LEVEL=INFO
   ```

   - `config/sensors.json.example`
   ```json
   {
     "sensors": [
       {
         "mac": "A4:C1:38:98:7B:B6",
         "name": "mi_temp_1",
         "comment": "Example sensor"
       }
     ]
   }
   ```

3. Copy templates to actual config files (not in git):
   ```bash
   cp config/sensorpajen.env.example config/sensorpajen.env
   cp config/sensors.json.example config/sensors.json
   chmod 600 config/sensorpajen.env
   # Edit both files with your actual configurationnsorpajen/sensorpajen.env
   chmod 600 /home/fredrik/.config/sensorpajen/sensorpajen.env
   ```

4. Document all configuration variables in README

---
config/sensorpajen.env
   config/sensors.json
   *.deb
   debian/.debhelper/
   debian/sensorpajen/
   debian/files
   debian/*.log
   debian/*.substvars
### Phase 4: Virtual Environment & Dependencies ✅ DONE (2025-12-27)
**Goal**: Set up isolated Python environment

**Notes**: Tested on Raspberry Pi, paho-mqtt v2.x compatibility fixed

#### Tasks:
1. Create virtual environment:
   ```bash
   python3 -m venv .venv
   ```

2. Update .gitignore:
   ```
   .venv/
   __pycache__/
   *.pyc
   .env
   sensorpajen.env
   ```

3. Install dependencies:
   ```bash
   source .venv/bin/activate
   pip install --upgrade pip
   pip install bluepy paho-mqtt
   pip install -e .  # Install package in development mode
   ```

4. Document virtual environment usage in README

---✅ DONE (2025-12-27)
**Goal**: Allow non-root user to access Bluetooth

**Notes**: Tested on Raspberry Pi with setcap on actual Python binary
### Phase 5: Bluetooth Permissions ✅ DONE (2025-12-27)
**Goal**: Allow non-root user to access Bluetooth

**Notes**: Tested on Raspberry Pi with setcap on actual Python binary

#### Tasks:
- ✅ Bluetooth capabilities set with setcap
- ✅ Documented in SETUP_ON_PI.md with correct readlink -f usage
- ✅ Tested successfully on Raspberry Pi

---

### Phase 6: Systemd Service Creation ✅ DONE (2025-12-27)
**Goal**: Create and configure systemd user service

**Notes**:
- User service for easier management (no sudo required)
- Service ready for installation on Raspberry Pi
- Comprehensive documentation provided
- **Important discoveries**:
  - `AmbientCapabilities` does NOT work in user services (only system services)
  - Must use `setcap` on the Python binary instead
  - `NoNewPrivileges=true` prevents file capabilities from working - must be disabled
  - Capabilities must be set on actual binary, not symlinks: `setcap ... $(readlink -f python3)`

#### Tasks:
- ✅ Created systemd/sensorpajen.service
- ✅ Created systemd/README.md with full documentation
- ✅ Service management and troubleshooting guides included
- ✅ Tested and verified working on Raspberry Pi

---

### Phase 7: Testing & Validation ✅ DONE (2025-12-27)
**Goal**: Verify new service works before removing legacy

**Notes**:
- Service tested and running successfully
- Legacy cron/tmux system stopped
- All sensors reporting correctly via systemd service

#### Tasks:
- ✅ Stopped legacy cron/tmux processes
- ✅ Started new systemd service
- ✅ Monitored logs - no errors
- ✅ Verified all 8 sensors reporting
- ✅ Confirmed MQTT publishing working
- ✅ Tested service restart and auto-recovery

---

### Phase 8: APT Package Creation ✅ DONE (2025-12-27)
**Goal**: Create Debian package for easy installation on Raspberry Pi

**Notes**:
- Complete debian/ directory structure created
- System-wide installation to /opt/sensorpajen
- Configuration in /etc/sensorpajen
- Dedicated sensorpajen system user
- Automatic venv creation in postinst
- Bluetooth capabilities set automatically
- Config preserved on remove/purge for safety
- Dual-mode support: system installation and development
- config.py auto-detects installation type

#### Files Created:
- ✅ debian/control - Package metadata and dependencies
- ✅ debian/compat - Debhelper compatibility level
- ✅ debian/changelog - Package version history
- ✅ debian/rules - Build instructions
- ✅ debian/install - File installation mappings
- ✅ debian/postinst - Post-installation script (user, venv, setcap)
- ✅ debian/prerm - Pre-removal script (stop service)
- ✅ debian/postrm - Post-removal script (cleanup)
- ✅ debian/sensorpajen.service - System-wide systemd unit

#### Code Updates:
- ✅ Updated src/sensorpajen/config.py to detect system installation
  - Checks for /opt/sensorpajen existence
  - Uses /etc/sensorpajen for config in system mode
  - Falls back to PROJECT_ROOT/config for development
- ✅ Updated scripts/approve-sensors.sh for dual-mode operation
  - Detects system vs development installation
  - Uses correct venv and config paths
- ✅ Created scripts/verify-deb.sh - Automated build and verification

#### Package Details:
- Package name: sensorpajen
- Version: 2.0.0-dev
- Architecture: all
- System paths:
  - Application: /opt/sensorpajen/
  - Configuration: /etc/sensorpajen/
  - Service file: /etc/systemd/system/sensorpajen.service
  - Examples: /usr/share/doc/sensorpajen/examples/
- Runs as dedicated sensorpajen user (system account)
- Auto-enables service but waits for configuration before starting

#### Build and Test:
```bash
# Build package
./scripts/verify-deb.sh

# Or manually:
dpkg-buildpackage -us -uc -b
lintian ../sensorpajen_*.deb

# Install on Raspberry Pi:
scp ../sensorpajen_*.deb pi@raspberrypi:~/
ssh pi@raspberrypi
sudo apt install ./sensorpajen_*.deb

# Configure:
sudo nano /etc/sensorpajen/sensorpajen.env
sudo nano /etc/sensorpajen/sensors.json

# Start:
sudo systemctl start sensorpajen
sudo journalctl -u sensorpajen -f
```

---

### Phase 9: Cleanup & Documentation ✅ DONE (2025-12-27)
**Goal**: Remove legacy code and finalize documentation

**Notes**:
- Legacy cron/tmux scripts removed
- Documentation focused on practical usage
- INSTALL.md created for sysadmins

#### Tasks:
- ✅ Deleted legacy/ folder (old cron/tmux scripts)
- ✅ Created INSTALL.md with concise installation guide
- ✅ Updated README.md troubleshooting section
- ✅ Documentation assumes sysadmin familiarity

---

## Migration Complete! 🎉

All phases completed. The system has been successfully migrated from a legacy cron/tmux-based system to a modern systemd service with:

- ✅ Python package structure
- ✅ Environment-based configuration (no .ini files)
- ✅ Systemd user service with auto-restart
- ✅ Automatic sensor discovery with approval workflow
- ✅ Configuration auto-reload (no restart needed)
- ✅ ntfy notifications for new sensors
- ✅ Comprehensive documentation

**Version**: 2.0.0-dev  
**Status**: Production-ready
   ```markdown
   ## Installation
   
   ### 1. Clone Repository
   git clone <repo> /home/fredrik/dev/sensorpajen
   cd /home/fredrik/dev/sensorpajen
   
   ### 2. Create Virtual Environment
   python3 -m venv .venv
   source .venv/bin/activate
   pip install -e .
   
   ### 3. Configure
   mkdir -p ~/.config/sensorpajen
   cp systemd/sensorpajen.env.example ~/.config/sensorpajen/sensorpajen.env
   # Edit configuration
   nano ~/.config/sensorpajen/sensorpajen.env
   chmod 600 ~/.config/sensorpajen/sensorpajen.env
   
   ### 4. Convert Sensor Configuration
   # Create sensors.json from your sensor list
   
   ### 5. Install Service
   cp systemd/sensorpajen.service ~/.config/systemd/user/
   systemctl --user daemon-reload
   systemctl --user enable sensorpajen
   systemctl --user start sensorpajen
   
   ### 6. Verify
   systemctl --user status sensorpajen
   journalctl --user -u sensorpajen -f
   ```

5. Add troubleshooting section:
   - Bluetooth permission issues
   - MQTT connection problems
   - Service won't start
   - Log locations

---

## Configuration File Locations (Linux Best Practices)

### User Service Configuration
- **Service files**: `~/.config/systemd/user/`
- **Application config**: `~/.config/sensorpajen/`
- **Environment file**: `~/.config/sensorpajen/sensorpajen.env` (0600)
- **Sensor mapping**: `~/.config/sensorpajen/sensors.json` (0644)

### System Service (Alternative - Not Recommended)
If running as system service (not user service):
- **Service file**: `/etc/systemd/system/sensorpajen.service`
- **Config directory**: `/etc/sensorpajen/`
- **Environment file**: `/etc/sensorpajen/sensorpajen.env` (0600)

**Recommendation**: Use user service (current approach) since:
- No sudo required for service management
- Easier permission management
- Better security isolation
- Simpler Bluetooth access

---

## Success Criteria

The migration is complete when:

- ✅ Service starts automatically on boot
- ✅ All 8 Bluetooth sensors are being read
- ✅ MQTT messages are published correctly
- ✅ Service recovers automatically from crashes
- ✅ No hardcoded credentials in code
- ✅ Logs are visible via journalctl
- ✅ DHT11 functionality completely removed
- ✅ Legacy scripts removed
- ✅ Documentation is complete and accurate
- ✅ Service runs as user (not root)
- ✅ Virtual environment is working

---

## Rollback Plan

If issues arise during migration:

1. Stop new service:
   ```bash
   systemctl --user stop sensorpajen
   systemctl --user disable sensorpajen
   ```

2. Restore legacy scripts from legacy/ folder:
   ```bash
   cp legacy/* .
   ```

3. Restore cron jobs:
   ```bash
   crontab -e
   # Uncomment:
   # @reboot /home/fredrik/dev/sensorpajen/sensorer.sh
   ```

4. Reboot or manually start tmux session

---

## Future Enhancements

After successful migration, consider:

- [ ] Add Prometheus metrics endpoint
- [ ] Add systemd watchdog support
- [ ] Implement graceful sensor failure handling
- [ ] Add MQTT TLS support
- [ ] Create web dashboard for sensor status
- [ ] Add sensor calibration configuration
- [ ] Implement sensor auto-discovery
- [ ] Add health check endpoint

---

## Notes

- Keep legacy scripts during migration for safety
- Test thoroughly before removing cron jobs
- Monitor for at least 1-2 weeks before final cleanup
- Document any issues encountered during migration
- Take notes of actual MAC addresses and sensor names during conversion

---

## References

- systemd user services: `man systemd.service`
- XDG Base Directory: `~/.config/` for user configuration
- Bluetooth capabilities: `man capabilities`
- journalctl: `man journalctl`
- Python logging: https://docs.python.org/3/library/logging.html