Add project documentation: AGENTS.md and ROADMAP.md

- AGENTS.md: Guidelines for AI agents working on this project
- ROADMAP.md: Complete migration plan from tmux/cron to systemd
- Configuration strategy using relative paths
- APT package creation plan (Phase 8)
- Progress tracking instructions
This commit is contained in:
2025-12-27 13:10:06 +01:00
parent 1c20f50379
commit 219ef3240c
2 changed files with 991 additions and 0 deletions

753
ROADMAP.md Normal file
View File

@@ -0,0 +1,753 @@
# ROADMAP: Modernizing Sensorpajen
## Overview
This roadmap outlines the migration from the current tmux/cron-based system to a modern systemd service running on Raspberry Pi.
**Migration Date**: Started December 27, 2025
**Target Completion**: TBD
---
## Current State
### What We Have
- LYWSD03MMC.py: Main Bluetooth sensor reader
- temperatur_koksfonstret.py: DHT11 sensor reader (to be removed)
- bluetooth_utils.py: Bluetooth utility functions
- sensorer.ini: MAC address to sensor name mapping
- sendToMQTT.sh: MQTT publishing callback (hardcoded credentials)
- startup.sh/sensorer.sh: tmux-based startup scripts
- Cron jobs for scheduling
### Known Issues
- MQTT credentials hardcoded in shell scripts
- Legacy pirate_audio references in startup.sh
- Manual tmux orchestration
- Mixed configuration sources
- DHT11 functionality to be removed
---
## Target Architecture
### Final Structure
```
sensorpajen/
├── src/
│ └── sensorpajen/
│ ├── __init__.py
│ ├── main.py # Entry point
│ ├── config.py # Configuration management
│ ├── sensor_reader.py # Bluetooth sensor logic
│ ├── mqtt_publisher.py # MQTT publishing
│ └── utils.py # Utilities (from bluetooth_utils.py)
├── config/ # Configuration directory (relative)
│ ├── sensors.json.example # Sensor mapping template
│ ├── sensorpajen.env.example # Environment file template
│ ├── sensors.json # Actual sensor mapping (not in git)
│ └── sensorpajen.env # Actual environment file (not in git)
├── debian/ # APT package files
│ ├── control
│ ├── rules
│ ├── changelog
│ └── ... # Other Debian package files
├── pyproject.toml # Project metadata and dependencies
├── requirements.txt # Dependencies (bluepy, paho-mqtt)
├── README.md # Updated documentation
├── AGENTS.md # Agent guidelines
├── ROADMAP.md # This file
├── legacy/ # Legacy scripts (moved here temporarily)
│ ├── LYWSD03MMC.py
│ ├── temperatur_koksfonstret.py
│ ├── sendToMQTT.sh
│ ├── startup.sh
│ ├── sensorer.sh
│ └── sensorer.ini
└── systemd/
├── sensorpajen.service # Systemd service unit
└── README.md # Systemd installation instructions
```
### Configuration Strategy
Using relative paths for portability across systems:
1. **Sensor Mapping**: `config/sensors.json` (relative to project root)
- Maps MAC addresses to sensor names
- JSON format for Python ease
- Not committed to git (use sensors.json.example as template)
2. **MQTT Credentials**: `config/sensorpajen.env` (relative to project root)
- Contains sensitive MQTT configuration
- Permissions: 0600 (owner read/write only)
- Not committed to git (use sensorpajen.env.example as template)
3. **Environment Variables** (via systemd EnvironmentFile):
```
MQTT_HOST=192.168.0.114
MQTT_USER=hasse
MQTT_PASSWORD=casablanca
MQTT_CLIENT_ID=mibridge
SENSOR_CONFIG_FILE=config/sensors.json
```
4. **Git Ignore**: Add to .gitignore:
```
config/sensors.json
config/sensorpajen.env
```
---
## Migration Phases
### Phase 1: Preparation & Cleanup ✓ TODO
**Goal**: Reorganize repository without breaking existing functionality
#### Tasks:
1. Create new directory structure
```bash
mkdir -p src/sensorpajen
mkdir -p config
mkdir -p legacy
mkdir -p systemd
mkdir -p debian
```
2. Create pyproject.toml with dependencies:
- bluepy
- paho-mqtt
3. Remove DHT11 functionality:
- Delete temperatur_koksfonstret.py
- Remove DHT11 cron job from documentation
- Update README.md
4. Move legacy scripts to legacy/ folder:
- LYWSD03MMC.py
- sendToMQTT.sh
- startup.sh
- sensorer.sh
- sensorer.ini
- bluetooth_utils.py
5. Verify existing system still works with legacy scripts
---
### Phase 2: Python Package Structure ✓ TODO
**Goal**: Create modern Python package with proper entry point
#### Tasks:
1. Create `src/sensorpajen/__init__.py`
- Package initialization
- Version information
2. Create `src/sensorpajen/config.py`
- Environment variable loading
- Configuration validation
- Default values
- Fail-fast on missing required config
```python
import os
import json
from pathlib import Path
# MQTT Configuration from environment
MQTT_HOST = os.environ.get("MQTT_HOST")
MQTT_PORT = int(os.environ.get("MQTT_PORT", "1883"))
MQTT_USER = os.environ.get("MQTT_USER")
MQTT_PASSWORD = os.environ.get("MQTT_PASSWORD")
MQTT_CLIENT_ID = os.environ.get("MQTT_CLIENT_ID", "sensorpajen")
# Validate required config
if not MQTT_HOST:
raise RuntimeError("MQTT_HOST environment variable must be set")
(relative to project root)
PROJECT_ROOT = Path(__file__).parent.parent.parent
SENSOR_CONFIG_FILE = os.environ.get(
"SENSOR_CONFIG_FILE",
str(PROJECT_ROOT / "config
str(Path.home() / ".config/sensorpajen/sensors.json")
)
# Bluetooth settings
WATCHDOG_TIMEOUT = int(os.environ.get("WATCHDOG_TIMEOUT", "5"))
ENABLE_BATTERY = os.environ.get("ENABLE_BATTERY", "true").lower() == "true"
```
3. Create `src/sensorpajen/utils.py`
- Port bluetooth_utils.py functionality
- Clean up and modernize
4. Create `src/sensorpajen/sensor_reader.py`
- Extract sensor reading logic from LYWSD03MMC.py
- Remove callback/shell script dependency
- Direct Python MQTT integration
5. Create `src/sensorpajen/mqtt_publisher.py`
- MQTT client setup and connection
- Publishing logic (replacing sendToMQTT.sh)
- Error handling and reconnection
6. Create `src/sensorpajen/main.py`
- Entry point for the application
- Signal handling (SIGTERM, SIGINT)
- Logging setup (to stdout for journald)
- Main loop
```python
#!/usr/bin/env python3
import logging
import signal
import sys
from . import config
from .sensor_reader import SensorReader
from .mqtt_publisher import MQTTPublisher
def main():
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
stream=sys.stdout
)
logger = logging.getLogger(__name__)
logger.info("Starting sensorpajen service")
# Setup signal handlers
def signal_handler(sig, frame):
logger.info("Received shutdown signal")
sys.exit(0)
signal.signal(signal.SIGTERM, signal_handler)
signal.signal(signal.SIGINT, signal_handler)
# Main application logic here
# ...
if __name__ == "__main__":
main()
```
---
### Phase 3: Configuration Migration ✓ TODO
**Goal**: Replace .ini file with JSON and environment variables
#### Tasks:
1. Create sensor mapping converter script
- Read sensorer.ini
- Output to sensors.json
```json
{
"sensors": [
{
"mac": "A4:C1:38:98:7B:B6",
"name": "mi_temp_1"
},
{
"mac": "A4:C1:38:29:03:0D",
"name": "mi_temp_2"
}
]
}
```
configuration file templates
- `config/sensorpajen.env.example`
```bash
# MQTT Configuration
MQTT_HOST=192.168.0.114
MQTT_PORT=1883
MQTT_USER=hasse
MQTT_PASSWORD=casablanca
MQTT_CLIENT_ID=mibridge
# Sensor Configuration (relative to project root)
SENSOR_CONFIG_FILE=config/sensors.json
# Application Settings
WATCHDOG_TIMEOUT=5
ENABLE_BATTERY=true
LOG_LEVEL=INFO
```
- `config/sensors.json.example`
```json
{
"sensors": [
{
"mac": "A4:C1:38:98:7B:B6",
"name": "mi_temp_1",
"comment": "Example sensor"
}
]
}
```
3. Copy templates to actual config files (not in git):
```bash
cp config/sensorpajen.env.example config/sensorpajen.env
cp config/sensors.json.example config/sensors.json
chmod 600 config/sensorpajen.env
# Edit both files with your actual configurationnsorpajen/sensorpajen.env
chmod 600 /home/fredrik/.config/sensorpajen/sensorpajen.env
```
4. Document all configuration variables in README
---
config/sensorpajen.env
config/sensors.json
*.deb
debian/.debhelper/
debian/sensorpajen/
debian/files
debian/*.log
debian/*.substvars
### Phase 4: Virtual Environment & Dependencies ✓ TODO
**Goal**: Set up isolated Python environment
#### Tasks:
1. Create virtual environment:
```bash
python3 -m venv .venv
```
2. Update .gitignore:
```
.venv/
__pycache__/
*.pyc
.env
sensorpajen.env
```
3. Install dependencies:
```bash
source .venv/bin/activate
pip install --upgrade pip
pip install bluepy paho-mqtt
pip install -e . # Install package in development mode
```
4. Document virtual environment usage in README
---
### Phase 5: Bluetooth Permissions ✓ TODO
**Goal**: Allow non-root user to access Bluetooth
#### Tasks:
1. Add user to bluetooth group:
```bash
sudo usermod -a -G bluetooth fredrik
```
2. Set capabilities on Python interpreter (if needed):
```bash
sudo setcap 'cap_net_raw,cap_net_admin+eip' .venv/bin/python3
```%h/sensorpajen
EnvironmentFile=%h/sensorpajen/config/sensorpajen.env
ExecStart=%h/sensorpajen/.venv/bin/python -m sensorpajen.main
Restart=always
RestartSec=10
# Bluetooth capabilities
AmbientCapabilities=CAP_NET_RAW CAP_NET_ADMIN
# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=sensorpajen
[Install]
WantedBy=default.target
```
Note: `%h` expands to the user's home directorycription=Sensorpajen - Bluetooth Temperature Sensor Monitor
After=network.target bluetooth.target
Wants=bluetooth.target
[Service]
Type=simple
WorkingDirectory=/home/fredrik/dev/sensorpajen
EnvironmentFile=/home/fredrik/.config/sensorpajen/sensorpajen.env
ExecStart=/home/fredrik/dev/sensorpajen/.venv/bin/python -m sensorpajen.main
Restart=always
RestartSec=10
# Bluetooth capabilities
AmbientCapabilities=CAP_NET_RAW CAP_NET_ADMIN
# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=sensorpajen
[Install]
WantedBy=default.target
```
2. Install service (user service):
```bash
mkdir -p ~/.config/systemd/user/
cp systemd/sensorpajen.service ~/.config/systemd/user/
systemctl --user daemon-reload
```
3. Enable lingering (service runs without login):
```bash
sudo loginctl enable-linger fredrik
```
4. Document systemd commands in README
---
### Phase 7: Testing & Validation ✓ TODO
**Goal**: Verify new service works before removing legacy
#### Tasks:
1. Stop legacy cron/tmux processes:
```bash
crontab -e # Comment out sensorpajen entries
tmux kill-session -t sensorer
```
2. Start new service:
```bash
systemctl --user start sensorpajen
```
3. Monitor logs:
```bash
journalctl --user -u sensorpajen -f
```APT Package Creation ✓ TODO
**Goal**: Create Debian package for easy installation on Raspberry Pi
#### Tasks:
1. Create debian/ directory structure:
```bash
mkdir -p debian
```
2. Create `debian/control`:
``APT package installation instructions
- Development installation instructions
- Configuration guide (relative paths)
- Service management commands
- Troubleshooting section
- Remove DHT11 references
- Remove pirate_audio references
3. Create INSTALL.md:
- APT package installation steps
- Manual installation steps
- Configuration examples
- First-time setup guide
- Raspberry Pi specific instructionsds}, ${misc:Depends},
python3-bluepy,
python3-paho-mqtt,
bluetooth,
bluez
Description: Bluetooth temperature sensor monitor
Monitors Xiaomi Mijia LYWSD03MMC Bluetooth temperature sensors
and publishes data to MQTT broker.
```
3. Create `debian/rules`:
```makefile
#!/usr/bin/make -f
%:
dh $@ --with python3 --buildsystem=pybuild
override_dh_auto_install:
pytOption 1: APT Package (Recommended for Raspberry Pi)
1. Download and install the .deb package:
```bash
sudo dpkg -i sensorpajen_1.0.0_all.deb
sudo apt-get install -f # Fix any dependencies
```
2. Configure:
```bash
mkdir -p ~/sensorpajen/config
cp /usr/share/doc/sensorpajen/examples/sensorpajen.env.example ~/sensorpajen/config/sensorpajen.env
cp /usr/share/doc/sensorpajen/examples/sensors.json.example ~/sensorpajen/config/sensors.json
# Edit both files
nano ~/sensorpajen/config/sensorpajen.env
nano ~/sensorpajen/config/sensors.json
chmod 600 ~/sensorpajen/config/sensorpajen.env
```
3. Enable and start service:
```bash
systemctl --user enable sensorpajen
systemctl --user start sensorpajen
```
### Option 2: Development Installation
1. Clone Repository
```bash
git clone <repo> ~/sensorpajen
cd ~/sensorpajen
```
2. Create Virtual Environment
```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
```
### Relative Paths (For Portability)
- **Project root**: `~/sensorpajen/` (or wherever you clone/install)
- **Application config**: `~/sensorpajen/config/`
- **Environment file**: `~/sensorpajen/config/sensorpajen.env` (0600)
- **Sensor mapping**: `~/sensorpajen/config/sensors.json` (0644)
- **Service file**: `~/.config/systemd/user/sensorpajen.service`
### Advantages of Relative Paths
- Works on any system (development, production, multiple Raspberry Pis)
- Easy to backup/restore entire directory
- No hardcoded paths in code
- Simple to deploy via git pull or package installation
- User service runs without sudo
### APT Package Installation
When installed via .deb package:
- **Python package**: `/usr/lib/python3/dist-packages/sensorpajen/`
- **Service file**: `/lib/systemd/user/sensorpajen.service`
- **Config templates**: `/usr/share/doc/sensorpajen/examples/`
- **User config**: `~/sensorpajen/config/` (created by user)sensorpajen
```
5. Verify
```bash
systemctl --user status sensorpajen
journalctl --user -u sensorpajen -f
```uetooth access
if [ "$1" = "configure" ]; then
PYTHON_PATH=$(readlink -f /usr/bin/python3)
setcap 'cap_net_raw,cap_net_admin+eip' "$PYTHON_PATH" || true
fi
#DEBHELPER#
```
7. Create `debian/README.Debian`:
- Installation instructions
- Configuration guide
- Service management
8. Build the package:
```bash
dpkg-buildpackage -us -uc -b
```
9. Test installation on Raspberry Pi:
```bash
sudo dpkg -i ../sensorpajen_1.0.0_all.deb
sudo apt-get install -f # Fix dependencies if needed
```
10. Create installation documentation:
- Package installation instructions
- Configuration setup after installation
- Service enablement
---
### Phase 9:
4. Verify MQTT messages:
```bash
mosquitto_sub -h 192.168.0.114 -u hasse -P casablanca -t "MiTemperature2/#" -v
```
5. Test service restart:
```bash
systemctl --user restart sensorpajen
```
6. Test crash recovery (kill process, verify auto-restart)
7. Test boot behavior:
```bash
systemctl --user enable sensorpajen
sudo reboot
# After reboot, verify service is running
systemctl --user status sensorpajen
```
---
### Phase 8: Cleanup & Document
- [ ] Publish APT package to personal repository
- [ ] Create automated build pipeline for .deb packages
- [ ] Add support for multiple MQTT brokers
- [ ] Implement configuration validation toolation ✓ TODO
**Goal**: Remove legacy code and finalize documentation
#### Tasks:
1. Once new service is stable (run for 1-2 weeks):
- Delete legacy/ folder
- Remove cron jobs completely
- Remove tmux session references
2. Update README.md:
- Installation instructions
- Configuration guide
- Service management commands
- Troubleshooting section
- Remove DHT11 references
- Remove pirate_audio references
3. Create INSTALL.md:
- Fresh installation steps
- Configuration examples
- First-time setup guide
4. Document in README:
```markdown
## Installation
### 1. Clone Repository
git clone <repo> /home/fredrik/dev/sensorpajen
cd /home/fredrik/dev/sensorpajen
### 2. Create Virtual Environment
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
### 3. Configure
mkdir -p ~/.config/sensorpajen
cp systemd/sensorpajen.env.example ~/.config/sensorpajen/sensorpajen.env
# Edit configuration
nano ~/.config/sensorpajen/sensorpajen.env
chmod 600 ~/.config/sensorpajen/sensorpajen.env
### 4. Convert Sensor Configuration
# Create sensors.json from your sensor list
### 5. Install Service
cp systemd/sensorpajen.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable sensorpajen
systemctl --user start sensorpajen
### 6. Verify
systemctl --user status sensorpajen
journalctl --user -u sensorpajen -f
```
5. Add troubleshooting section:
- Bluetooth permission issues
- MQTT connection problems
- Service won't start
- Log locations
---
## Configuration File Locations (Linux Best Practices)
### User Service Configuration
- **Service files**: `~/.config/systemd/user/`
- **Application config**: `~/.config/sensorpajen/`
- **Environment file**: `~/.config/sensorpajen/sensorpajen.env` (0600)
- **Sensor mapping**: `~/.config/sensorpajen/sensors.json` (0644)
### System Service (Alternative - Not Recommended)
If running as system service (not user service):
- **Service file**: `/etc/systemd/system/sensorpajen.service`
- **Config directory**: `/etc/sensorpajen/`
- **Environment file**: `/etc/sensorpajen/sensorpajen.env` (0600)
**Recommendation**: Use user service (current approach) since:
- No sudo required for service management
- Easier permission management
- Better security isolation
- Simpler Bluetooth access
---
## Success Criteria
The migration is complete when:
- ✅ Service starts automatically on boot
- ✅ All 8 Bluetooth sensors are being read
- ✅ MQTT messages are published correctly
- ✅ Service recovers automatically from crashes
- ✅ No hardcoded credentials in code
- ✅ Logs are visible via journalctl
- ✅ DHT11 functionality completely removed
- ✅ Legacy scripts removed
- ✅ Documentation is complete and accurate
- ✅ Service runs as user (not root)
- ✅ Virtual environment is working
---
## Rollback Plan
If issues arise during migration:
1. Stop new service:
```bash
systemctl --user stop sensorpajen
systemctl --user disable sensorpajen
```
2. Restore legacy scripts from legacy/ folder:
```bash
cp legacy/* .
```
3. Restore cron jobs:
```bash
crontab -e
# Uncomment:
# @reboot /home/fredrik/dev/sensorpajen/sensorer.sh
```
4. Reboot or manually start tmux session
---
## Future Enhancements
After successful migration, consider:
- [ ] Add Prometheus metrics endpoint
- [ ] Add systemd watchdog support
- [ ] Implement graceful sensor failure handling
- [ ] Add MQTT TLS support
- [ ] Create web dashboard for sensor status
- [ ] Add sensor calibration configuration
- [ ] Implement sensor auto-discovery
- [ ] Add health check endpoint
---
## Notes
- Keep legacy scripts during migration for safety
- Test thoroughly before removing cron jobs
- Monitor for at least 1-2 weeks before final cleanup
- Document any issues encountered during migration
- Take notes of actual MAC addresses and sensor names during conversion
---
## References
- systemd user services: `man systemd.service`
- XDG Base Directory: `~/.config/` for user configuration
- Bluetooth capabilities: `man capabilities`
- journalctl: `man journalctl`
- Python logging: https://docs.python.org/3/library/logging.html