fredrik/sensorpajen
1
0
Fork 0
Code Issues Pull Requests Projects Releases 2 Wiki Activity

feat: implement Textual TUI and SQLite database for sensor management

Browse Source
This commit is contained in:
fredrik
2025-12-29 09:39:33 +01:00
parent 4213b6101a
commit cfa24d1fa5
22 changed files with 1734 additions and 723 deletions
Download Patch File Download Diff File Expand all files Collapse all files

577
ROADMAP.md
View File

@@ -1,564 +1,43 @@
# ROADMAP: Modernizing Sensorpajen
# ROADMAP: Sensorpajen Modernization & TUI
## Overview
This roadmap outlines the migration from the current tmux/cron-based system to a modern systemd service running on Raspberry Pi.
**Migration Date**: Started December 27, 2025
**Target Completion**: TBD
This roadmap defines the evolution of Sensorpajen from a CLI-based tool to a full-featured TUI application for sensor management and monitoring.
---
## Current State
### What We Have
- LYWSD03MMC.py: Main Bluetooth sensor reader
- temperatur_koksfonstret.py: DHT11 sensor reader (to be removed)
- bluetooth_utils.py: Bluetooth utility functions
- sensorer.ini: MAC address to sensor name mapping
- sendToMQTT.sh: MQTT publishing callback (hardcoded credentials)
- startup.sh/sensorer.sh: tmux-based startup scripts
- Cron jobs for scheduling
### Known Issues
- MQTT credentials hardcoded in shell scripts
- Legacy pirate_audio references in startup.sh
- Manual tmux orchestration
- Mixed configuration sources
- DHT11 functionality to be removed
---
## Target Architecture
### Final Structure
```
sensorpajen/
├── src/
│ └── sensorpajen/
│ ├── __init__.py
│ ├── main.py # Entry point
│ ├── config.py # Configuration management
│ ├── sensor_reader.py # Bluetooth sensor logic
│ ├── mqtt_publisher.py # MQTT publishing
│ └── utils.py # Utilities (from bluetooth_utils.py)
├── config/ # Configuration directory (relative)
│ ├── sensors.json.example # Sensor mapping template
│ ├── sensorpajen.env.example # Environment file template
│ ├── sensors.json # Actual sensor mapping (not in git)
│ └── sensorpajen.env # Actual environment file (not in git)
├── debian/ # APT package files
│ ├── control
│ ├── rules
│ ├── changelog
│ └── ... # Other Debian package files
├── pyproject.toml # Project metadata and dependencies
├── requirements.txt # Dependencies (bluepy, paho-mqtt)
├── README.md # Updated documentation
├── AGENTS.md # Agent guidelines
├── ROADMAP.md # This file
├── legacy/ # Legacy scripts (moved here temporarily)
│ ├── LYWSD03MMC.py
│ ├── temperatur_koksfonstret.py
│ ├── sendToMQTT.sh
│ ├── startup.sh
│ ├── sensorer.sh
│ └── sensorer.ini
└── systemd/
├── sensorpajen.service # Systemd service unit
└── README.md # Systemd installation instructions
```
### Configuration Strategy
Using relative paths for portability across systems:
1. **Sensor Mapping**: `config/sensors.json` (relative to project root)
- Maps MAC addresses to sensor names
- JSON format for Python ease
- Not committed to git (use sensors.json.example as template)
2. **MQTT Credentials**: `config/sensorpajen.env` (relative to project root)
- Contains sensitive MQTT configuration
- Permissions: 0600 (owner read/write only)
- Not committed to git (use sensorpajen.env.example as template)
3. **Environment Variables** (via systemd EnvironmentFile):
```
MQTT_HOST=192.168.0.114
MQTT_USER=hasse
MQTT_PASSWORD=casablanca
MQTT_CLIENT_ID=mibridge
SENSOR_CONFIG_FILE=config/sensors.json
```
4. **Git Ignore**: Add to .gitignore:
```
config/sensors.json
config/sensorpajen.env
```
---
## Migration Phases
### Phase 1: Preparation & Cleanup ✅ DONE (2025-12-27)
**Goal**: Reorganize repository without breaking existing functionality
**Notes**:
- Created modern Python package structure with src/ layout
- Converted INI sensor config to JSON format (sensors.json.example)
- Environment-based configuration instead of hardcoded values
- DHT11 sensor functionality removed as planned
- Legacy scripts preserved in legacy/ folder
#### Tasks:
- ✅ Create new directory structure
- ✅ Create pyproject.toml with dependencies
- ✅ Remove DHT11 functionality
- ✅ Move legacy scripts to legacy/ folder
- ✅ Create config file templates (sensors.json.example, sensorpajen.env.example)
- ✅ Preserve requirements.txt for backward compatibility
---
### Phase 2: Python Package Structure ✅ DONE (2025-12-27)
**Goal**: Create modern Python package with proper entry point
## Phase 1: Modern TUI Management & Data Persistence ✅ DONE (2025-12-29)
**Goal**: Replace the basic CLI with a full-screen Textual TUI and improve discovery data persistence.
**Notes**:
- Used src/ layout for better packaging practices
- Direct Python MQTT integration (no shell script callbacks)
- ATC firmware BLE advertisement reading (passive scanning)
- Watchdog thread for BLE connection recovery
- Clean separation of concerns (config, MQTT, sensors, main)
- Migrated discovery data to SQLite for better metadata tracking.
- Implemented a full-screen TUI using Textual with Discovery, Configured, and Ignored views.
- Added support for interactive Approve, Ignore, Edit, and Remove actions.
#### Tasks:
- ✅ Created src/sensorpajen/__init__.py with version info
- ✅ Created src/sensorpajen/config.py
- Environment variable loading with validation
- SensorConfig class for JSON sensor mapping
- Relative path resolution (PROJECT_ROOT)
- Configuration validation and logging
- ✅ Created src/sensorpajen/utils.py
- Ported bluetooth_utils.py (MIT licensed, Colin GUYON)
- BLE scanning and advertisement parsing
- ✅ Created src/sensorpajen/mqtt_publisher.py
- MQTTPublisher class with connection management
- Direct publishing (replaces sendToMQTT.sh)
- Automatic reconnection support
- Battery data publishing (optional)
- ✅ Created src/sensorpajen/sensor_reader.py
- SensorReader class for BLE scanning
- ATC packet parsing
- Duplicate packet filtering
- Watchdog for BLE recovery
- Measurement dataclass
- ✅ Created src/sensorpajen/main.py
- Application entry point
- Signal handling (SIGTERM, SIGINT)
- Graceful shutdown
- Logging to stdout for journald
### Tasks:
- ✅ **Database Migration**: Replace `discovered_sensors.json` with a SQLite database.
- ✅ **Textual TUI Scaffolding**: Initialize a full-screen TUI using the `Textual` library.
- ✅ **Sensor Management View**: Interactive management of all sensor states.
- ✅ **Branching Strategy**: Developed in `feature/tui-management`.
---
### Phase 3: Configuration Migration ✅ DONE (2025-12-27)
**Goal**: Replace .ini file with JSON and environment variables
## Phase 2: Live Monitoring & Global Configuration
**Goal**: Add real-time visibility and full system configuration to the TUI.
**Notes**: Templates created in Phase 1, successfully tested on Raspberry Pi
#### Tasks:
1. Create sensor mapping converter script
- Read sensorer.ini
- Output to sensors.json
```json
{
"sensors": [
{
"mac": "A4:C1:38:98:7B:B6",
"name": "mi_temp_1"
},
{
"mac": "A4:C1:38:29:03:0D",
"name": "mi_temp_2"
}
]
}
```
configuration file templates
- `config/sensorpajen.env.example`
```bash
# MQTT Configuration
MQTT_HOST=192.168.0.114
MQTT_PORT=1883
MQTT_USER=hasse
MQTT_PASSWORD=casablanca
MQTT_CLIENT_ID=mibridge
# Sensor Configuration (relative to project root)
SENSOR_CONFIG_FILE=config/sensors.json
# Application Settings
WATCHDOG_TIMEOUT=5
ENABLE_BATTERY=true
LOG_LEVEL=INFO
```
- `config/sensors.json.example`
```json
{
"sensors": [
{
"mac": "A4:C1:38:98:7B:B6",
"name": "mi_temp_1",
"comment": "Example sensor"
}
]
}
```
3. Copy templates to actual config files (not in git):
```bash
cp config/sensorpajen.env.example config/sensorpajen.env
cp config/sensors.json.example config/sensors.json
chmod 600 config/sensorpajen.env
# Edit both files with your actual configurationnsorpajen/sensorpajen.env
chmod 600 /home/fredrik/.config/sensorpajen/sensorpajen.env
```
4. Document all configuration variables in README
---
config/sensorpajen.env
config/sensors.json
*.deb
debian/.debhelper/
debian/sensorpajen/
debian/files
debian/*.log
debian/*.substvars
### Phase 4: Virtual Environment & Dependencies ✅ DONE (2025-12-27)
**Goal**: Set up isolated Python environment
**Notes**: Tested on Raspberry Pi, paho-mqtt v2.x compatibility fixed
#### Tasks:
1. Create virtual environment:
```bash
python3 -m venv .venv
```
2. Update .gitignore:
```
.venv/
__pycache__/
*.pyc
.env
sensorpajen.env
```
3. Install dependencies:
```bash
source .venv/bin/activate
pip install --upgrade pip
pip install bluepy paho-mqtt
pip install -e . # Install package in development mode
```
4. Document virtual environment usage in README
---✅ DONE (2025-12-27)
**Goal**: Allow non-root user to access Bluetooth
**Notes**: Tested on Raspberry Pi with setcap on actual Python binary
### Phase 5: Bluetooth Permissions ✅ DONE (2025-12-27)
**Goal**: Allow non-root user to access Bluetooth
**Notes**: Tested on Raspberry Pi with setcap on actual Python binary
#### Tasks:
- ✅ Bluetooth capabilities set with setcap
- ✅ Documented in SETUP_ON_PI.md with correct readlink -f usage
- ✅ Tested successfully on Raspberry Pi
### Tasks:
- [ ] **Live Dashboard**:
- Real-time display of temperature, humidity, and battery levels.
- Visual indicators for sensor health/connectivity.
- [ ] **Global Configuration**:
- Edit MQTT settings (Host, Port, Credentials).
- Edit application settings (Watchdog, Log Level, etc.).
- [ ] **System Integration**:
- View service logs within the TUI.
- Restart/Stop service from the TUI.
---
### Phase 6: Systemd Service Creation ✅ DONE (2025-12-27)
**Goal**: Create and configure systemd user service
**Notes**:
- User service for easier management (no sudo required)
- Service ready for installation on Raspberry Pi
- Comprehensive documentation provided
- **Important discoveries**:
- `AmbientCapabilities` does NOT work in user services (only system services)
- Must use `setcap` on the Python binary instead
- `NoNewPrivileges=true` prevents file capabilities from working - must be disabled
- Capabilities must be set on actual binary, not symlinks: `setcap ... $(readlink -f python3)`
#### Tasks:
- ✅ Created systemd/sensorpajen.service
- ✅ Created systemd/README.md with full documentation
- ✅ Service management and troubleshooting guides included
- ✅ Tested and verified working on Raspberry Pi
---
### Phase 7: Testing & Validation ✅ DONE (2025-12-27)
**Goal**: Verify new service works before removing legacy
**Notes**:
- Service tested and running successfully
- Legacy cron/tmux system stopped
- All sensors reporting correctly via systemd service
#### Tasks:
- ✅ Stopped legacy cron/tmux processes
- ✅ Started new systemd service
- ✅ Monitored logs - no errors
- ✅ Verified all 8 sensors reporting
- ✅ Confirmed MQTT publishing working
- ✅ Tested service restart and auto-recovery
---
### Phase 8: APT Package Creation ✅ DONE (2025-12-27)
**Goal**: Create Debian package for easy installation on Raspberry Pi
**Notes**:
- Complete debian/ directory structure created
- System-wide installation to /opt/sensorpajen
- Configuration in /etc/sensorpajen
- Dedicated sensorpajen system user
- Automatic venv creation in postinst
- Bluetooth capabilities set automatically
- Config preserved on remove/purge for safety
- Dual-mode support: system installation and development
- config.py auto-detects installation type
#### Files Created:
- ✅ debian/control - Package metadata and dependencies
- ✅ debian/compat - Debhelper compatibility level
- ✅ debian/changelog - Package version history
- ✅ debian/rules - Build instructions
- ✅ debian/install - File installation mappings
- ✅ debian/postinst - Post-installation script (user, venv, setcap)
- ✅ debian/prerm - Pre-removal script (stop service)
- ✅ debian/postrm - Post-removal script (cleanup)
- ✅ debian/sensorpajen.service - System-wide systemd unit
#### Code Updates:
- ✅ Updated src/sensorpajen/config.py to detect system installation
- Checks for /opt/sensorpajen existence
- Uses /etc/sensorpajen for config in system mode
- Falls back to PROJECT_ROOT/config for development
- ✅ Updated scripts/approve-sensors.sh for dual-mode operation
- Detects system vs development installation
- Uses correct venv and config paths
- ✅ Created scripts/verify-deb.sh - Automated build and verification
#### Package Details:
- Package name: sensorpajen
- Version: 2.0.0-dev
- Architecture: all
- System paths:
- Application: /opt/sensorpajen/
- Configuration: /etc/sensorpajen/
- Service file: /etc/systemd/system/sensorpajen.service
- Examples: /usr/share/doc/sensorpajen/examples/
- Runs as dedicated sensorpajen user (system account)
- Auto-enables service but waits for configuration before starting
#### Build and Test:
```bash
# Build package
./scripts/verify-deb.sh
# Or manually:
dpkg-buildpackage -us -uc -b
lintian ../sensorpajen_*.deb
# Install on Raspberry Pi:
scp ../sensorpajen_*.deb pi@raspberrypi:~/
ssh pi@raspberrypi
sudo apt install ./sensorpajen_*.deb
# Configure:
sudo nano /etc/sensorpajen/sensorpajen.env
sudo nano /etc/sensorpajen/sensors.json
# Start:
sudo systemctl start sensorpajen
sudo journalctl -u sensorpajen -f
```
---
### Phase 9: Cleanup & Documentation ✅ DONE (2025-12-27)
**Goal**: Remove legacy code and finalize documentation
**Notes**:
- Legacy cron/tmux scripts removed
- Documentation focused on practical usage
- INSTALL.md created for sysadmins
#### Tasks:
- ✅ Deleted legacy/ folder (old cron/tmux scripts)
- ✅ Created INSTALL.md with concise installation guide
- ✅ Updated README.md troubleshooting section
- ✅ Documentation assumes sysadmin familiarity
---
## Migration Complete! 🎉
All phases completed. The system has been successfully migrated from a legacy cron/tmux-based system to a modern systemd service with:
- ✅ Python package structure
- ✅ Environment-based configuration (no .ini files)
- ✅ Systemd user service with auto-restart
- ✅ Automatic sensor discovery with approval workflow
- ✅ Configuration auto-reload (no restart needed)
- ✅ ntfy notifications for new sensors
- ✅ Comprehensive documentation
**Version**: 2.0.0-dev
**Status**: Production-ready
```markdown
## Installation
### 1. Clone Repository
git clone <repo> /home/fredrik/dev/sensorpajen
cd /home/fredrik/dev/sensorpajen
### 2. Create Virtual Environment
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
### 3. Configure
mkdir -p ~/.config/sensorpajen
cp systemd/sensorpajen.env.example ~/.config/sensorpajen/sensorpajen.env
# Edit configuration
nano ~/.config/sensorpajen/sensorpajen.env
chmod 600 ~/.config/sensorpajen/sensorpajen.env
### 4. Convert Sensor Configuration
# Create sensors.json from your sensor list
### 5. Install Service
cp systemd/sensorpajen.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable sensorpajen
systemctl --user start sensorpajen
### 6. Verify
systemctl --user status sensorpajen
journalctl --user -u sensorpajen -f
```
5. Add troubleshooting section:
- Bluetooth permission issues
- MQTT connection problems
- Service won't start
- Log locations
---
## Configuration File Locations (Linux Best Practices)
### User Service Configuration
- **Service files**: `~/.config/systemd/user/`
- **Application config**: `~/.config/sensorpajen/`
- **Environment file**: `~/.config/sensorpajen/sensorpajen.env` (0600)
- **Sensor mapping**: `~/.config/sensorpajen/sensors.json` (0644)
### System Service (Alternative - Not Recommended)
If running as system service (not user service):
- **Service file**: `/etc/systemd/system/sensorpajen.service`
- **Config directory**: `/etc/sensorpajen/`
- **Environment file**: `/etc/sensorpajen/sensorpajen.env` (0600)
**Recommendation**: Use user service (current approach) since:
- No sudo required for service management
- Easier permission management
- Better security isolation
- Simpler Bluetooth access
---
## Success Criteria
The migration is complete when:
- ✅ Service starts automatically on boot
- ✅ All 8 Bluetooth sensors are being read
- ✅ MQTT messages are published correctly
- ✅ Service recovers automatically from crashes
- ✅ No hardcoded credentials in code
- ✅ Logs are visible via journalctl
- ✅ DHT11 functionality completely removed
- ✅ Legacy scripts removed
- ✅ Documentation is complete and accurate
- ✅ Service runs as user (not root)
- ✅ Virtual environment is working
---
## Rollback Plan
If issues arise during migration:
1. Stop new service:
```bash
systemctl --user stop sensorpajen
systemctl --user disable sensorpajen
```
2. Restore legacy scripts from legacy/ folder:
```bash
cp legacy/* .
```
3. Restore cron jobs:
```bash
crontab -e
# Uncomment:
# @reboot /home/fredrik/dev/sensorpajen/sensorer.sh
```
4. Reboot or manually start tmux session
---
## Future Enhancements
After successful migration, consider:
- [ ] Add Prometheus metrics endpoint
- [ ] Add systemd watchdog support
- [ ] Implement graceful sensor failure handling
- [ ] Add MQTT TLS support
- [ ] Create web dashboard for sensor status
- [ ] Add sensor calibration configuration
- [ ] Implement sensor auto-discovery
- [ ] Add health check endpoint
---
## Notes
- Keep legacy scripts during migration for safety
- Test thoroughly before removing cron jobs
- Monitor for at least 1-2 weeks before final cleanup
- Document any issues encountered during migration
- Take notes of actual MAC addresses and sensor names during conversion
---
## References
- systemd user services: `man systemd.service`
- XDG Base Directory: `~/.config/` for user configuration
- Bluetooth capabilities: `man capabilities`
- journalctl: `man journalctl`
- Python logging: https://docs.python.org/3/library/logging.html
## Completed Phases
- ✅ **Phase 0: Preparation & Cleanup** (2025-12-27)
- ✅ **Phase 0.1: Testing Infrastructure** (2025-12-29)
- ✅ **Phase 1: Modern TUI Management & Data Persistence** (2025-12-29)