Production hardening: runtime, registry, packaging and CI #14

Merged
Otto merged 3 commits from otto/production-hardening-20260611 into main 2026-06-11 21:15:27 +02:00
31 changed files with 446 additions and 154 deletions

View File

@@ -12,6 +12,5 @@ node_modules
.vscode
.git
.gitignore
README.md
.dockerignore
docker-compose*.yml

View File

@@ -1,2 +1,3 @@
SILLYHOME_HA_URL=http://homeassistant.local:8123
SILLYHOME_HA_TOKEN=REPLACE_ME_WITH_LONG_LIVED_TOKEN
SILLYHOME_MODEL_STORE=.model_store

View File

@@ -0,0 +1,24 @@
name: quality
on:
push:
branches: ["main", "otto/**", "feature/**"]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.11", "3.13"]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: pip
- run: python -m pip install --upgrade pip
- run: python -m pip install -e ".[dev]"
- run: python -m pytest
- run: ruff check .
- run: mypy

View File

@@ -3,3 +3,8 @@
## Unreleased
- Projektinitiierung
- Architektur, ADRs und Roadmap
- Einheitliche produktive FastAPI-App für HA- und ML-Routen
- Funktionierende ENV-Konfiguration und sauberer HA-503-Zustand
- Persistente, validierte und gegen Path Traversal gehärtete Model Registry
- Reproduzierbares Packaging, CI-Gates und gehärteter non-root Container
- Definierte API-Fehler und korrigierte Evaluationsmetriken

View File

@@ -1,15 +1,27 @@
FROM python:3.13-slim
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1 \
SILLYHOME_MODEL_STORE=/app/data/models
WORKDIR /app
COPY pyproject.toml ./
RUN python -m pip install --upgrade pip && \
pip install --no-cache-dir -e ".[dev]"
RUN addgroup --system sillyhome && adduser --system --ingroup sillyhome sillyhome
COPY . .
COPY pyproject.toml README.md ./
COPY app ./app
COPY backend ./backend
RUN python -m pip install --upgrade pip && \
python -m pip install . && \
mkdir -p /app/data/models && \
chown -R sillyhome:sillyhome /app/data
EXPOSE 8000
USER sillyhome
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
CMD ["python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=2)"]
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

View File

@@ -1,6 +1,13 @@
# SillyHome Next
Modern, lokal-first und datenschutzfreundliches Smart-Home-Intelligenzsystem für Home Assistant.
Lokaler, datenschutzfreundlicher API-Prototyp für Home Assistant.
## Reifegrad
Version `0.1.0` stellt eine gehärtete technische Basis bereit: Home-Assistant-Entities
lesen, regelbasierte Bausteine und eine persistente Modell-Artefakt-Registry. Die
aktuelle Trainings- und Vorhersagelogik ist noch eine deterministische
Schnittstellen-Implementierung und **kein produktives Machine-Learning-Modell**.
## Motivation
TheSillyHome zeigte die Idee: statt statischer Regeln das Zuhause aus Verhaltensmustern verstehen. Diese Architektur modernisiert den Ansatz in Richtung Explainable AI, hybride Intelligenzebenen und langlebige Wartbarkeit.
@@ -13,14 +20,12 @@ TheSillyHome zeigte die Idee: statt statischer Regeln das Zuhause aus Verhaltens
- Lokal-first ohne Cloudpflicht
- Erweiterbar, testbar, dokumentiert
## APPENDIX
### Quickstart
## Quickstart
1. Python-Venv anlegen und Abhängigkeiten installieren:
```bash
python -m venv .venv
source .venv/bin/activate
pip install -e .
pip install -e ".[dev]"
```
2. Konfiguration aus `.env.example` übernehmen und anpassen:
@@ -37,17 +42,32 @@ uvicorn app.main:app --reload
- `http://127.0.0.1:8000/health` - Health-Check
- `http://127.0.0.1:8000/docs/` - OpenAPI-Dokumentation
- `http://127.0.0.1:8000/v1/entities` - Home-Assistant-Entities
- `http://127.0.0.1:8000/ml/health` - ML-Serving Health (ab ML-005)
- `http://127.0.0.1:8000/ml/health` - Registry-/Serving-Health
Ohne vollständige HA-Konfiguration liefert `/v1/entities` bewusst `503`.
### Docker Compose
```bash
cp .env.example .env
docker compose up --build -d
curl --fail http://127.0.0.1:8000/health
```
Compose veröffentlicht die API standardmäßig nur auf `127.0.0.1`. Für Zugriff aus
dem Netz muss ein authentifizierender Reverse Proxy vorgeschaltet werden.
### ENV-Konfiguration (`.env.example`)
- `SILLYHOME_HA_URL` Basis-URL deiner Home-Assistant-Instanz (z. B. `http://homeassistant.local:8123`)
- `SILLYHOME_HA_TOKEN` Long-Lived Access Token aus Home Assistant (nur lesen)
- `SILLYHOME_HA_TOKEN` Long-Lived Access Token eines dedizierten HA-Benutzers mit minimalen Rechten
- `SILLYHOME_MODEL_STORE` Verzeichnis für persistierte Modell-Metadaten
Hinweis: Nutze ausschließlich Long-Lived Access Tokens mit Leserechten. Niemals Administrator-Tokens oder Passwörter eintragen. `.env` gehört nicht in Versionskontrollsysteme.
Niemals Administrator-Tokens oder Passwörter eintragen. `.env` gehört nicht ins
Versionskontrollsystem.
### Tests
```bash
pytest -q
pytest
ruff check .
mypy app tests
mypy
```

View File

@@ -2,38 +2,20 @@ from __future__ import annotations
from typing import List
from fastapi import APIRouter, HTTPException, Request
from fastapi import APIRouter, Depends
from app.dependencies import get_ha_reader
from app.ha.models import HaEntitySummary
from app.ha.reader import HaReader
from app.rules.recommender import Recommender
router = APIRouter(prefix="/v1", tags=["entities"])
def _state_ha_reader(request: Request) -> HaReader:
try:
return request.app.state.ha_reader
except AttributeError as exc:
raise HTTPException(status_code=503, detail="HA-Reader nicht initialisiert.") from exc
def _state_recommender(request: Request) -> Recommender:
try:
return request.app.state.recommender
except AttributeError as exc:
raise HTTPException(status_code=503, detail="Recommender nicht initialisiert.") from exc
@router.get(
"/entities",
summary="Home-Assistant-Entities auflisten",
description="Gibt eine kompakte Zusammenfassung aller erreichbaren HA-Entitäten zurück.",
response_model=List[HaEntitySummary],
)
def list_entities(request: Request) -> List[HaEntitySummary]:
ha_reader = _state_ha_reader(request)
recommender = _state_recommender(request)
entities = ha_reader.read_entities()
recommender.run(entities)
return entities
def list_entities(ha_reader: HaReader = Depends(get_ha_reader)) -> List[HaEntitySummary]:
return list(ha_reader.read_entities())

View File

@@ -8,6 +8,7 @@ from dataclasses import dataclass
class Settings:
ha_url: str | None = None
ha_token: str | None = None
model_store: str = ".model_store"
@property
def ha_configured(self) -> bool:
@@ -18,4 +19,5 @@ def load_settings() -> Settings:
return Settings(
ha_url=os.getenv("SILLYHOME_HA_URL") or os.getenv("HA_URL"),
ha_token=os.getenv("SILLYHOME_HA_TOKEN") or os.getenv("HA_TOKEN"),
model_store=os.getenv("SILLYHOME_MODEL_STORE", ".model_store"),
)

View File

@@ -31,10 +31,13 @@ class HaClient:
"Content-Type": "application/json",
})
def close(self) -> None:
self._session.close()
def list_entities(self) -> list[dict[str, object]]:
try:
response = self._session.get(
f"{self._settings.url}/api/states",
f"{self._settings.url.rstrip('/')}/api/states",
timeout=self._settings.timeout_seconds,
)
except requests.Timeout as exc:

View File

@@ -15,9 +15,10 @@ class HaReader:
entities = self._client.list_entities()
summaries: list[HaEntitySummary] = []
for item in entities:
entity_id = item.get("entity_id", "")
if "." not in entity_id:
raw_entity_id = item.get("entity_id")
if not isinstance(raw_entity_id, str) or "." not in raw_entity_id:
continue
entity_id = raw_entity_id
domain = entity_id.split(".", 1)[0]
raw_attributes = item.get("attributes") or {}
attributes: dict[str, Any] = raw_attributes if isinstance(raw_attributes, dict) else {}

View File

@@ -1,34 +1,38 @@
from contextlib import asynccontextmanager
from collections.abc import AsyncIterator
from typing import cast
from fastapi import FastAPI
from app.api.v1.entities import router as entities_router
from app.config import load_settings
from app.core.exception_handlers import register_exception_handlers
from app.ha.client import HaClient, HaClientSettings
from app.ha.reader import HaReader
from app.rules.recommender import Recommender
from app.rules.heating import HeatingRule
from app.ml.registry.model_registry import ModelRegistry
from backend.routes.ml import init_ml_routes
@asynccontextmanager
async def lifespan(app: FastAPI):
async def lifespan(app: FastAPI) -> AsyncIterator[None]:
settings = app.state.settings
ha_url = getattr(settings, "ha_url", None)
ha_token = getattr(settings, "ha_token", None)
client: HaClient | None = None
app.state.registry = ModelRegistry(settings.model_store)
if hasattr(app.state, "ha_reader"):
del app.state.ha_reader
if settings.ha_configured:
client = HaClient(
settings=HaClientSettings(
url=ha_url or "",
token=ha_token or "",
url=cast(str, settings.ha_url),
token=cast(str, settings.ha_token),
)
)
app.state.ha_reader = HaReader(client=client)
app.state.recommender = Recommender(rules=[HeatingRule()])
try:
yield
class Settings:
ha_url: str = "http://localhost:8123"
ha_token: str = ""
finally:
if client is not None:
client.close()
app = FastAPI(
@@ -37,9 +41,10 @@ app = FastAPI(
version="0.1.0",
lifespan=lifespan,
)
app.state.settings = Settings()
app.state.settings = load_settings()
register_exception_handlers(app)
app.include_router(entities_router)
init_ml_routes(app, model_store=app.state.settings.model_store)
@app.get("/health")

View File

@@ -1,5 +1,5 @@
"""Machine-Learning-Grundbausteine für SillyHome Next."""
__all__ = ["FeatureStore", "FeatureVector"]
__all__ = ["FeatureStore", "FeatureVector", "TrainedArtifact", "TrainingPipeline"]
from app.ml.feature_store import FeatureStore, FeatureVector
from app.ml.training import TrainedArtifact, TrainingPipeline

View File

@@ -1,11 +1,10 @@
from __future__ import annotations
import logging
from dataclasses import dataclass, field
from typing import Sequence
from collections.abc import Sequence
from dataclasses import dataclass
from app.ml.feature_store import FeatureVector
from app.ml.training import TrainingPipeline, TrainedArtifact
from app.ml.training import TrainingPipeline
logger = logging.getLogger(__name__)
@@ -29,15 +28,16 @@ class Evaluator:
self._pipeline = pipeline
def evaluate(self, artifact_id: str, predictions: Sequence[str]) -> EvalReport:
artifacts = list(self._pipeline._artifacts)
if not artifacts:
raise ValueError("Kein trainiertes Modell für Evaluation vorhanden.")
try:
supported_sensors = set(self._pipeline.export(artifact_id).supported_sensors)
except KeyError as exc:
raise ValueError("Kein trainiertes Modell für Evaluation vorhanden.") from exc
supported_sensors = self._pipeline.export(artifact_id).supported_sensors
unknown_hits = sum(1 for prediction in predictions if ":" not in prediction)
supported_references = sum(1 for sensor in supported_sensors for prediction in predictions if sensor in prediction)
parsed_sensors = [_prediction_sensor(prediction) for prediction in predictions]
supported_hits = sum(sensor in supported_sensors for sensor in parsed_sensors)
unknown_hits = sum(sensor not in supported_sensors for sensor in parsed_sensors)
sample_size = len(predictions)
coverage = supported_references / sample_size if sample_size else 0.0
coverage = supported_hits / sample_size if sample_size else 0.0
unknown_rate = unknown_hits / sample_size if sample_size else 0.0
coverage_metric = Metric(name="coverage", value=coverage, threshold=0.8)
@@ -55,3 +55,10 @@ class Evaluator:
unknown_rate,
)
return report
def _prediction_sensor(prediction: str) -> str | None:
parts = prediction.split(":", 2)
if len(parts) != 3 or not parts[0] or not parts[1]:
return None
return parts[1]

View File

@@ -1,8 +1,8 @@
from __future__ import annotations
from collections import defaultdict
from dataclasses import dataclass, field
from typing import Iterable
from collections.abc import Iterable
from dataclasses import dataclass
@dataclass(frozen=True)

View File

@@ -1,26 +1,34 @@
from __future__ import annotations
import json
import logging
import os
from pathlib import Path
from typing import Iterable
import re
from collections.abc import Iterable
from app.ml.training import TrainedArtifact
logger = logging.getLogger(__name__)
_ARTIFACT_ID_PATTERN = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$")
class ModelRegistry:
def __init__(self, root: str | Path) -> None:
self._root = Path(root)
self._root = Path(root).resolve()
self._root.mkdir(parents=True, exist_ok=True)
self._artifacts: dict[str, TrainedArtifact] = {}
self._load_existing()
def register(self, artifact: TrainedArtifact) -> TrainedArtifact:
self._artifacts[artifact.artifact_id] = artifact
self._validate_artifact_id(artifact.artifact_id)
self._persist(artifact)
self._artifacts[artifact.artifact_id] = artifact
return artifact
def load_artifact(self, artifact_id: str) -> TrainedArtifact:
self._validate_artifact_id(artifact_id)
if artifact_id not in self._artifacts:
raise KeyError(f"Artifact '{artifact_id}' nicht registriert.")
return self._artifacts[artifact_id]
@@ -28,10 +36,45 @@ class ModelRegistry:
def list_models(self) -> Iterable[TrainedArtifact]:
return list(self._artifacts.values())
def _load_existing(self) -> None:
for source in sorted(self._root.glob("*.json")):
try:
raw = json.loads(source.read_text(encoding="utf-8"))
artifact_id = raw["artifact_id"]
supported_sensors = raw["supported_sensors"]
if not isinstance(artifact_id, str) or not isinstance(supported_sensors, list):
raise ValueError("invalid artifact structure")
self._validate_artifact_id(artifact_id)
if source.name != f"{artifact_id}.json":
raise ValueError("artifact id does not match filename")
if not all(isinstance(sensor, str) for sensor in supported_sensors):
raise ValueError("supported_sensors must contain strings")
except (KeyError, TypeError, ValueError, json.JSONDecodeError) as exc:
raise ValueError(f"Ungültiges Modell-Artefakt: {source.name}") from exc
self._artifacts[artifact_id] = TrainedArtifact(
artifact_id=artifact_id,
supported_sensors=tuple(supported_sensors),
)
def _persist(self, artifact: TrainedArtifact) -> None:
target = self._root / f"{artifact.artifact_id}.json"
target.write_text(
f"{artifact.artifact_id}\t{','.join(artifact.supported_sensors)}\n",
temporary = target.with_suffix(".json.tmp")
payload = {
"artifact_id": artifact.artifact_id,
"supported_sensors": list(artifact.supported_sensors),
}
temporary.write_text(
json.dumps(payload, ensure_ascii=True, sort_keys=True) + "\n",
encoding="utf-8",
)
os.replace(temporary, target)
logger.info("Modell gespeichert: %s", target)
@staticmethod
def _validate_artifact_id(artifact_id: str) -> None:
if not _ARTIFACT_ID_PATTERN.fullmatch(artifact_id) or ".." in artifact_id:
raise ValueError(
"artifact_id darf nur Buchstaben, Ziffern, Punkt, Unterstrich "
"und Bindestrich enthalten."
)

View File

@@ -2,9 +2,8 @@ from __future__ import annotations
import logging
from dataclasses import dataclass
from typing import Sequence
from app.ml.feature_store import FeatureVector, FeatureStore
from app.ml.feature_store import FeatureStore
logger = logging.getLogger(__name__)

1
backend/__init__.py Normal file
View File

@@ -0,0 +1 @@
"""Secondary application entry points for SillyHome Next."""

View File

@@ -1,4 +1,8 @@
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
from fastapi import FastAPI
from starlette.datastructures import State
from backend.routes.ml import init_ml_routes
from app.ml.registry.model_registry import ModelRegistry
@@ -6,18 +10,20 @@ from app.ml.training import TrainingPipeline
from app.ml.feature_store import FeatureStore, FeatureVector
@asynccontextmanager
async def lifespan(application: FastAPI) -> AsyncIterator[None]:
application.state.registry = ModelRegistry(application.state.model_store)
_seed_default_model(application.state)
yield
def create_app() -> FastAPI:
application = FastAPI(title="SillyHome Next ML")
application = FastAPI(title="SillyHome Next ML", lifespan=lifespan)
init_ml_routes(application)
_seed_default_model(application.state if hasattr(application, "state") else application)
return application
def _app_state(): # noqa: ANN001
return app.state
def _seed_default_model(state) -> None: # noqa: ANN001
def _seed_default_model(state: State) -> None:
registry = getattr(state, "registry", None)
if registry is None:
registry = ModelRegistry(".model_store")

View File

@@ -0,0 +1 @@
"""API route modules."""

View File

@@ -2,15 +2,14 @@ from __future__ import annotations
import logging
from datetime import datetime, timezone
from typing import List, Sequence
from collections.abc import Sequence
from fastapi import APIRouter
from fastapi import APIRouter, FastAPI, HTTPException, Request, status
from pydantic import BaseModel, Field
from app.ml.feature_store import FeatureVector
from app.ml.predictor import Predictor
from app.ml.registry.model_registry import ModelRegistry
from app.ml.training import TrainedArtifact
logger = logging.getLogger(__name__)
@@ -25,7 +24,7 @@ class HealthResponse(BaseModel):
class PredictRequest(BaseModel):
model_id: str = Field(..., alias="modelId")
sensor_id: str
values: dict
values: dict[str, float]
class PredictResponse(BaseModel):
@@ -43,7 +42,7 @@ class BatchResponse(BaseModel):
class ModelsResponse(BaseModel):
models: List[str]
models: list[str]
@router.get("/health", response_model=HealthResponse, status_code=200)
@@ -52,53 +51,66 @@ def health() -> HealthResponse:
@router.get("/models", response_model=ModelsResponse, status_code=200)
def list_models() -> ModelsResponse:
registry = _require_registry()
def list_models(request: Request) -> ModelsResponse:
registry = _require_registry(request)
models = [artifact.artifact_id for artifact in registry.list_models()]
return ModelsResponse(models=models)
@router.post("/predict", response_model=PredictResponse, status_code=200)
def predict(request: PredictRequest) -> PredictResponse:
registry = _require_registry()
def predict(payload: PredictRequest, request: Request) -> PredictResponse:
registry = _require_registry(request)
predictor = Predictor(registry=registry)
vector = FeatureVector(sensor_id=request.sensor_id, values=request.values)
vector = FeatureVector(sensor_id=payload.sensor_id, values=payload.values)
try:
prediction = predictor.predict(request.model_id, vector)
except KeyError as exc: # unknown artifact
raise _not_found_error(str(exc)) from exc
return PredictResponse(model_id=request.model_id, sensor_id=request.sensor_id, prediction=prediction)
prediction = predictor.predict(payload.model_id, vector)
except KeyError as exc:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=str(exc)) from exc
except ValueError as exc:
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
detail=str(exc),
) from exc
return PredictResponse(
model_id=payload.model_id,
sensor_id=payload.sensor_id,
prediction=prediction,
)
@router.post("/batch", response_model=BatchResponse, status_code=200)
def predict_batch(request: BatchRequest) -> BatchResponse:
registry = _require_registry()
def predict_batch(payload: BatchRequest, request: Request) -> BatchResponse:
registry = _require_registry(request)
predictor = Predictor(registry=registry)
responses: List[PredictResponse] = []
for item in request.requests:
responses: list[PredictResponse] = []
for item in payload.requests:
vector = FeatureVector(sensor_id=item.sensor_id, values=item.values)
try:
prediction = predictor.predict(item.model_id, vector)
except KeyError as exc:
raise _not_found_error(str(exc)) from exc
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=str(exc)) from exc
except ValueError as exc:
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
detail=str(exc),
) from exc
responses.append(
PredictResponse(model_id=item.model_id, sensor_id=item.sensor_id, prediction=prediction)
)
return BatchResponse(predictions=responses)
def _require_registry() -> ModelRegistry:
from backend.app import _app_state
state_obj = _app_state()
registry = getattr(state_obj, "registry", None)
if registry is None:
raise RuntimeError("ML registry nicht initialisiert.")
def _require_registry(request: Request) -> ModelRegistry:
registry = getattr(request.app.state, "registry", None)
if not isinstance(registry, ModelRegistry):
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail="ML registry nicht initialisiert.",
)
return registry
def init_ml_routes(app) -> None: # noqa: ANN001
registry = ModelRegistry(".model_store")
app.state.registry = registry
def init_ml_routes(app: FastAPI, model_store: str = ".model_store") -> None:
app.state.model_store = model_store
app.include_router(router)
logger.info("ML routes registered")

View File

@@ -2,7 +2,22 @@ services:
api:
build: .
ports:
- "8000:8000"
- "127.0.0.1:8000:8000"
env_file:
- .env
- path: .env
required: false
environment:
SILLYHOME_MODEL_STORE: /app/data/models
volumes:
- model-data:/app/data/models
read_only: true
tmpfs:
- /tmp
security_opt:
- no-new-privileges:true
cap_drop:
- ALL
restart: unless-stopped
volumes:
model-data:

View File

@@ -1,6 +1,11 @@
# ML-Serving-API
Diese Dokumentation beschreibt die REST-Endpoints für ML-Vorhersagen in SillyHome Next.
Diese Dokumentation beschreibt die REST-Endpunkte der aktuellen
Modell-Artefakt- und Vorhersage-Schnittstelle.
> Hinweis: Version 0.1.0 enthält noch kein statistisch trainiertes ML-Modell.
> Die Vorhersage ist eine deterministische Referenzimplementierung für den
> späteren Modellvertrag.
## Basis-URL
@@ -10,7 +15,8 @@ Diese Dokumentation beschreibt die REST-Endpoints für ML-Vorhersagen in SillyHo
- Einzelvorhersage: `/predict`
- Batchvorhersage: `/batch`
Der Standard-Start erfolgt über `uvicorn backend.app:app --reload`, danach steht die API unter `/ml` bereit.
Der Standard-Start erfolgt über `uvicorn app.main:app`, danach stehen HA- und
ML-Routen in derselben Anwendung bereit.
## Endpoints
@@ -101,13 +107,15 @@ Batch-Vorhersage für mehrere Sensorwerte.
## Fehlerfälle
- `400 Bad Request`: Fehlende oder ungültige Felder.
- `404 Not Found`: Modell oder Sensor nicht registriert.
- `500 Internal Server Error`: Registry nicht initialisiert oder unerwarteter Fehler.
- `404 Not Found`: Modell nicht registriert.
- `422 Unprocessable Content`: Sensor wird vom Modell nicht unterstützt oder Eingabe ist ungültig.
- `503 Service Unavailable`: Registry ist nicht initialisiert.
## Betrieb
Beim Start wird automatisch ein Default-Artefakt erstellt, falls noch kein Modell registriert ist. Neue Modelle müssen zusätzlich über `ModelRegistry.register(...)` eingetragen werden.
Die produktive App lädt Artefakte aus `SILLYHOME_MODEL_STORE`. Neue Artefakte
werden derzeit intern über `ModelRegistry.register(...)` registriert. Die
Registry speichert validiertes JSON atomisch und lädt es beim Neustart.
## Verweise

View File

@@ -1,12 +1,14 @@
# ML Training- und Evaluations-Workflow
Dieser Workflow beschreibt, wie Modelle trainiert, evaluiert und an der Serving-Layer registriert werden.
Dieser Workflow beschreibt den aktuellen Platzhalter für Modell-Metadaten,
Evaluation und Serving. Er trainiert in Version 0.1.0 noch kein statistisches
Modell.
## 1. Daten sammeln
Alle Trainingsvektoren werden über `FeatureStore.add(...)` oder `add_batch(...)` eingepflegt. Jeder Vektor enthält eine Sensor-ID sowie ein Dictionary mit Merkmalen.
## 2. Modell trainieren
## 2. Artefakt-Metadaten erzeugen
```python
store = FeatureStore()
@@ -16,7 +18,9 @@ artifact = pipeline.run("my_artifact")
pipeline.export("my_artifact")
```
`TrainingPipeline.run(...)` erzeugt ein `TrainedArtifact` mit den unterstützten Sensor-IDs.
`TrainingPipeline.run(...)` erzeugt ein `TrainedArtifact` mit den unterstützten
Sensor-IDs. Gewichte, Parameter oder ein echtes Modell werden noch nicht
berechnet.
## 3. Modell evaluieren
@@ -37,3 +41,4 @@ Das trainierte Artefakt kann anschließend über `ModelRegistry.register(artifac
## Hinweise
- Für reproduzierbare Sensor-Reihenfolgen wird in `TrainingPipeline.run(...)` eine sortierte Sensor-Liste verwendet.
- Fehlende Trainingsdaten lösen `ValueError` aus; nicht registrierte Artefakte lösen `KeyError` aus.
- `coverage` zählt nur exakte Sensor-Referenzen und bleibt im Bereich 0 bis 1.

View File

@@ -1,3 +1,7 @@
[build-system]
requires = ["setuptools>=69"]
build-backend = "setuptools.build_meta"
[project]
name = "sillyhome-next"
version = "0.1.0"
@@ -24,6 +28,10 @@ addopts = "-q"
[tool.mypy]
strict = true
files = ["app", "backend", "tests"]
[tool.setuptools.packages.find]
include = ["app*", "backend*"]
[tool.ruff]
line-length = 100

View File

@@ -49,8 +49,6 @@ def test_entities_returns_reader_data() -> None:
def test_entities_returns_503_without_home_assistant_config() -> None:
with TestClient(app) as client:
if hasattr(app.state, "ha_reader"):
delattr(app.state, "ha_reader")
response = client.get("/v1/entities")
assert response.status_code == 503

View File

@@ -0,0 +1,52 @@
from __future__ import annotations
from pathlib import Path
from fastapi.testclient import TestClient
from app.main import app
def test_ml_routes_are_exposed_by_production_app() -> None:
with TestClient(app) as client:
health = client.get("/ml/health")
models = client.get("/ml/models")
assert health.status_code == 200
assert models.status_code == 200
assert isinstance(models.json()["models"], list)
def test_unknown_model_returns_404() -> None:
with TestClient(app) as client:
response = client.post(
"/ml/predict",
json={
"modelId": "missing",
"sensor_id": "sensor.kitchen",
"values": {"temperature": 21.0},
},
)
assert response.status_code == 404
def test_unsupported_sensor_returns_422(tmp_path: Path) -> None:
from app.ml.registry.model_registry import ModelRegistry
from app.ml.training import TrainedArtifact
registry = ModelRegistry(tmp_path)
registry.register(TrainedArtifact("default", ("sensor.kitchen",)))
with TestClient(app) as client:
app.state.registry = registry
response = client.post(
"/ml/predict",
json={
"modelId": "default",
"sensor_id": "sensor.unknown",
"values": {"temperature": 21.0},
},
)
assert response.status_code == 422

View File

@@ -21,13 +21,35 @@ def evaluator_factory() -> Evaluator:
def test_evaluate_returns_report_with_metrics() -> None:
evaluator = evaluator_factory()
report = evaluator.evaluate("artifact_v1", ["artifact_v1:sensor.kitchen:{'temperature': 21.0}", "artifact_v1:sensor.bedroom:{'temperature': 18.5}"])
report = evaluator.evaluate(
"artifact_v1",
[
"artifact_v1:sensor.kitchen:{'temperature': 21.0}",
"artifact_v1:sensor.bedroom:{'temperature': 18.5}",
],
)
assert report.artifact_id == "artifact_v1"
assert report.sample_size == 2
assert {metric.name for metric in report.metrics} == {"coverage", "unknown_rate"}
assert next(metric.value for metric in report.metrics if metric.name == "coverage") == 1.0
def test_evaluate_without_training_raises_value_error() -> None:
evaluator = Evaluator(TrainingPipeline(FeatureStore()))
with pytest.raises(ValueError):
evaluator.evaluate("artifact_v1", [])
def test_coverage_is_bounded_and_requires_exact_sensor_match() -> None:
evaluator = evaluator_factory()
report = evaluator.evaluate(
"artifact_v1",
[
"artifact_v1:sensor.kitchen:{'note': 'sensor.bedroom'}",
"artifact_v1:sensor.kitchen_extra:{}",
"malformed",
],
)
metrics = {metric.name: metric.value for metric in report.metrics}
assert metrics == {"coverage": pytest.approx(1 / 3), "unknown_rate": pytest.approx(2 / 3)}

View File

@@ -1,7 +1,5 @@
from __future__ import annotations
import pytest
from app.ml.feature_store import FeatureStore, FeatureVector
@@ -31,12 +29,18 @@ def test_add_batch_appends_all_vectors() -> None:
]
store.add_batch(vectors)
assert len(store.all()) == 3
assert store.latest("sensor.kitchen").values["temperature"] == 20.0
latest = store.latest("sensor.kitchen")
assert latest is not None
assert latest.values["temperature"] == 20.0
def test_different_sensors_are_stored_independently() -> None:
store = FeatureStore()
store.add(_vector("sensor.living_room", 21.0))
store.add(_vector("sensor.bedroom", 18.5))
assert store.latest("sensor.living_room").values["temperature"] == 21.0
assert store.latest("sensor.bedroom").values["temperature"] == 18.5
living_room = store.latest("sensor.living_room")
bedroom = store.latest("sensor.bedroom")
assert living_room is not None
assert bedroom is not None
assert living_room.values["temperature"] == 21.0
assert bedroom.values["temperature"] == 18.5

View File

@@ -0,0 +1,39 @@
from __future__ import annotations
import json
from pathlib import Path
import pytest
from app.ml.registry.model_registry import ModelRegistry
from app.ml.training import TrainedArtifact
def test_registry_loads_persisted_artifacts_after_restart(tmp_path: Path) -> None:
registry = ModelRegistry(tmp_path)
artifact = TrainedArtifact("model-v1", ("sensor.kitchen", "sensor.bedroom"))
registry.register(artifact)
restarted = ModelRegistry(tmp_path)
assert restarted.load_artifact("model-v1") == artifact
@pytest.mark.parametrize("artifact_id", ["../escape", "nested/model", "..", ""])
def test_registry_rejects_unsafe_artifact_ids(tmp_path: Path, artifact_id: str) -> None:
registry = ModelRegistry(tmp_path)
with pytest.raises(ValueError):
registry.register(TrainedArtifact(artifact_id, ("sensor.kitchen",)))
assert list(tmp_path.parent.glob("escape.json")) == []
def test_registry_rejects_corrupt_persisted_artifact(tmp_path: Path) -> None:
(tmp_path / "broken.json").write_text(
json.dumps({"artifact_id": "../broken", "supported_sensors": []}),
encoding="utf-8",
)
with pytest.raises(ValueError, match="broken.json"):
ModelRegistry(tmp_path)

View File

@@ -3,7 +3,7 @@ from __future__ import annotations
import pytest
from app.ml.feature_store import FeatureStore, FeatureVector
from app.ml.training import TrainingPipeline, TrainedArtifact
from app.ml.training import TrainingPipeline
def _vector(sensor_id: str, temperature: float, label: str | None = None) -> FeatureVector:

18
tests/test_config.py Normal file
View File

@@ -0,0 +1,18 @@
from __future__ import annotations
from pytest import MonkeyPatch
from app.config import load_settings
def test_load_settings_reads_documented_environment(monkeypatch: MonkeyPatch) -> None:
monkeypatch.setenv("SILLYHOME_HA_URL", "http://ha.local:8123")
monkeypatch.setenv("SILLYHOME_HA_TOKEN", "secret")
monkeypatch.setenv("SILLYHOME_MODEL_STORE", "/tmp/models")
settings = load_settings()
assert settings.ha_url == "http://ha.local:8123"
assert settings.ha_token == "secret"
assert settings.model_store == "/tmp/models"
assert settings.ha_configured