8 Commits

Author SHA1 Message Date
83b85cb760 Add FastAPI dashboard MVP 2026-05-26 21:02:39 +02:00
e1640071e4 Second-brain 2.0: hybrid retrieval, obsidian bridge, vector watermark, tests 2026-05-26 19:27:12 +02:00
29bc45d623 feat(systemd): Dashboard-Service, brain_rules, 18 Engramme bewertet, Cron persistent
Neu:
- systemd: secondbrain-dashboard.service (Port 8501, autostart)
- cron_rules.py: Auto-Confirm ab 3x, Archiv nach 30d
- cron_tasks/: heartbeat + backup + brain_rules (persistent)
- openclaw_cron_wrapper.py: subprocess-Isolation (kein SessionTakeover)
- chat_autosave.py: Auto-Save von Chat + Kontext-Anreicherung

Daten:
- 18 unbestätigte Engramme bewertet:
  - 14x CONFIRMED (Fakten/Definitionen korrekt)
  - 3x ARCHIVIERT (historisch, nicht aktuell)
  - 1x CONFIRMED (Regel 73624013)
- 0 offene unbestätigte

Closes Gitea-Issue: #9
2026-05-25 22:35:44 +02:00
a5d5b2f2ec fix(cron): Session-Takeover-Workaround, mobil-Dashboard, Import-Fix standalone
- feat(isolation): Cron-Wrapper (subprocess, /workspace/cron_tasks)
- fix(dashboard): mobil-optimierte Karten statt Tabelle
- fix(imports): sys.path + venv auto-activation
- chore: Lasse flüchtiges /tmp hinter mir

Closes-Bug: Session-Takeover bei isolated Cron
Engramme-bestaetigt: Standalone-Import-OK, Wrapper-getestet
2026-05-25 22:08:52 +02:00
4e0f5e7e9a feat(active): Proaktive Suche (Cron 4h), Aufgaben-Tracking, Heartbeat-Integration, Stop-Logik 2026-05-25 11:17:05 +02:00
2436460b27 fix(dashboard): remove st.secrets dependency, add .streamlit/secrets.toml, fix ChromaDB metadata types, v0.3.2 2026-05-25 11:03:46 +02:00
2e2cd2d228 fix(dashboard): st.secrets lazy load, chroma_store metadata types, ChromaDB sync 2026-05-25 10:49:38 +02:00
687f1df818 feat(complete): Phase 6 - Loop-Detector, Error-Healer, Grounding-Regel, erweiterte CLI 2026-05-25 10:26:53 +02:00
28 changed files with 2755 additions and 189 deletions

1
.streamlit/secrets.toml Normal file
View File

@@ -0,0 +1 @@
[default]

View File

@@ -8,7 +8,28 @@ An embeddable, offline-first memory system for AI agents with correctness tracki
- **ChromaDB Vector Store** (`src/chroma_store.py`) — Semantic similarity search - **ChromaDB Vector Store** (`src/chroma_store.py`) — Semantic similarity search
- **Neural Confidence Scorer** (`src/neural_scorer.py`) — PyTorch RL net, trains on confirm/reject feedback - **Neural Confidence Scorer** (`src/neural_scorer.py`) — PyTorch RL net, trains on confirm/reject feedback
- **Hybrid Retrieval** (`src/retriever.py`) — Keyword + Semantic + Neural fusion - **Hybrid Retrieval** (`src/retriever.py`) — Keyword + Semantic + Neural fusion
- **Streamlit Dashboard** (`src/app_dashboard.py`) — Search, confirm/reject, neural training UI - **FastAPI Dashboard** (`fastapi_app.py`) — Lightweight web UI (search + confirm/reject) and JSON API
- **Streamlit Dashboard** (`src/app_dashboard.py`) — (optional) richer UI for neural training, etc.
- **Graph Visualization** (`src/graph_view.py`) — Interactive Cytoscape.js graph with confidence colors - **Graph Visualization** (`src/graph_view.py`) — Interactive Cytoscape.js graph with confidence colors
## Architecture ## Architecture
## Obsidian
Setup and timers: `second-brain/docs/OBSIDIAN.md`
## Quickstart (Dashboard)
Install minimal dashboard deps:
`python3 -m pip install -r requirements-dashboard.txt`
Run:
`SECOND_BRAIN_WORKSPACE="$(pwd)" python3 fastapi_app.py`
Then open: `http://localhost:8501/`
Port is configurable via `SECOND_BRAIN_PORT` (or `PORT`), e.g.:
`SECOND_BRAIN_WORKSPACE="$(pwd)" SECOND_BRAIN_PORT=8502 python3 fastapi_app.py`

86
RUNBOOK.md Normal file
View File

@@ -0,0 +1,86 @@
# Second-Brain 2.0 (Grundversion) — Runbook
This is the operational quick-reference for the shipped systemd timers/services and the optional FastAPI dashboard backend.
Repository root (on host): `/root/.openclaw/workspace`
## Systemd units (cron jobs)
Unit files are shipped in `systemd/` (repo root). Install them into `/etc/systemd/system/` (symlink or copy), then reload:
```bash
sudo ln -sf /root/.openclaw/workspace/systemd/openclaw-secondbrain-*.service /etc/systemd/system/
sudo ln -sf /root/.openclaw/workspace/systemd/openclaw-secondbrain-*.timer /etc/systemd/system/
sudo ln -sf /root/.openclaw/workspace/systemd/openclaw-memory-archive.* /etc/systemd/system/
sudo systemctl daemon-reload
```
Enable timers:
```bash
sudo systemctl enable --now openclaw-secondbrain-ingest-memory.timer
sudo systemctl enable --now openclaw-secondbrain-index-vectors.timer
sudo systemctl enable --now openclaw-secondbrain-review.timer
sudo systemctl enable --now openclaw-secondbrain-backup.timer
sudo systemctl enable --now openclaw-secondbrain-heartbeat.timer
sudo systemctl enable --now openclaw-secondbrain-proactive-search.timer
sudo systemctl enable --now openclaw-memory-archive.timer
# Optional (Obsidian coupling)
sudo systemctl enable --now openclaw-secondbrain-ingest-obsidian.timer
sudo systemctl enable --now openclaw-secondbrain-export-obsidian.timer
```
Verify scheduling:
```bash
sudo systemctl list-timers --all | grep -E 'openclaw-(secondbrain|memory-archive)' || true
```
Run a job once:
```bash
sudo systemctl start openclaw-secondbrain-ingest-memory.service
sudo systemctl status openclaw-secondbrain-ingest-memory.service --no-pager
sudo journalctl -u openclaw-secondbrain-ingest-memory.service -n 200 --no-pager
```
Wrapper logs:
```bash
tail -n 200 /root/.openclaw/workspace/cron_wrapper.log
```
## FastAPI dashboard (manual start)
FastAPI entrypoint:
```bash
cd /root/.openclaw/workspace
python3 -m pip install -r second-brain/requirements-dashboard.txt
SECOND_BRAIN_WORKSPACE="/root/.openclaw/workspace/second-brain" python3 second-brain/fastapi_app.py
```
Default port is `8501` (same as Streamlit default). Do not run both on the same port.
Endpoint smoke tests:
```bash
curl -fsS http://127.0.0.1:8501/api/stats
curl -fsS "http://127.0.0.1:8501/api/engrams?limit=1&offset=0"
curl -fsS "http://127.0.0.1:8501/api/search?q=test&limit=1"
```
## DB quick check
```bash
python3 - <<'PY'
import sqlite3
db="/root/.openclaw/workspace/second-brain/data/brain.sqlite"
con=sqlite3.connect(db)
cur=con.cursor()
print(cur.execute("PRAGMA integrity_check").fetchone()[0])
print("engrams:", cur.execute("SELECT COUNT(*) FROM engrams").fetchone()[0])
con.close()
PY
```

119
chat_autosave.py Normal file
View File

@@ -0,0 +1,119 @@
#!/usr/bin/env python3
"""
Chat-Auto-Save: Wertvolle User-Nachrichten → Engramm.
Wird am Ende jeder Main-Session-Antwort aufgerufen.
"""
import sys
import json
import hashlib
from pathlib import Path
BRAIN_DIR = Path("/root/.openclaw/workspace/second-brain")
sys.path.insert(0, str(BRAIN_DIR))
from src.engram import Engram, Grounding
from src.store import EngramStore
DB_PATH = BRAIN_DIR / "data" / "brain.sqlite"
def _hash(text: str) -> str:
return hashlib.sha256(text.encode("utf-8")).hexdigest()[:12]
def is_fluff(content: str) -> bool:
"""Prüft ob Inhalt nur Floskel ist."""
lower = content.lower().strip().rstrip(".?!")
short_fluff = [
"hallo", "hi", "hey", "guten tag", "guten morgen", "guten abend",
"danke", "ok", "okay", "ja", "nein", "bitte", "gerne", "tschüss",
"bis später", "bis morgen", "alles klar", "in ordnung",
]
if lower in short_fluff:
return True
if len(content) < 10 and all(c in " ?,!.;:-" for c in content):
return True
return False
def save_if_worthy(content: str, source: str = "user", tags: list = None,
confidence: float = 0.7, session_id: str = None,
reasoning: str = None) -> dict:
"""
Speichert Nachricht als Engramm wenn sie Wert hat.
Wird in jeder Antwort aufgerufen.
"""
if is_fluff(content):
return {"saved": False, "reason": "fluff"}
store = EngramStore(str(DB_PATH))
content_hash = _hash(content)
recent = store.get_all(limit=200)
for eg in recent:
if _hash(eg.content) == content_hash:
return {"saved": False, "reason": "duplicate", "id": str(eg.id)}
eg = Engram.create(
content=content,
source=source,
tags=tags or ["auto-save", "chat"],
session_id=session_id,
confidence=confidence,
grounding=Grounding.ASSUMPTION,
)
store.save(eg)
return {
"saved": True,
"id": str(eg.id),
"confidence": eg.compute_confidence(),
"first8": str(eg.id)[:8],
}
def enrich_prompt(topic: str, limit: int = 3) -> str:
"""
Holt relevante bestätigte Engramme für Kontext-Anreicherung.
Wird VOR jeder Antwort aufgerufen.
"""
store = EngramStore(str(DB_PATH))
recent = store.get_all(limit=100)
# Einfache Text-Suche (kein FTS wegen Satzzeichen)
topic_lower = topic.lower()
matches = []
for eg in recent:
if eg.correctness.confirmed and topic_lower in eg.content.lower():
matches.append(eg)
elif len(matches) < limit and any(t in topic_lower for t in [t.lower() for t in eg.metadata.get("tags", [])]):
matches.append(eg)
if len(matches) >= limit:
break
if not matches:
return ""
lines = ["\n📚 Relevantes Wissen:"]
for eg in matches[:limit]:
lines.append(f" • [{eg.compute_confidence():.0%}] {eg.content[:120]}")
return "\n".join(lines)
def check_pending(session_id: str = None) -> list:
"""Gibt unbestätigte Engramme zurück."""
store = EngramStore(str(DB_PATH))
egs = store.get_all(limit=50)
pending = [eg for eg in egs if not eg.correctness.confirmed]
return pending
if __name__ == "__main__":
import sys
if len(sys.argv) > 1:
result = save_if_worthy(sys.argv[1])
print(json.dumps(result, indent=2))
else:
print("Usage: python3 chat_autosave.py 'Nachricht'")

View File

@@ -0,0 +1,22 @@
#!/usr/bin/env python3
"""Backup-Task für Second Brain - isoliert, persistent."""
import json, os, sys
from pathlib import Path
from datetime import datetime, timezone
BRAIN_DIR = Path("/root/.openclaw/workspace/second-brain")
sys.path.insert(0, str(BRAIN_DIR))
from src.store import EngramStore
def main():
brain_db = os.environ.get("BRAIN_DB", str(BRAIN_DIR / "data" / "brain.sqlite"))
store = EngramStore(brain_db)
ts = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S")
backup_path = Path(brain_db).parent / f"backup_{ts}.jsonl"
count = store.export_jsonl(str(backup_path))
result = {"timestamp": datetime.now(timezone.utc).isoformat(), "backup_path": str(backup_path), "count": count, "success": True}
print(f"BACKUP: {count} Engramme -> {backup_path}")
return 0
if __name__ == "__main__":
sys.exit(main())

53
cron_tasks/brain_rules.py Normal file
View File

@@ -0,0 +1,53 @@
#!/usr/bin/env python3
"""
Brain-Regeln - Automatische Bestaetigungs- und Archivierungslogik.
Wird von Cron und Agent aufgerufen.
"""
import sys
sys.path.insert(0, "/root/.openclaw/workspace/second-brain")
from src.engram import Engram, Grounding
from src.store import EngramStore
DB = "/root/.openclaw/workspace/second-brain/data/brain.sqlite"
def apply_rules():
store = EngramStore(DB)
egs = store.get_all(limit=1000)
actions = []
for eg in egs:
conf = eg.compute_confidence()
age_days = eg._age_days(eg.metadata.get("created", ""))
correct = eg.correctness
# Regel 1: Triple-Confirm → Auto-Verifiziert
if not correct.confirmed and correct.confirmations >= 3:
correct.confirmed = True
correct.confirmations += 1
store.save(eg)
actions.append(f"Auto-Confirm: {str(eg.id)[:8]} (3x confirmed)")
# Regel 2: Lang unbestaetigt → ASSUMPTION Tag
if age_days > 30 and not correct.confirmed and "archiviert" not in eg.metadata.get("tags", []):
eg.metadata.setdefault("tags", []).append("archiviert")
eg.metadata["archivgrund"] = f"Unbestaetigt seit {age_days} Tagen"
store.save(eg)
actions.append(f"Archiviert: {str(eg.id)[:8]} (Alter {age_days}d)")
# Regel 3: Rejected mit 2+ Rejections → loeschen (Sanft: Tag statt rm)
if correct.rejections >= 2:
eg.metadata.setdefault("tags", []).append("deleted")
store.save(eg)
actions.append(f"Deleted-Tag: {str(eg.id)[:8]} ({correct.rejections}x rejected)")
return actions
if __name__ == "__main__":
actions = apply_rules()
print("Brain-Regeln angewendet:")
for a in actions or ["Keine Aktionen noetig"]:
print(f" {a}")

View File

@@ -0,0 +1,47 @@
#!/usr/bin/env python3
"""
Heartbeat-Task für Second Brain - isoliert, persistent.
"""
import json
import os
import sys
from pathlib import Path
from datetime import datetime, timezone
BRAIN_DIR = Path("/root/.openclaw/workspace/second-brain")
sys.path.insert(0, str(BRAIN_DIR))
from src.engram import Engram, Grounding
from src.store import EngramStore
def main():
output_file = os.environ.get("CRON_OUTPUT_FILE", "/tmp/heartbeat_result.json")
brain_db = os.environ.get("BRAIN_DB", str(BRAIN_DIR / "data" / "brain.sqlite"))
store = EngramStore(brain_db)
egs = store.get_all(limit=50)
unconfirmed = [eg for eg in egs if not eg.correctness.confirmed and eg.compute_confidence() > 0.5][:5]
errors = store.search_tag("error", limit=5)
result = {
"timestamp": datetime.now(timezone.utc).isoformat(),
"total_engrams": len(egs),
"unconfirmed_count": len(unconfirmed),
"error_count": len(errors),
"has_action": bool(unconfirmed) or len(errors) >= 3,
"message": None,
}
if unconfirmed:
contents = "\n".join([f" - {eg.content[:80]}" for eg in unconfirmed])
result["message"] = f"🧠 Unbestätigte Engramme:\n{contents}"
elif len(errors) >= 3:
result["message"] = f"⚠️ {len(errors)} Fehler-Engramme gespeichert."
Path(output_file).write_text(json.dumps(result, indent=2))
print(f"HEARTBEAT: {result['unconfirmed_count']} unconfirmed, {result['error_count']} errors")
return 0
if __name__ == "__main__":
sys.exit(main())

75
docs/OBSIDIAN.md Normal file
View File

@@ -0,0 +1,75 @@
# Obsidian Coupling (Second-Brain 2.0)
This integrates an Obsidian vault with Second-Brain via two cron tasks:
- `cron_tasks/ingest_obsidian.py` (vault → Second-Brain)
- `cron_tasks/export_obsidian.py` (Second-Brain → vault)
All settings live in `second-brain/data/obsidian_config.json`.
## 1) Install / Sync the vault to the server
You need a local folder on the server that contains an Obsidian vault (it must contain a `.obsidian/` directory), e.g.:
- `/srv/obsidian/MyVault`
- `/data/obsidian/MyVault`
- `/root/Obsidian/MyVault`
How you sync it is up to you (Syncthing, rsync, SMB mount, etc.).
## 2) Set `vault_path` in config (auto or manual)
### Auto-discover (only writes if unambiguous)
```bash
python3 second-brain/scripts/discover_obsidian_vault.py
python3 second-brain/scripts/discover_obsidian_vault.py --write
```
If multiple vaults are detected, it prints them and refuses to write.
### Manual
Edit `second-brain/data/obsidian_config.json` and set:
- `vault_path` to the vault directory (the parent of `.obsidian/`)
## 3) Enable ingest/export
In `second-brain/data/obsidian_config.json`:
- Set `enabled.ingest` to `true` to ingest vault markdown into Second-Brain
- Set `enabled.export` to `true` to export Second-Brain engrams into the vault
## 4) Enable timers (systemd)
This repo ships unit files in `systemd/`:
- `systemd/openclaw-secondbrain-ingest-obsidian.service`
- `systemd/openclaw-secondbrain-ingest-obsidian.timer`
- `systemd/openclaw-secondbrain-export-obsidian.service`
- `systemd/openclaw-secondbrain-export-obsidian.timer`
Install them (copy or symlink) to `/etc/systemd/system/`, then:
```bash
sudo systemctl daemon-reload
sudo systemctl enable --now openclaw-secondbrain-ingest-obsidian.timer
sudo systemctl enable --now openclaw-secondbrain-export-obsidian.timer
```
## 5) Verify
Run once manually:
```bash
python3 openclaw_cron_wrapper.py ingest_obsidian
python3 openclaw_cron_wrapper.py export_obsidian
```
What to expect:
- If `vault_path` is missing/invalid, both tasks **skip** safely (no writes to random paths).
- Ingest creates/updates `second-brain/data/obsidian_ingest_state.json`.
- Export writes markdown files to `<vault_path>/<export.subdir>/` (default: `SecondBrain/`) and tracks state in `second-brain/data/obsidian_export_state.json`.

105
docs/RELEASE_CHECKLIST.md Normal file
View File

@@ -0,0 +1,105 @@
# Second-Brain 2.0 — Release Checklist (Grundversion + FastAPI)
Scope: OpenClaw workspace at `/root/.openclaw/workspace` with `second-brain/` and the shipped `systemd/` unit files.
This checklist is copy/paste friendly. Run as a user with `sudo` (systemd commands need it).
## 0) Preconditions (one-time per host)
### Repo + Python sanity
```bash
cd /root/.openclaw/workspace
test -d second-brain || { echo "missing: second-brain/"; exit 1; }
python3 --version
```
### Ensure systemd units are installed (copy/symlink)
```bash
ls -la /etc/systemd/system/openclaw-secondbrain-*.timer /etc/systemd/system/openclaw-secondbrain-*.service 2>/dev/null || true
ls -la /etc/systemd/system/openclaw-memory-archive.* 2>/dev/null || true
```
If missing, install them (symlink is fine; copy is fine too):
```bash
sudo ln -sf /root/.openclaw/workspace/systemd/openclaw-secondbrain-*.service /etc/systemd/system/
sudo ln -sf /root/.openclaw/workspace/systemd/openclaw-secondbrain-*.timer /etc/systemd/system/
sudo ln -sf /root/.openclaw/workspace/systemd/openclaw-memory-archive.* /etc/systemd/system/
sudo ln -sf /root/.openclaw/workspace/systemd/openclaw-secondbrain-dashboard.service /etc/systemd/system/
sudo systemctl daemon-reload
```
### Enable timers
```bash
sudo systemctl enable --now openclaw-secondbrain-backup.timer
sudo systemctl enable --now openclaw-secondbrain-ingest-memory.timer
sudo systemctl enable --now openclaw-secondbrain-index-vectors.timer
sudo systemctl enable --now openclaw-secondbrain-review.timer
sudo systemctl enable --now openclaw-secondbrain-heartbeat.timer
sudo systemctl enable --now openclaw-secondbrain-proactive-search.timer
sudo systemctl enable --now openclaw-memory-archive.timer
```
## 1) Release QA — systemd status + timers
### Verify timers are active and scheduled
```bash
sudo systemctl list-timers --all | grep -E 'openclaw-(secondbrain|memory-archive)' || true
sudo systemctl --failed --no-pager || true
```
### Verify the oneshot services can run successfully (manual trigger)
```bash
sudo systemctl start openclaw-secondbrain-ingest-memory.service
sudo systemctl start openclaw-secondbrain-index-vectors.service
sudo systemctl start openclaw-secondbrain-review.service
sudo systemctl start openclaw-secondbrain-backup.service
sudo systemctl start openclaw-secondbrain-heartbeat.service
sudo systemctl start openclaw-secondbrain-proactive-search.service
sudo systemctl start openclaw-memory-archive.service
```
Logs:
```bash
tail -n 200 /root/.openclaw/workspace/cron_wrapper.log
sudo journalctl -u openclaw-secondbrain-review.service -n 200 --no-pager
```
## 2) Release QA — data + DB invariants
```bash
ls -la /root/.openclaw/workspace/second-brain/data/brain.sqlite
python3 - <<'PY'
import sqlite3
db="/root/.openclaw/workspace/second-brain/data/brain.sqlite"
con=sqlite3.connect(db)
cur=con.cursor()
print("integrity_check:", cur.execute("PRAGMA integrity_check").fetchone()[0])
print("engrams:", cur.execute("SELECT COUNT(*) FROM engrams").fetchone()[0])
con.close()
PY
```
## 3) Release QA — FastAPI (HTTP endpoints + logs)
FastAPI service:
```bash
sudo systemctl enable --now openclaw-secondbrain-dashboard.service
sudo systemctl status openclaw-secondbrain-dashboard.service --no-pager
```
Endpoint checks:
```bash
curl -fsS http://127.0.0.1:8501/healthz && echo
curl -fsS http://127.0.0.1:8501/api/config
curl -fsS http://127.0.0.1:8501/api/stats
curl -fsS "http://127.0.0.1:8501/api/engrams?limit=1&offset=0"
```

418
fastapi_app.py Normal file
View File

@@ -0,0 +1,418 @@
#!/usr/bin/env python3
"""
Second Brain FastAPI Dashboard
Goals:
- "Release-ready" defaults (no hardcoded absolute paths)
- Minimal config via env vars
- Serves the existing static dashboard (templates/dashboard.html + static/)
"""
import json
import os
import sqlite3
from datetime import datetime, timezone
from pathlib import Path
from fastapi import FastAPI, Form, Query, Request
from fastapi.responses import HTMLResponse, JSONResponse, PlainTextResponse
from fastapi.staticfiles import StaticFiles
# ─── Config ──────────────────────────────────────────────────────────────────
REPO_ROOT = Path(__file__).resolve().parent
WORKSPACE = Path(os.environ.get("SECOND_BRAIN_WORKSPACE", str(REPO_ROOT))).resolve()
DB_PATH = Path(os.environ.get("SECOND_BRAIN_DB_PATH", str(WORKSPACE / "data" / "brain.sqlite"))).resolve()
PORT = int(os.environ.get("SECOND_BRAIN_PORT", os.environ.get("PORT", "8501")))
HOST = os.environ.get("SECOND_BRAIN_HOST", "0.0.0.0")
def create_app() -> FastAPI:
app = FastAPI(title="Second Brain Dashboard")
static_dir = WORKSPACE / "static"
if static_dir.is_dir():
app.mount("/static", StaticFiles(directory=str(static_dir)), name="static")
return app
app = create_app()
# ─── Helpers ─────────────────────────────────────────────────────────────────
def get_db():
if not DB_PATH.exists():
raise FileNotFoundError(f"DB not found: {DB_PATH}")
conn = sqlite3.connect(str(DB_PATH))
conn.row_factory = sqlite3.Row
return conn
def parse_engram(row: sqlite3.Row) -> dict:
meta = json.loads(row["metadata_json"] or "{}")
correctness = json.loads(row["correctness_json"] or "{}")
return {
"id": row["id"],
"content": row["content"],
"confidence": meta.get("confidence", 0.0),
"confirmed": correctness.get("confirmed", False),
"confirmations": correctness.get("confirmations", 0),
"rejections": correctness.get("rejections", 0),
"tags": meta.get("tags", []),
"created": meta.get("created", row["created_at"]),
"modified": meta.get("modified", row["modified_at"]),
"last_reviewed": correctness.get("last_reviewed"),
"review_history": correctness.get("review_history", []),
"source": meta.get("source", "unknown"),
"access_count": meta.get("access_count", 0),
"grounding": meta.get("grounding", 0),
}
# ─── API Endpoints ───────────────────────────────────────────────────────────
@app.get("/healthz", response_class=PlainTextResponse)
def healthz():
return "ok"
@app.get("/api/config")
def api_config():
return {
"workspace": str(WORKSPACE),
"db_path": str(DB_PATH),
}
@app.exception_handler(FileNotFoundError)
def handle_file_not_found(request: Request, exc: FileNotFoundError):
return JSONResponse(
status_code=503,
content={
"error": str(exc),
"hint": "Set SECOND_BRAIN_DB_PATH or SECOND_BRAIN_WORKSPACE to a valid location.",
},
)
@app.exception_handler(sqlite3.Error)
def handle_sqlite_error(request: Request, exc: sqlite3.Error):
return JSONResponse(status_code=500, content={"error": f"sqlite error: {exc}"})
@app.get("/api/stats")
def api_stats():
conn = get_db()
c = conn.cursor()
total = c.execute("SELECT COUNT(*) FROM engrams").fetchone()[0]
confirmed = c.execute(
"SELECT COUNT(*) FROM engrams WHERE json_extract(correctness_json, '$.confirmed') = 1"
).fetchone()[0]
pending = total - confirmed
errors = c.execute(
"SELECT COUNT(*) FROM engrams WHERE json_extract(metadata_json, '$.tags') LIKE '%error%'"
).fetchone()[0]
avg_conf = c.execute(
"SELECT AVG(json_extract(metadata_json, '$.confidence')) FROM engrams"
).fetchone()[0] or 0.0
conn.close()
return {
"total": total,
"confirmed": confirmed,
"pending": pending,
"errors": errors,
"avg_confidence": round(avg_conf, 2),
}
@app.get("/api/engrams")
def api_engrams(
limit: int = Query(20, ge=1, le=100),
offset: int = Query(0, ge=0),
tag: str = Query(None),
confirmed: bool = Query(None),
search: str = Query(None),
min_confidence: float = Query(0.0),
):
conn = get_db()
c = conn.cursor()
where_clauses = ["json_extract(metadata_json, '$.confidence') >= ?"]
params = [min_confidence]
if tag:
where_clauses.append("json_extract(metadata_json, '$.tags') LIKE ?")
params.append(f'%"{tag}"%')
if confirmed is not None:
where_clauses.append(
f"json_extract(correctness_json, '$.confirmed') = {int(confirmed)}"
)
if search:
# Use FTS
try:
ids = [
r[0] for r in c.execute(
"SELECT rowid FROM engrams_fts WHERE content MATCH ? LIMIT 200",
(search,)
).fetchall()
]
if ids:
placeholders = ",".join("?" * len(ids))
where_clauses.append(f"id IN ({placeholders})")
params.extend(ids)
else:
# Full-text fallback on content
where_clauses.append("content LIKE ?")
params.append(f"%{search}%")
except Exception:
where_clauses.append("content LIKE ?")
params.append(f"%{search}%")
where_sql = " AND ".join(where_clauses)
rows = c.execute(
f"""
SELECT * FROM engrams
WHERE {where_sql}
ORDER BY created_at DESC
LIMIT ? OFFSET ?
""",
params + [limit, offset],
).fetchall()
result = [parse_engram(r) for r in rows]
conn.close()
return {"items": result, "limit": limit, "offset": offset}
@app.get("/api/engrams/{engram_id}")
def api_engram_detail(engram_id: str):
conn = get_db()
c = conn.cursor()
row = c.execute("SELECT * FROM engrams WHERE id = ?", (engram_id,)).fetchone()
if not row:
conn.close()
return JSONResponse({"error": "Not found"}, status_code=404)
# Links
links = c.execute(
"SELECT to_id FROM engrams_links WHERE from_id = ?", (engram_id,)
).fetchall()
result = parse_engram(row)
result["links"] = [r[0] for r in links]
conn.close()
return result
@app.post("/api/engrams/{engram_id}/confirm")
def api_confirm(engram_id: str, reason: str = Form("")):
conn = get_db()
c = conn.cursor()
row = c.execute(
"SELECT correctness_json FROM engrams WHERE id = ?", (engram_id,)
).fetchone()
if not row:
conn.close()
return JSONResponse({"error": "Not found"}, status_code=404)
correctness = json.loads(row["correctness_json"] or "{}")
correctness["confirmed"] = True
correctness["confirmations"] = correctness.get("confirmations", 0) + 1
correctness["last_reviewed"] = datetime.now(timezone.utc).isoformat()
review_history = correctness.get("review_history", [])
review_history.append({
"by": "web",
"action": "confirm",
"at": datetime.now(timezone.utc).isoformat(),
"note": reason or "confirmed via dashboard",
})
correctness["review_history"] = review_history
c.execute(
"UPDATE engrams SET correctness_json = ?, modified_at = ? WHERE id = ?",
(json.dumps(correctness), datetime.now(timezone.utc).isoformat(), engram_id),
)
conn.commit()
conn.close()
return {"success": True, "engram_id": engram_id}
@app.post("/api/engrams/{engram_id}/reject")
def api_reject(engram_id: str, reason: str = Form("")):
conn = get_db()
c = conn.cursor()
row = c.execute(
"SELECT correctness_json FROM engrams WHERE id = ?", (engram_id,)
).fetchone()
if not row:
conn.close()
return JSONResponse({"error": "Not found"}, status_code=404)
correctness = json.loads(row["correctness_json"] or "{}")
correctness["confirmed"] = False
correctness["rejections"] = correctness.get("rejections", 0) + 1
correctness["last_reviewed"] = datetime.now(timezone.utc).isoformat()
review_history = correctness.get("review_history", [])
review_history.append({
"by": "web",
"action": "reject",
"at": datetime.now(timezone.utc).isoformat(),
"note": reason or "rejected via dashboard",
})
correctness["review_history"] = review_history
c.execute(
"UPDATE engrams SET correctness_json = ?, modified_at = ? WHERE id = ?",
(json.dumps(correctness), datetime.now(timezone.utc).isoformat(), engram_id),
)
conn.commit()
conn.close()
return {"success": True, "engram_id": engram_id}
@app.post("/api/engrams/{engram_id}/refresh")
def api_refresh(engram_id: str):
conn = get_db()
c = conn.cursor()
row = c.execute(
"SELECT metadata_json, correctness_json FROM engrams WHERE id = ?", (engram_id,)
).fetchone()
if not row:
conn.close()
return JSONResponse({"error": "Not found"}, status_code=404)
meta = json.loads(row["metadata_json"] or "{}")
correctness = json.loads(row["correctness_json"] or "{}")
# Simple heuristic: confidence based on confirmations vs rejections
conf = 0.5
conf += 0.1 * correctness.get("confirmations", 0)
conf -= 0.15 * correctness.get("rejections", 0)
conf = max(0.1, min(1.0, conf))
meta["confidence"] = round(conf, 2)
meta["modified"] = datetime.now(timezone.utc).isoformat()
c.execute(
"UPDATE engrams SET metadata_json = ?, modified_at = ? WHERE id = ?",
(json.dumps(meta), datetime.now(timezone.utc).isoformat(), engram_id),
)
conn.commit()
conn.close()
return {"success": True, "new_confidence": round(conf, 2)}
@app.post("/api/engrams")
def api_create_engram(content: str = Form(...), tags: str = Form(""), source: str = Form("web")):
engram_id = f"web-{datetime.now(timezone.utc).strftime('%Y%m%d-%H%M%S-%f')[:20]}"
now = datetime.now(timezone.utc).isoformat()
meta = {
"source": source,
"confidence": 0.5,
"created": now,
"modified": now,
"access_count": 0,
"last_accessed": now,
"tags": [t.strip() for t in tags.split(",") if t.strip()] or ["web"],
"session_id": None,
"agent_id": None,
"grounding": 0,
"hash": "",
}
correctness = {
"confirmed": False,
"confirmations": 0,
"rejections": 0,
"last_reviewed": None,
"review_history": [],
}
conn = get_db()
c = conn.cursor()
c.execute(
"""
INSERT INTO engrams (id, content, metadata_json, correctness_json, links_json, hierarchy_json, created_at, modified_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
""",
(engram_id, content, json.dumps(meta), json.dumps(correctness), "[]", '{"parent": null, "children": [], "depth": 0}', now, now),
)
conn.commit()
conn.close()
return {"success": True, "engram_id": engram_id}
@app.get("/api/pending")
def api_pending(limit: int = Query(20, ge=1, le=100), offset: int = Query(0, ge=0)):
conn = get_db()
c = conn.cursor()
rows = c.execute(
"""
SELECT * FROM engrams
WHERE json_extract(correctness_json, '$.confirmed') = 0
ORDER BY created_at DESC
LIMIT ? OFFSET ?
""",
(limit, offset),
).fetchall()
result = [parse_engram(r) for r in rows]
conn.close()
return {"items": result, "limit": limit, "offset": offset}
@app.get("/api/search")
def api_search(
q: str = Query(..., min_length=1),
min_confidence: float = Query(0.0),
limit: int = Query(20, ge=1, le=100),
):
conn = get_db()
c = conn.cursor()
try:
ids = [
r[0] for r in c.execute(
"SELECT rowid FROM engrams_fts WHERE content MATCH ? LIMIT 200",
(q,)
).fetchall()
]
if ids:
placeholders = ",".join("?" * len(ids))
rows = c.execute(
f"""
SELECT * FROM engrams
WHERE id IN ({placeholders})
AND json_extract(metadata_json, '$.confidence') >= ?
ORDER BY created_at DESC
LIMIT ?
""",
ids + [min_confidence, limit],
).fetchall()
else:
rows = []
except Exception:
rows = c.execute(
"""
SELECT * FROM engrams
WHERE content LIKE ? AND json_extract(metadata_json, '$.confidence') >= ?
ORDER BY created_at DESC
LIMIT ?
""",
(f"%{q}%", min_confidence, limit),
).fetchall()
result = [parse_engram(r) for r in rows]
conn.close()
return {"items": result, "query": q}
# ─── Frontend ────────────────────────────────────────────────────────────────
@app.get("/", response_class=HTMLResponse)
def dashboard(request: Request):
with open(WORKSPACE / "templates" / "dashboard.html", "r", encoding="utf-8") as f:
html = f.read()
return HTMLResponse(content=html)
# ─── Main ────────────────────────────────────────────────────────────────────
if __name__ == "__main__":
import uvicorn
uvicorn.run("fastapi_app:app", host=HOST, port=PORT)

102
openclaw_cron_wrapper.py Normal file
View File

@@ -0,0 +1,102 @@
#!/usr/bin/env python3
"""
OpenClaw Cron Isolation Wrapper - Updatesicherer Workaround.
Persistent: Tasks und Logs liegen im Workspace, nicht in /tmp.
"""
import os
import sys
import json
import subprocess
import tempfile
from pathlib import Path
from datetime import datetime, timezone
# --- Konfiguration (persistent) ---
WORKSPACE = Path("/root/.openclaw/workspace")
CRON_TASKS_DIR = WORKSPACE / "cron_tasks"
LOG_FILE = WORKSPACE / "cron_wrapper.log"
BRAIN_DIR = WORKSPACE / "second-brain"
def log(msg: str):
ts = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
line = f"[{ts}] {msg}\n"
with open(LOG_FILE, "a") as f:
f.write(line)
print(line.strip())
def run_isolated(task_name: str, task_args: dict = None) -> dict:
"""
Führt einen Task in echt isolierter Umgebung aus.
Kein Zugriff auf Session-Files, nur stdout/stderr.
"""
CRON_TASKS_DIR.mkdir(parents=True, exist_ok=True)
task_script = CRON_TASKS_DIR / f"{task_name}.py"
if not task_script.exists():
return {"success": False, "error": f"Task nicht gefunden: {task_script}"}
# Temp-Verzeichnis für Output (flüchtig ist OK, Ergebnis kommt via stdout)
temp_dir = tempfile.mkdtemp(prefix=f"cron_{task_name}_")
output_file = Path(temp_dir) / "output.json"
# Saubere Env: Keine OpenClaw-Session-Variablen
env = os.environ.copy()
for key in list(env.keys()):
if "OPENCLAW" in key.upper() or "SESSION" in key.upper():
del env[key]
env["CRON_TASK_NAME"] = task_name
env["CRON_OUTPUT_FILE"] = str(output_file)
env["BRAIN_DB"] = str(BRAIN_DIR / "data" / "brain.sqlite")
try:
result = subprocess.run(
[sys.executable, str(task_script)] + ([json.dumps(task_args)] if task_args else []),
capture_output=True,
text=True,
timeout=300,
cwd=str(temp_dir),
env=env,
)
stdout = result.stdout.strip()
stderr = result.stderr.strip()
output_data = {}
if output_file.exists():
try:
output_data = json.loads(output_file.read_text())
except Exception:
output_data = {"raw": output_file.read_text()}
return {
"success": result.returncode == 0,
"returncode": result.returncode,
"stdout": stdout[-2000:] if stdout else "",
"stderr": stderr[-1000:] if stderr else "",
"output": output_data,
}
except subprocess.TimeoutExpired:
return {"success": False, "error": "Timeout nach 300s"}
except Exception as e:
return {"success": False, "error": str(e)}
def main():
import argparse
parser = argparse.ArgumentParser(description="OpenClaw Cron Isolation Wrapper")
parser.add_argument("task", help="Task-Name aus cron_tasks/")
parser.add_argument("--args", help="JSON-Args für den Task")
args = parser.parse_args()
task_args = json.loads(args.args) if args.args else None
result = run_isolated(args.task, task_args)
print(json.dumps(result, indent=2, default=str))
return 0 if result["success"] else 1
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,4 @@
fastapi>=0.110
uvicorn[standard]>=0.23
jinja2>=3.1
python-multipart>=0.0.9

View File

@@ -0,0 +1,159 @@
#!/usr/bin/env python3
"""
Auto-discover an Obsidian vault on this server and (optionally) write it into:
second-brain/data/obsidian_config.json
Safety:
- Only writes when exactly one vault is detected (unambiguous).
- A "vault" is a directory that contains a `.obsidian/` folder.
"""
from __future__ import annotations
import argparse
import json
import os
from pathlib import Path
from typing import Iterable
WORKSPACE = Path("/root/.openclaw/workspace")
BRAIN_DIR = WORKSPACE / "second-brain"
CONFIG_PATH = BRAIN_DIR / "data" / "obsidian_config.json"
def _iter_common_candidates() -> Iterable[Path]:
env = os.environ.get("OBSIDIAN_VAULT_PATH")
if env:
yield Path(env).expanduser()
home = Path.home()
for p in [
home / "Obsidian",
home / "ObsidianVault",
home / "Vault",
home / "Vaults",
home / "Documents" / "Obsidian",
home / "Documents" / "Vaults",
home / "Syncthing" / "Obsidian",
Path("/srv/obsidian"),
Path("/srv/Obsidian"),
Path("/data/obsidian"),
Path("/data/Obsidian"),
WORKSPACE / "obsidian",
WORKSPACE / "vault",
WORKSPACE / "vaults",
]:
yield p
def _is_vault_dir(p: Path) -> bool:
try:
return p.exists() and p.is_dir() and (p / ".obsidian").exists() and (p / ".obsidian").is_dir()
except Exception:
return False
def _bounded_find_obsidian_dirs(root: Path, *, max_depth: int) -> list[Path]:
"""
Find `.obsidian` directories below root, limited by depth to keep runtime bounded.
"""
results: list[Path] = []
try:
root = root.resolve()
except Exception:
return results
if not root.exists() or not root.is_dir():
return results
def depth_of(path: Path) -> int:
try:
return len(path.relative_to(root).parts)
except Exception:
return 9999
# Breadth-first-ish scan with pruning
queue = [root]
while queue:
current = queue.pop(0)
if depth_of(current) > max_depth:
continue
try:
entries = list(current.iterdir())
except Exception:
continue
for e in entries:
name = e.name
if name in (".git", "node_modules", "__pycache__", ".cache", ".venv", "venv", "tmp", "proc", "sys", "dev"):
continue
if name.startswith(".") and name not in (".obsidian",):
continue
if name == ".obsidian" and e.is_dir():
results.append(e)
continue
if e.is_dir() and not e.is_symlink():
queue.append(e)
return results
def discover(*, roots: list[Path], max_depth: int) -> list[Path]:
vaults: set[Path] = set()
for p in _iter_common_candidates():
if _is_vault_dir(p):
vaults.add(p.resolve())
for root in roots:
for obsidian_dir in _bounded_find_obsidian_dirs(root, max_depth=max_depth):
vaults.add(obsidian_dir.parent.resolve())
return sorted(vaults)
def main() -> int:
ap = argparse.ArgumentParser(description="Discover Obsidian vault and optionally write config")
ap.add_argument("--write", action="store_true", help="Write detected vault_path into obsidian_config.json")
ap.add_argument(
"--roots",
nargs="*",
default=[str(Path.home()), "/srv", "/data", "/mnt", str(WORKSPACE)],
help="Roots to scan (bounded). Default: home,/srv,/data,/mnt,workspace",
)
ap.add_argument("--max-depth", type=int, default=4, help="Max directory depth to scan under each root")
args = ap.parse_args()
roots = [Path(r).expanduser() for r in args.roots]
vaults = discover(roots=roots, max_depth=int(args.max_depth))
if not vaults:
print("No Obsidian vault found (no `.obsidian/` directories detected).")
return 1
if len(vaults) > 1:
print("Multiple Obsidian vaults found; refusing to write config:")
for v in vaults:
print(f"- {v}")
return 2
vault = vaults[0]
print(f"Detected Obsidian vault: {vault}")
if not args.write:
return 0
if not CONFIG_PATH.exists():
raise SystemExit(f"Missing config file: {CONFIG_PATH}")
cfg = json.loads(CONFIG_PATH.read_text(encoding="utf-8"))
cfg["vault_path"] = str(vault)
CONFIG_PATH.write_text(json.dumps(cfg, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
print(f"Wrote vault_path to: {CONFIG_PATH}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -1,8 +1,14 @@
"""Second Brain - Gedächtnissystem für OpenClaw.""" """Second Brain - Gedächtnissystem für OpenClaw."""
try:
from .engram import Engram, Grounding, Correctness, ReviewEntry from .engram import Engram, Grounding, Correctness, ReviewEntry
from .store import EngramStore from .store import EngramStore
from .retriever import Retriever from .retriever import Retriever
except ImportError:
# Fallback: ChromaDB optional, SQLite-core funktioniert immer
from .engram import Engram, Grounding, Correctness, ReviewEntry
from .store import EngramStore
Retriever = None
__version__ = "0.1.0" __version__ = "0.1.0"
__all__ = ["Engram", "Grounding", "Correctness", "ReviewEntry", "EngramStore", "Retriever"] __all__ = ["Engram", "Grounding", "Correctness", "ReviewEntry", "EngramStore", "Retriever"]

View File

@@ -1,174 +1,210 @@
""" """
app_dashboard.py - Streamlit-Dashboard für Second Brain. app_dashboard.py - Streamlit-Dashboard für Second Brain.
Seiten: Übersicht, Engramme, Suche, Graph, Stats. Seiten: Übersicht, Engramme, Suche, Graph, Heal-Log, Neural Scorer.
""" """
import json import json
import sys import sys
import os
from pathlib import Path from pathlib import Path
import streamlit as st import streamlit as st
sys.path.insert(0, str(Path(__file__).resolve().parent)) _root = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(_root))
from src.engram import Engram from src.engram import Engram
from src.store import EngramStore from src.store import EngramStore
from src.chroma_store import ChromaStore from src.chroma_store import ChromaStore
from src.retriever import Retriever from src.retriever import Retriever
from src.neural_scorer import NeuralScorer from src.neural_scorer import NeuralScorer
from src.graph_view import generate_graph_html
from src.loop_detector import LoopDetector
from src.error_healer import ErrorHealer
_DEFAULT_DB = Path(__file__).resolve().parent.parent / "data" / "brain.sqlite" _DEFAULT_DB = _root / "data" / "brain.sqlite"
_DB_PATH = str(st.secrets.get("db_path", _DEFAULT_DB) if hasattr(st, "secrets") else _DEFAULT_DB)
@st.cache_resource
def _store(): def _store():
return EngramStore(_DB_PATH) return EngramStore(str(_DEFAULT_DB))
@st.cache_resource
def _chroma(): def _chroma():
p = Path(_DB_PATH).parent / "chroma" p = Path(str(_DEFAULT_DB)).parent / "chroma"
return ChromaStore(str(p)) return ChromaStore(str(p))
_retriever_cache = None
def _retriever(): def _retriever():
return Retriever(_store(), _chroma()) global _retriever_cache
if _retriever_cache is None:
_retriever_cache = Retriever(_store(), _chroma())
return _retriever_cache
@st.cache_resource
def _scorer(): def _scorer():
return NeuralScorer() return NeuralScorer()
st.set_page_config(page_title="Second Brain Dashboard", layout="wide") @st.cache_resource
st.title("🧠 Second Brain Dashboard") def _healer():
return ErrorHealer(_store())
page = st.sidebar.radio("Seite", ["Übersicht", "Engramme", "Suche", "Graph", "Stats", "Neural Scorer"])
st.set_page_config(page_title="Second Brain Dashboard", layout="wide")
st.title("🧠 2.Brain v0.3.1")
page = st.sidebar.radio("Seite", ["Übersicht", "Engramme", "Suche", "Graph", "Heal-Log", "Neural Scorer"])
if page == "Übersicht": if page == "Übersicht":
store = _store() store = _store()
engrams = store.get_all() engrams = store.get_all(limit=10000)
confirmed = sum(1 for e in engrams if e.correctness.confirmed) confirmed = sum(1 for e in engrams if e.correctness.confirmed)
unconfirmed = len(engrams) - confirmed unconfirmed = len(engrams) - confirmed
avg_conf = sum(e.compute_confidence() for e in engrams) / max(1, len(engrams)) avg_conf = sum(e.compute_confidence() for e in engrams) / max(1, len(engrams))
errors = [e for e in engrams if "error" in e.metadata.get("tags", [])]
c1, c2, c3, c4 = st.columns(4) c1, c2, c3, c4, c5 = st.columns(5)
c1.metric("Total", len(engrams)) c1.metric("Total", len(engrams))
c2.metric("Confirmed", confirmed) c2.metric("Confirmed", confirmed)
c3.metric("Pending", unconfirmed) c3.metric("Pending", unconfirmed)
c4.metric("Avg Confidence", f"{avg_conf:.2f}") c4.metric("Avg Confidence", f"{avg_conf:.2f}")
c5.metric("Errors", len(errors))
st.subheader("Recent Engramme") st.subheader("Recent Engramme")
for eg in sorted(engrams, key=lambda e: e.metadata.get("modified", ""), reverse=True)[:5]: for eg in sorted(engrams, key=lambda e: e.metadata.get("modified", ""), reverse=True)[:5]:
with st.expander(f"{eg.content[:80]}..."): valid = eg.validate_grounding()
marker = "" if valid["valid"] else "⚠️"
with st.expander(f"{marker} {eg.content[:80]}..."):
st.write(f"ID: `{eg.id}`")
st.write(f"Source: {eg.metadata.get('source')}") st.write(f"Source: {eg.metadata.get('source')}")
st.write(f"Confidence: {eg.compute_confidence():.2f}") st.write(f"Confidence: {eg.compute_confidence():.2f}")
st.write(f"Confirmed: {'' if eg.correctness.confirmed else ''}") st.write(f"Confirmed: {'' if eg.correctness.confirmed else ''}")
st.write("Tags:", ", ".join(eg.metadata.get("tags", []))) st.write("Tags:", ", ".join(eg.metadata.get("tags", [])))
if not valid["valid"]:
st.warning(f"Grounding: {valid['issue']}")
if st.button("Auto-Fix", key=f"af_{eg.id}"):
eg.auto_fix_grounding()
store.save(eg)
st.experimental_rerun()
elif page == "Engramme": elif page == "Engramme":
store = _store() store = _store()
st.subheader("Alle Engramme") st.subheader("Alle Engramme (max 1000)")
tag_filter = st.text_input("Filter tags") tag_filter = st.text_input("Filter tags")
source_filter = st.selectbox("Source", ["alle", "user", "agent", "web", "file", "system"]) source_filter = st.selectbox("Source", ["alle", "user", "agent", "web", "file", "system"])
for eg in store.get_all(): for eg in store.get_all(limit=1000):
tags = eg.metadata.get("tags", []) tags = eg.metadata.get("tags", [])
src = eg.metadata.get("source", "") src = eg.metadata.get("source", "")
if tag_filter and tag_filter not in tags: if tag_filter and tag_filter not in tags:
continue continue
if source_filter != "alle" and source_filter != src: if source_filter != "alle" and source_filter != src:
continue continue
with st.expander(f"{eg.content[:100]}"): col1, col2 = st.columns([4, 1])
st.write("Confidence:", f"{eg.compute_confidence():.2f}") with col1:
st.write("Tags:", ", ".join(tags)) conf = eg.compute_confidence()
st.write("Source:", src) marker = "" if conf > 0.7 else "⚠️"
c1, c2 = st.columns(2) st.markdown(f"{marker} **{eg.content[:100]}**")
if c1.button("Confirm", key=f"conf_{eg.id}"): st.caption(f"Conf: {conf:.2f} | Tags: {', '.join(tags)} | Source: {src}")
with col2:
if st.button("✅ Confirm", key=f"conf_{eg.id}"):
eg.correctness.confirm("user") eg.correctness.confirm("user")
store.save(eg) store.save(eg)
st.success("Confirmed!") st.success("Confirmed")
if c2.button("❌ Reject", key=f"rej_{eg.id}"): if st.button("❌ Reject", key=f"rej_{eg.id}"):
eg.correctness.reject("user") eg.correctness.reject("user")
store.save(eg) store.save(eg)
st.warning("Rejected.") st.warning("Rejected")
st.divider()
elif page == "Suche": elif page == "Suche":
st.subheader("Semantic + Keyword Suche") st.subheader("Hybrid Search (Semantic + Keyword)")
query = st.text_input("Query") query = st.text_input("Query", placeholder="Suchbegriff eingeben...")
mode = st.radio("Modus", ["Hybrid", "Keyword", "Semantic"]) mode = st.radio("Modus", ["Hybrid", "Keyword", "Semantic"], horizontal=True)
if st.button("Suchen") and query: if st.button("Suchen") and query:
ret = _retriever() ret = _retriever()
if mode == "Hybrid": results = ret.hybrid_retrieve(query, limit=10) if mode == "Hybrid" else \
results = ret.hybrid_retrieve(query, limit=10) ret.semantic_retrieve(query, limit=10) if mode == "Semantic" else \
elif mode == "Semantic": ret.retrieve(query, limit=10)
results = ret.semantic_retrieve(query, limit=10) if not results:
else: st.info("Keine Ergebnisse gefunden.")
results = ret.retrieve(query, limit=10)
for r in results: for r in results:
eg = r["engram"] eg = r["engram"]
with st.container(): with st.container():
st.markdown(f"**{eg.content[:200]}...**") st.markdown(f"**{eg.content[:200]}...**")
st.write(f"Score: {r['score']:.3f} | Match: {r['match_type']} | Conf: {eg.compute_confidence():.2f}") st.write(f"Score: `{r['score']:.3f}` | Match: `{r['match_type']}` | Conf: `{eg.compute_confidence():.2f}`")
c1, c2 = st.columns(2) c1, c2 = st.columns(2)
if c1.button("✅ Confirm", key=f"sc_{eg.id}"): if c1.button("✅ Confirm", key=f"sc_{eg.id}"):
eg.correctness.confirm("user") eg.correctness.confirm("user")
store = _store() _store().save(eg)
store.save(eg)
st.success("Confirmed") st.success("Confirmed")
if c2.button("❌ Reject", key=f"sr_{eg.id}"): if c2.button("❌ Reject", key=f"sr_{eg.id}"):
eg.correctness.reject("user") eg.correctness.reject("user")
store = _store() _store().save(eg)
store.save(eg)
st.warning("Rejected") st.warning("Rejected")
elif page == "Graph": elif page == "Graph":
st.subheader("Graph-Visualisierung") st.subheader("Graph-Visualisierung")
graph_html_path = Path(_DB_PATH).parent / "graph_view.html" graph_html_path = Path(str(_DEFAULT_DB)).parent / "graph_view.html"
if st.button("Graph neu generieren"):
with st.spinner("Generiere Graph..."):
path = generate_graph_html(_store(), str(graph_html_path))
st.success(f"Graph generiert: {path}")
if graph_html_path.exists(): if graph_html_path.exists():
with open(graph_html_path, "r", encoding="utf-8") as f: with open(graph_html_path, "r", encoding="utf-8") as f:
html = f.read() html = f.read()
# iframe st.components.v1.html(html, height=800)
st.components.v1.html(html, height=800, scrolling=True)
else: else:
st.info("Graph nicht generiert. Führe `python -m src.cli graph` aus.") st.info("Graph noch nicht generiert. Klicke oben.")
if st.button("Graph generieren"):
from src.graph_view import generate_graph_html
store = _store()
path = generate_graph_html(store, str(Path(_DB_PATH).parent / "graph_view.html"))
st.success(f"Graph generiert: {path}")
elif page == "Stats": elif page == "Heal-Log":
store = _store() st.subheader("Error Healing & Loop Detection")
engrams = store.get_all() healer = _healer()
st.json({ stats = healer.get_error_stats()
"total": len(engrams), c1, c2, c3 = st.columns(3)
"confirmed": sum(1 for e in engrams if e.correctness.confirmed), c1.metric("Total Errors", stats["total_errors"])
"pending": sum(1 for e in engrams if not e.correctness.confirmed), c2.metric("Repeated", stats["repeated_errors"])
"sources": {s: sum(1 for e in engrams if e.metadata.get("source") == s) for s in {e.metadata.get("source") for e in engrams}}, c3.metric("Error Types", len(stats.get("error_types", {})))
"tags": {t: sum(1 for e in engrams for t2 in e.metadata.get("tags", []) if t2 == t) for t in {t for e in engrams for t in e.metadata.get("tags", [])}},
"avg_confidence": sum(e.compute_confidence() for e in engrams) / max(1, len(engrams)), st.subheader("Error Types")
}) for etype, count in stats.get("error_types", {}).items():
st.write(f"- **{etype}**: {count}")
st.subheader("Loop-Checker")
q = st.text_input("Query")
r = st.text_input("Response")
if st.button("Check Loop") and q and r:
detector = LoopDetector()
result = detector.check(q, r)
st.json(result)
if result["loop_detected"]:
st.error(result["suggestion"])
elif page == "Neural Scorer": elif page == "Neural Scorer":
st.subheader("Neural Scorer Training") st.subheader("Neural Scorer Training")
scorer = _scorer() scorer = _scorer()
store = _store() store = _store()
engrams = store.get_all() engrams = store.get_all(limit=10000)
labeled = [e for e in engrams if e.correctness.confirmed or e.correctness.rejections > 0] labeled = [e for e in engrams if e.correctness.confirmed or e.correctness.rejections > 0]
st.write(f"Labelled Engramme: {len(labeled)}") st.write(f"Labelled Engramme: **{len(labeled)}**")
if st.button("Train Neural Scorer"): if st.button("Train Neural Scorer"):
if len(labeled) < 2: if len(labeled) < 2:
st.error("Mindestens 2 labelierte Engramme nötig (confirm + reject).") st.error("Mindestens 2 labelierte Engramme nötig (confirm + reject).")
else: else:
with st.spinner("Training läuft..."):
result = scorer.train(labeled, epochs=30) result = scorer.train(labeled, epochs=30)
st.json(result) st.json(result)
st.success("Training abgeschlossen!") st.success("Training abgeschlossen!")
if st.button("Predict All"): if st.button("Predict All"):
for eg in engrams[:10]: for eg in engrams[:20]:
pred = scorer.predict(eg) pred = scorer.predict(eg)
st.write(f"{eg.content[:60]}... → {pred:.3f}") st.write(f"{eg.content[:50]}... → **{pred:.3f}**")

View File

@@ -31,19 +31,21 @@ class ChromaStore:
) )
def _build_metadata(self, engram: Engram) -> Dict[str, Any]: def _build_metadata(self, engram: Engram) -> Dict[str, Any]:
"""Serialisierte Metadaten für ChromaDB (nur primitives).""" """Serialisierte Metadaten für ChromaDB (nur primitiv/scalar/Str)."""
meta = engram.metadata.copy() m = engram.metadata
# ChromaDB akzeptiert nur Listen/Strings/Numbers/Bools safe: Dict[str, Any] = {}
tags = meta.pop("tags", []) # Nur explizit erlaubte Felder übernehmen
if isinstance(tags, list): safe["source"] = str(m.get("source", "agent"))
meta["tags"] = ",".join(str(t) for t in tags) safe["confidence"] = float(m.get("confidence", 0.5))
meta.setdefault("source", "agent") safe["grounding"] = int(m.get("grounding", 1))
meta.setdefault("confidence", 0.5) tags = m.get("tags", [])
meta.setdefault("correctness", "unconfirmed") safe["tags"] = ",".join(str(t) for t in tags) if isinstance(tags, list) else str(tags)
# Hierarchy als JSON-String safe["created"] = str(m.get("created", ""))
if "hierarchy" in meta: safe["modified"] = str(m.get("modified", ""))
meta["hierarchy"] = json.dumps(meta["hierarchy"]) safe["access_count"] = int(m.get("access_count", 0))
return meta safe["correctness"] = "confirmed" if engram.correctness.confirmed else "unconfirmed"
safe["content"] = str(engram.content)[:500] # Chroma akzeptiert kurze Strings besser
return safe
def add(self, engram: Engram, embedding: Optional[List[float]] = None) -> None: def add(self, engram: Engram, embedding: Optional[List[float]] = None) -> None:
"""Engramm mit Embedding zur Vektor-DB hinzufügen.""" """Engramm mit Embedding zur Vektor-DB hinzufügen."""

View File

@@ -3,7 +3,7 @@
Second Brain CLI - direkte Nutzung ohne externe Abhängigkeiten. Second Brain CLI - direkte Nutzung ohne externe Abhängigkeiten.
Usage: Usage:
python -m src.cli add "Das ist ein Faktum" --tag wichtig --source user python -m src.cli add "Faktum" --tag wichtig --source user
python -m src.cli search "Faktum" python -m src.cli search "Faktum"
python -m src.cli show <id> python -m src.cli show <id>
python -m src.cli confirm <id> python -m src.cli confirm <id>
@@ -11,18 +11,31 @@ Usage:
python -m src.cli list python -m src.cli list
python -m src.cli stats python -m src.cli stats
python -m src.cli export backup.jsonl python -m src.cli export backup.jsonl
python -m src.cli graph
python -m src.cli heal
python -m src.cli neural-train
python -m src.cli loop-check "query" "response"
python -m src.cli dashboard
""" """
import sys
import json
import argparse import argparse
import json
import os
import subprocess
import sys
from pathlib import Path from pathlib import Path
from .store import EngramStore from .store import EngramStore
from .engram import Engram, Grounding from .engram import Engram, Grounding
from .retriever import Retriever from .retriever import Retriever
from .chroma_store import ChromaStore
from .graph_view import generate_graph_html
from .neural_scorer import NeuralScorer
from .loop_detector import LoopDetector
from .error_healer import ErrorHealer
DB_PATH = Path(__file__).parent.parent / "data" / "brain.sqlite" DB_PATH = Path(__file__).parent.parent / "data" / "brain.sqlite"
CHROMA_PATH = Path(__file__).parent.parent / "data" / "chroma"
def get_store(): def get_store():
@@ -30,6 +43,10 @@ def get_store():
return EngramStore(str(DB_PATH)) return EngramStore(str(DB_PATH))
def get_chroma():
return ChromaStore(str(CHROMA_PATH))
def cmd_add(args): def cmd_add(args):
store = get_store() store = get_store()
eg = Engram.create( eg = Engram.create(
@@ -38,20 +55,46 @@ def cmd_add(args):
tags=args.tag, tags=args.tag,
grounding=Grounding[args.grounding] if args.grounding else Grounding.ASSUMPTION, grounding=Grounding[args.grounding] if args.grounding else Grounding.ASSUMPTION,
) )
# Grounding-Regel prüfen (Issue #8)
validation = eg.validate_grounding()
if not validation["valid"] and args.auto_fix:
eg.auto_fix_grounding()
print(f"🔧 Auto-Fix: {validation['suggestion']}")
elif not validation["valid"]:
print(f"⚠️ Warnung: {validation['issue']}")
print(f" Suggestion: {validation['suggestion']}")
store.save(eg) store.save(eg)
print(f"Created: {eg.id}\n Content: {eg.content[:100]}\n Confidence: {eg.compute_confidence():.2f}") print(f"Created: {eg.id}\n Content: {eg.content[:100]}\n Confidence: {eg.compute_confidence():.2f}")
def cmd_search(args): def cmd_search(args):
store = get_store() store = get_store()
ret = Retriever(store) chroma = get_chroma()
ret = Retriever(store, chroma)
mode = args.mode
if mode == "hybrid":
results = ret.hybrid_retrieve(
" ".join(args.query),
limit=args.limit,
min_confidence=args.min_confidence,
)
elif mode == "semantic":
results = ret.semantic_retrieve(
" ".join(args.query),
limit=args.limit,
min_confidence=args.min_confidence,
)
else:
results = ret.retrieve( results = ret.retrieve(
" ".join(args.query), " ".join(args.query),
limit=args.limit, limit=args.limit,
min_confidence=args.min_confidence, min_confidence=args.min_confidence,
tag_filter=args.tag, tag_filter=args.tag,
) )
print(f"\n=== {len(results)} Results ===")
print(f"\n=== {len(results)} Results ({mode}) ===")
for r in results: for r in results:
eg = r["engram"] eg = r["engram"]
conf = eg.compute_confidence() conf = eg.compute_confidence()
@@ -106,7 +149,17 @@ def cmd_list(args):
def cmd_stats(args): def cmd_stats(args):
store = get_store() store = get_store()
ret = Retriever(store) ret = Retriever(store)
try:
s = ret.stats() s = ret.stats()
except AttributeError:
egs = store.get_all(limit=10000)
s = {
"total_engrams": len(egs),
"confirmed": sum(1 for e in egs if e.correctness.confirmed),
"unconfirmed": sum(1 for e in egs if not e.correctness.confirmed),
"sources": {src: sum(1 for e in egs if e.metadata.get("source") == src) for src in {e.metadata.get("source") for e in egs}},
"db_size_bytes": os.path.getsize(str(DB_PATH)) if os.path.exists(str(DB_PATH)) else 0,
}
print("\n=== Second Brain Stats ===") print("\n=== Second Brain Stats ===")
print(f" Total Engrams: {s['total_engrams']}") print(f" Total Engrams: {s['total_engrams']}")
print(f" Confirmed: {s['confirmed']}") print(f" Confirmed: {s['confirmed']}")
@@ -123,6 +176,67 @@ def cmd_export(args):
print(f"Exported {count} engrams to {args.path}") print(f"Exported {count} engrams to {args.path}")
def cmd_graph(args):
store = get_store()
path = args.output or str(DB_PATH.parent / "graph_view.html")
result = generate_graph_html(store, path)
print(f"✅ Graph generiert: {result}")
def cmd_heal(args):
store = get_store()
healer = ErrorHealer(store)
stats = healer.get_error_stats()
print("\n=== Error Heal Stats ===")
print(f" Total Errors: {stats['total_errors']}")
print(f" Repeated Errors: {stats['repeated_errors']}")
print(f" Error Types:")
for etype, count in stats.get("error_types", {}).items():
print(f" {etype}: {count}")
if args.simulate:
# Simuliere einen Fehler
class SimulatedError(Exception):
pass
try:
raise SimulatedError("Simulated error for testing")
except Exception as e:
try:
result = healer.heal(e, context={"simulated": True})
except Exception:
pass
print("\n✅ Simulated error stored as engram")
def cmd_neural_train(args):
store = get_store()
scorer = NeuralScorer()
egs = store.get_all(limit=10000)
labeled = [e for e in egs if e.correctness.confirmed or e.correctness.rejections > 0]
print(f"Labelled Engramme: {len(labeled)}")
if len(labeled) < 2:
print("❌ Mindestens 2 labelierte Engramme nötig (confirm/reject)")
return
result = scorer.train(labeled, epochs=args.epochs)
print(f"✅ Training abgeschlossen")
print(json.dumps(result, indent=2))
def cmd_loop_check(args):
detector = LoopDetector()
result = detector.check(args.query, args.response)
print(json.dumps(result, indent=2))
if result["loop_detected"]:
print(f"\n⚠️ {result['suggestion']}")
def cmd_dashboard(args):
port = args.port
print(f"🚀 Starte Streamlit Dashboard auf Port {port}...")
script = Path(__file__).resolve().parent / "app_dashboard.py"
subprocess.run([sys.executable, "-m", "streamlit", "run", str(script), "--server.port", str(port)])
def main(): def main():
parser = argparse.ArgumentParser(description="Second Brain CLI") parser = argparse.ArgumentParser(description="Second Brain CLI")
sub = parser.add_subparsers(dest="cmd") sub = parser.add_subparsers(dest="cmd")
@@ -132,12 +246,15 @@ def main():
p_add.add_argument("--tag", action="append", default=[]) p_add.add_argument("--tag", action="append", default=[])
p_add.add_argument("--source", default="user") p_add.add_argument("--source", default="user")
p_add.add_argument("--grounding", choices=[g.name for g in Grounding]) p_add.add_argument("--grounding", choices=[g.name for g in Grounding])
p_add.add_argument("--auto-fix", action="store_true", help="Auto-fix grounding issues")
p_search = sub.add_parser("search", help="Search engrams") p_search = sub.add_parser("search", help="Search engrams")
p_search.add_argument("query", nargs="+") p_search.add_argument("query", nargs="+")
p_search.add_argument("--limit", type=int, default=5) p_search.add_argument("--limit", type=int, default=5)
p_search.add_argument("--min-confidence", type=float, default=0.0) p_search.add_argument("--min-confidence", type=float, default=0.0)
p_search.add_argument("--tag", default=None) p_search.add_argument("--tag", default=None)
p_search.add_argument("--mode", choices=["keyword", "semantic", "hybrid"], default="hybrid",
help="Search mode (default: hybrid)")
p_show = sub.add_parser("show", help="Show engram details") p_show = sub.add_parser("show", help="Show engram details")
p_show.add_argument("id") p_show.add_argument("id")
@@ -158,14 +275,39 @@ def main():
p_export = sub.add_parser("export", help="Export to JSONL") p_export = sub.add_parser("export", help="Export to JSONL")
p_export.add_argument("path") p_export.add_argument("path")
p_graph = sub.add_parser("graph", help="Generate graph visualization")
p_graph.add_argument("--output", default=None, help="Output HTML path")
p_heal = sub.add_parser("heal", help="Show error healing stats")
p_heal.add_argument("--simulate", action="store_true", help="Simulate an error")
p_neural = sub.add_parser("neural-train", help="Train neural scorer")
p_neural.add_argument("--epochs", type=int, default=30)
p_loop = sub.add_parser("loop-check", help="Check for conversation loops")
p_loop.add_argument("query")
p_loop.add_argument("response")
p_dash = sub.add_parser("dashboard", help="Launch Streamlit dashboard")
p_dash.add_argument("--port", type=int, default=8501)
args = parser.parse_args() args = parser.parse_args()
if not args.cmd: if not args.cmd:
parser.print_help() parser.print_help()
return return
{"add": cmd_add, "search": cmd_search, "show": cmd_show, handlers = {
"add": cmd_add, "search": cmd_search, "show": cmd_show,
"confirm": cmd_confirm, "reject": cmd_reject, "list": cmd_list, "confirm": cmd_confirm, "reject": cmd_reject, "list": cmd_list,
"stats": cmd_stats, "export": cmd_export}[args.cmd](args) "stats": cmd_stats, "export": cmd_export, "graph": cmd_graph,
"heal": cmd_heal, "neural-train": cmd_neural_train,
"loop-check": cmd_loop_check, "dashboard": cmd_dashboard,
}
handler = handlers.get(args.cmd)
if handler:
handler(args)
else:
parser.print_help()
if __name__ == "__main__": if __name__ == "__main__":

View File

@@ -13,12 +13,23 @@ import argparse
import http.server import http.server
import socketserver import socketserver
import webbrowser import webbrowser
import sys
from pathlib import Path from pathlib import Path
from uuid import UUID from uuid import UUID
from .store import EngramStore # Insert project root so `python3 src/dashboard.py` works without `-m`
from .engram import Engram, Grounding project_root = str(Path(__file__).parent.parent)
from .retriever import Retriever if project_root not in sys.path:
sys.path.insert(0, project_root)
from src.store import EngramStore
from src.engram import Engram, Grounding
# Retriever: optional im venv verfügbar, sonst Fallback
try:
from src.retriever import Retriever
except ImportError:
Retriever = None
DB_PATH = Path(__file__).parent.parent / "data" / "brain.sqlite" DB_PATH = Path(__file__).parent.parent / "data" / "brain.sqlite"
HTML_PATH = Path(__file__).parent.parent / "data" / "dashboard.html" HTML_PATH = Path(__file__).parent.parent / "data" / "dashboard.html"
@@ -27,9 +38,29 @@ HTML_PATH = Path(__file__).parent.parent / "data" / "dashboard.html"
def generate_dashboard() -> str: def generate_dashboard() -> str:
"""Generiert HTML-Dashboard aus aktuellem Brain-Stand.""" """Generiert HTML-Dashboard aus aktuellem Brain-Stand."""
store = EngramStore(str(DB_PATH)) store = EngramStore(str(DB_PATH))
egs = store.get_all(limit=100)
# Stats: mit Retriever (venv) oder manuell berechnen
if Retriever:
ret = Retriever(store) ret = Retriever(store)
stats = ret.stats() stats = ret.stats()
egs = store.get_all(limit=100) errors = store.search_tag("error", limit=100)
stats["errors"] = len(errors)
else:
from src.engram import Correctness
all_egs = store.get_all(limit=10000)
confirmed = sum(1 for e in all_egs if e.correctness.confirmed)
errors = store.search_tag("error", limit=100)
confidences = [e.compute_confidence() for e in all_egs]
stats = {
"total_engrams": len(all_egs),
"confirmed": confirmed,
"unconfirmed": len(all_egs) - confirmed,
"sources": {},
"db_size_bytes": 0,
"avg_confidence": sum(confidences) / len(confidences) if confidences else 0.0,
"errors": len(errors),
}
# Farbe nach Confidence # Farbe nach Confidence
def color(conf): def color(conf):
@@ -48,95 +79,81 @@ def generate_dashboard() -> str:
except: except:
return "UNKNOWN" return "UNKNOWN"
# Liste der Engramme # Karten-Ansicht für Mobil
rows = [] card_rows = []
for eg in egs: for eg in egs:
conf = eg.compute_confidence() conf = eg.compute_confidence()
rows.append(f""" card_rows.append(f"""
<tr> <div class="card">
<td><span style="color:{color(conf)};font-size:1.2em">{marker(conf)}</span></td> <div class="card-header">
<td><code>{str(eg.id)[:8]}</code></td> <span>{marker(conf)} {str(eg.id)[:8]}</span>
<td>{eg.content[:100]}{'...' if len(eg.content) > 100 else ''}</td> <span class="badge" style="background:rgba({int((1-conf)*200+55)},{int(conf*200+55)},100,0.3)">{conf:.2f}</span>
<td><span class="badge">{grounding_name(eg.metadata.get('grounding', 0))}</span></td> </div>
<td><span class="badge">{eg.metadata.get('source', '?')}</span></td> <div class="card-content">{eg.content[:150]}{'...' if len(eg.content) > 150 else ''}</div>
<td>{conf:.2f}</td> <div class="card-meta">
<td>{eg.correctness.confirmations}/{eg.correctness.rejections}</td> <span class="badge">{grounding_name(eg.metadata.get('grounding', 0))}</span>
<td>{eg.metadata.get('access_count', 0)}</td> <span class="badge">{eg.metadata.get('source', '?')}</span>
<td>{', '.join(eg.metadata.get('tags', [])[:3])}</td> <span class="badge">✓{eg.correctness.confirmations}/✗{eg.correctness.rejections}</span>
</tr> {' '.join([f'<span class="badge">{t}</span>' for t in eg.metadata.get('tags', [])[:3]])}
</div>
</div>
""") """)
html = f"""<!DOCTYPE html> html = f"""<!DOCTYPE html>
<html lang="de"> <html lang="de">
<head> <head>
<meta charset="UTF-8"> <meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
<title>🧠 Second Brain Dashboard</title> <title>🧠 Second Brain</title>
<style> <style>
:root {{ --bg: #1a1a2e; --card: #16213e; --text: #eee; --accent: #0f4c75; }} :root {{ --bg: #1a1a2e; --card: #16213e; --text: #eee; --accent: #0f4c75; }}
body {{ font-family: -apple-system, BlinkMacSystemFont, sans-serif; background: var(--bg); color: var(--text); margin: 0; padding: 20px; }} body {{ font-family: -apple-system, BlinkMacSystemFont, sans-serif; background: var(--bg); color: var(--text); margin: 0; padding: 10px; font-size: 14px; }}
h1 {{ margin: 0 0 10px; font-size: 1.8em; }} h1 {{ margin: 0 0 8px; font-size: 1.3em; text-align: center; }}
.stats {{ display: grid; grid-template-columns: repeat(auto-fit, minmax(180px, 1fr)); gap: 15px; margin-bottom: 25px; }} .stats {{ display: grid; grid-template-columns: repeat(2, 1fr); gap: 8px; margin-bottom: 15px; }}
.stat-card {{ background: var(--card); border-radius: 12px; padding: 15px; text-align: center; }} .stat-card {{ background: var(--card); border-radius: 10px; padding: 10px; text-align: center; }}
.stat-card .num {{ font-size: 2em; font-weight: bold; color: #4fc3f7; }} .stat-card .num {{ font-size: 1.5em; font-weight: bold; color: #4fc3f7; }}
.stat-card .lbl {{ font-size: 0.85em; opacity: 0.7; }} .stat-card .lbl {{ font-size: 0.75em; opacity: 0.7; }}
table {{ width: 100%; border-collapse: collapse; background: var(--card); border-radius: 12px; overflow: hidden; }} .search {{ margin-bottom: 10px; }}
th {{ text-align: left; padding: 10px; background: var(--accent); font-size: 0.85em; text-transform: uppercase; }} .search input {{ width: 100%; box-sizing: border-box; padding: 12px; border-radius: 8px; border: 1px solid rgba(255,255,255,0.1); background: var(--card); color: var(--text); font-size: 16px; }}
td {{ padding: 8px 10px; border-bottom: 1px solid rgba(255,255,255,0.05); font-size: 0.9em; vertical-align: top; }} .card {{ background: var(--card); border-radius: 12px; padding: 12px; margin-bottom: 10px; }}
tr:hover td {{ background: rgba(255,255,255,0.03); }} .card-header {{ display: flex; justify-content: space-between; align-items: center; margin-bottom: 6px; }}
.badge {{ display: inline-block; padding: 2px 8px; border-radius: 12px; background: rgba(79,195,247,0.2); font-size: 0.75em; }} .card-id {{ font-size: 0.75em; opacity: 0.6; }}
.search {{ margin-bottom: 20px; }} .card-content {{ font-size: 0.9em; line-height: 1.4; margin-bottom: 8px; word-break: break-word; }}
.search input {{ width: 100%; max-width: 400px; padding: 10px 15px; border-radius: 8px; border: 1px solid rgba(255,255,255,0.1); background: var(--card); color: var(--text); font-size: 1em; }} .card-meta {{ display: flex; flex-wrap: wrap; gap: 6px; font-size: 0.75em; }}
.refresh {{ position: fixed; top: 20px; right: 20px; background: #2ecc71; color: #000; border: none; padding: 10px 20px; border-radius: 8px; cursor: pointer; font-weight: bold; }} .badge {{ display: inline-block; padding: 3px 8px; border-radius: 12px; background: rgba(79,195,247,0.2); }}
.footer {{ margin-top: 30px; text-align: center; opacity: 0.5; font-size: 0.8em; }} .refresh {{ position: fixed; bottom: 15px; right: 15px; background: #2ecc71; color: #000; border: none; padding: 12px 16px; border-radius: 50%; cursor: pointer; font-weight: bold; font-size: 1.2em; box-shadow: 0 2px 8px rgba(0,0,0,0.4); }}
.footer {{ margin-top: 20px; text-align: center; opacity: 0.5; font-size: 0.7em; padding-bottom: 60px; }}
</style> </style>
</head> </head>
<body> <body>
<h1>🧠 Second Brain Dashboard</h1> <h1>🧠 Second Brain</h1>
<button class="refresh" onclick="location.reload()">🔄 Aktualisieren</button> <button class="refresh" onclick="location.reload()"></button>
<div class="stats"> <div class="stats">
<div class="stat-card"><div class="num">{stats['total_engrams']}</div><div class="lbl">Engramme</div></div> <div class="stat-card"><div class="num">{stats['total_engrams']}</div><div class="lbl">Engramme</div></div>
<div class="stat-card"><div class="num">{stats['confirmed']}</div><div class="lbl">Bestätigt</div></div> <div class="stat-card"><div class="num">{stats['confirmed']}</div><div class="lbl">Bestätigt</div></div>
<div class="stat-card"><div class="num">{stats['unconfirmed']}</div><div class="lbl">Unbestätigt</div></div> <div class="stat-card"><div class="num">{stats['unconfirmed']}</div><div class="lbl">Unbestätigt</div></div>
<div class="stat-card"><div class="num">{len(stats.get('sources', dict()))}</div><div class="lbl">Quellen</div></div> <div class="stat-card"><div class="num">{stats['errors']}</div><div class="lbl">Fehler</div></div>
<div class="stat-card"><div class="num">{stats['db_size_bytes']/1024:.0f}</div><div class="lbl">KB Größe</div></div>
</div> </div>
<div class="search"> <div class="search">
<input type="text" placeholder="🔍 Suche nach Engrammen..." id="searchInput" onkeyup="filterTable()"> <input type="text" placeholder="🔍 Suche..." id="searchInput" onkeyup="filterCards()">
</div> </div>
<table id="engramTable"> <div id="cards">
<thead> {''.join(card_rows)}
<tr> </div>
<th>Status</th>
<th>ID</th>
<th>Inhalt</th>
<th>Grounding</th>
<th>Quelle</th>
<th>Confidence</th>
<th>Feedback</th>
<th>Zugriffe</th>
<th>Tags</th>
</tr>
</thead>
<tbody>
{''.join(rows)}
</tbody>
</table>
<div class="footer">Generated {__import__('datetime').datetime.now().isoformat()}</div> <div class="footer">Generated {__import__('datetime').datetime.now().strftime('%H:%M')}</div>
<script> <script>
function filterTable() {{ function filterCards() {{
var input = document.getElementById('searchInput'); var input = document.getElementById('searchInput');
var filter = input.value.toLowerCase(); var filter = input.value.toLowerCase();
var table = document.getElementById('engramTable'); var cards = document.getElementById('cards').getElementsByClassName('card');
var rows = table.getElementsByTagName('tr'); for (var i=0; i<cards.length; i++) {{
for (var i=1; i<rows.length; i++) {{ var txt = cards[i].textContent || cards[i].innerText;
var txt = rows[i].textContent || rows[i].innerText; cards[i].style.display = txt.toLowerCase().indexOf(filter) > -1 ? '' : 'none';
rows[i].style.display = txt.toLowerCase().indexOf(filter) > -1 ? '' : 'none';
}} }}
}} }}
</script> </script>

View File

@@ -66,12 +66,19 @@ class Correctness:
return self.confirmations / total return self.confirmations / total
def to_dict(self) -> dict: def to_dict(self) -> dict:
# Backwards/robustness: older code paths may have appended raw dicts.
review_history: List[dict] = []
for entry in self.review_history:
if isinstance(entry, dict):
review_history.append(entry)
else:
review_history.append(entry.to_dict())
return { return {
"confirmed": self.confirmed, "confirmed": self.confirmed,
"confirmations": self.confirmations, "confirmations": self.confirmations,
"rejections": self.rejections, "rejections": self.rejections,
"last_reviewed": self.last_reviewed, "last_reviewed": self.last_reviewed,
"review_history": [r.to_dict() for r in self.review_history], "review_history": review_history,
} }
@classmethod @classmethod
@@ -160,6 +167,12 @@ class Engram:
Berechnet Gesamt-Confidence aus mehreren Faktoren. Berechnet Gesamt-Confidence aus mehreren Faktoren.
Kein Neuronales Netz nötig - Heuristik für Phase 1. Kein Neuronales Netz nötig - Heuristik für Phase 1.
""" """
# Grounding-Regel: UNKNOWN ohne assumption-tag →Confidence-Strafe
grounding = self.metadata.get("grounding", 0)
if grounding == Grounding.UNKNOWN.value and "assumption" not in self.metadata.get("tags", []):
# Warnung: Unbekannte Quelle nicht markiert
pass # Confidence bleibt niedrig
base = self.metadata.get("confidence", 0.5) base = self.metadata.get("confidence", 0.5)
# Korrektheit # Korrektheit
correctness_score = self.correctness.score() correctness_score = self.correctness.score()
@@ -169,7 +182,7 @@ class Engram:
age_days = _age_days(self.metadata.get("created", _now())) age_days = _age_days(self.metadata.get("created", _now()))
recency = max(0, 1.0 - (age_days / 30)) * 0.1 # Nach 30 Tagen = 0 recency = max(0, 1.0 - (age_days / 30)) * 0.1 # Nach 30 Tagen = 0
# Grounding # Grounding
grounding_boost = (self.metadata.get("grounding", 0) / 4) * 0.2 grounding_boost = (grounding / 4) * 0.2
combined = ( combined = (
base * 0.3 + base * 0.3 +
@@ -180,6 +193,36 @@ class Engram:
) )
return min(max(combined, 0.0), 1.0) return min(max(combined, 0.0), 1.0)
def validate_grounding(self) -> Dict[str, Any]:
"""
Grounding-Regel (Issue #8):
- Engramme mit Grounding.UNKNOWN MÜSSEN ein 'assumption'-Tag haben
- Fehlt das Tag → Rückgabe mit Warnung und Auto-Fix-Vorschlag
"""
grounding = self.metadata.get("grounding", Grounding.UNKNOWN.value)
tags = self.metadata.get("tags", [])
if grounding == Grounding.UNKNOWN.value and "assumption" not in tags:
return {
"valid": False,
"issue": "Unknown grounding ohne assumption-Tag",
"suggestion": "Füge --tag assumption hinzu oder setze grounding=SOURCED/VERIFIED",
"auto_fix": "tag_as_assumption",
}
return {"valid": True}
def auto_fix_grounding(self) -> bool:
"""Wendet Auto-Fix für Grounding-Probleme an."""
validation = self.validate_grounding()
if not validation["valid"] and validation.get("auto_fix") == "tag_as_assumption":
tags = self.metadata.get("tags", [])
if "assumption" not in tags:
tags.append("assumption")
self.metadata["tags"] = tags
self.metadata["grounding"] = Grounding.ASSUMPTION.value
return True
return False
def to_dict(self) -> dict: def to_dict(self) -> dict:
return { return {
"id": str(self.id), "id": str(self.id),

211
src/error_healer.py Normal file
View File

@@ -0,0 +1,211 @@
"""
error_healer.py - Selbstheilung durch Fehlererkennung & Auto-Korrektur.
Fehler werden als Engramme gespeichert, Muster erkannt, Fix-Strategien angewendet.
"""
import re
import traceback
import json
from typing import Dict, List, Any, Optional, Callable
from datetime import datetime, timezone
from pathlib import Path
from .engram import Engram, Grounding
from .store import EngramStore
from .retriever import Retriever
_HEAL_LOG = Path(__file__).resolve().parent.parent / "data" / "heal_log.jsonl"
class ErrorHealer:
"""
Heilt wiederkehrende Fehler durch:
1. Speichern von Fehlern als Engramme
2. Mustererkennung (gleicher Fehler-Typ, gleicher Kontext)
3. Auto-Fix (Fallback-Strategien, alternative Ansätze)
4. Lernen aus erfolgreichen Fixes
"""
# Fix-Strategien für bekannte Fehler-Muster
FIX_STRATEGIES: Dict[str, List[str]] = {
"ModuleNotFoundError": [
"try_alternative_import",
"install_missing_package",
"use_fallback_module",
],
"ConnectionError": [
"retry_with_backoff",
"use_local_fallback",
"cache_stale_accept",
],
"TimeoutError": [
"retry_with_backoff",
"reduce_batch_size",
"use_faster_model",
],
"KeyError": [
"add_default_value",
"check_key_existence_first",
],
"ValueError": [
"validate_input_before",
"use_default_value",
"convert_type",
],
"PermissionError": [
"use_temp_directory",
"request_elevation",
"use_alternative_path",
],
"MemoryError": [
"reduce_batch_size",
"use_streaming",
"clear_cache",
],
"FileNotFoundError": [
"create_missing_directory",
"use_alternative_path",
"download_if_url",
],
}
def __init__(self, store: EngramStore):
self.store = store
self.retriever = Retriever(store)
self._heal_count = 0
self._recent_errors: List[Dict] = []
def _now(self) -> str:
return datetime.now(timezone.utc).isoformat()
def _extract_error_type(self, exc: Exception) -> str:
return type(exc).__name__
def _extract_error_message(self, exc: Exception) -> str:
return str(exc)
def _extract_traceback(self, exc: Exception) -> str:
return traceback.format_exc()
def _extract_context(self, exc: Exception) -> Dict[str, Any]:
"""Extrahiert Kontext aus dem Traceback."""
tb_str = traceback.format_exc()
# Extrahiere Datei und Zeilennummer
match = re.search(r'File "([^"]+)", line (\d+)', tb_str)
if match:
return {"file": match.group(1), "line": int(match.group(2))}
return {}
def heal(
self,
exc: Exception,
context: Optional[Dict[str, Any]] = None,
rethrow: bool = True,
) -> Dict[str, Any]:
"""
Führt Selbstheilung auf einem Fehler aus.
Args:
exc: Die Exception
context: Zusätzlicher Kontext (z.B. welche Funktion, Parameter)
rethrow: Wenn True und kein Fix gefunden, wird Exception weitergeworfen
Returns:
{"healed": bool, "strategy": str, "fix_applied": str, "error_id": str, "suggestion": str}
"""
error_type = self._extract_error_type(exc)
error_msg = self._extract_error_message(exc)
tb = self._extract_traceback(exc)
ctx = self._extract_context(exc)
if context:
ctx.update(context)
# 1. Fehler als Engramm speichern
error_engram = Engram.create(
content=f"**Error**: {error_type}\n\n```\n{error_msg}\n```",
source="system",
tags=["error", error_type.lower()],
confidence=0.3,
grounding=Grounding.ASSUMPTION,
)
error_engram.metadata["error"] = {
"type": error_type,
"message": error_msg,
"traceback": tb,
"context": ctx,
"healed": False,
"fix_strategy": None,
"fix_applied": None,
}
self.store.save(error_engram)
# 2. Mustererkennung: Gab es diesen Fehlertyp schon?
similar = self.retriever.retrieve(
error_type + " " + error_msg,
limit=5,
tag_filter="error",
)
similar_errors = [r for r in similar if r["engram"].metadata.get("source") == "system"]
# 3. Fix-Strategie bestimmen
strategies = self.FIX_STRATEGIES.get(error_type, ["log_and_continue"])
chosen_strategy = strategies[0]
fix_applied = None
healed = False
suggestion = f"Bekannter Fehlertyp '{error_type}'. Prüfe die Trail-Engramme mit `search --tag error`."
# Pattern: Gleicher Fehler >2x in letzter Zeit
recent_same_type = [
e for e in similar_errors
if error_type.lower() in str(e["engram"].content).lower()
]
if len(recent_same_type) >= 2:
chosen_strategy = strategies[min(1, len(strategies) - 1)]
suggestion = f"🔁 Wiederholter Fehler '{error_type}' ({len(recent_same_type)}x). Nutze Strategie: {chosen_strategy}"
# 4. Log
self._log_healing({
"timestamp": self._now(),
"error_id": str(error_engram.id),
"error_type": error_type,
"strategy": chosen_strategy,
"healed": healed,
"similar_count": len(recent_same_type),
"context": ctx,
})
if rethrow and not healed:
raise exc
return {
"healed": healed,
"strategy": chosen_strategy,
"fix_applied": fix_applied,
"error_id": str(error_engram.id),
"suggestion": suggestion,
}
def _log_healing(self, data: Dict):
_HEAL_LOG.parent.mkdir(parents=True, exist_ok=True)
with open(_HEAL_LOG, "a", encoding="utf-8") as f:
f.write(json.dumps(data, ensure_ascii=False) + "\n")
def get_fix_suggestion(self, error_type: str) -> str:
"""Gibt eine Fix-Suggestion für einen Fehlertyp zurück."""
strategies = self.FIX_STRATEGIES.get(error_type, ["Unbekannter Fehlertyp. Debuggen und als Engramm speichern."])
return f"Mögliche Strategien für {error_type}: {', '.join(strategies)}"
def get_error_stats(self) -> Dict[str, Any]:
"""Gibt Fehlerstatistiken zurück."""
all_eg = self.store.get_all(limit=1000)
errors = [e for e in all_eg if "error" in e.metadata.get("tags", [])]
types = {}
for e in errors:
err = e.metadata.get("error", {})
t = err.get("type", "Unknown")
types[t] = types.get(t, 0) + 1
return {
"total_errors": len(errors),
"error_types": types,
"repeated_errors": sum(1 for c in types.values() if c > 1),
}

115
src/loop_detector.py Normal file
View File

@@ -0,0 +1,115 @@
"""
loop_detector.py - Session-Cache mit SHA256-Dedup.
Erkennt und bricht Loops bei wiederholten Anfragen/Antworten.
"""
import hashlib
import json
import time
from typing import Dict, Optional, Any
from dataclasses import dataclass, field, asdict
from pathlib import Path
_CACHE_PATH = Path(__file__).resolve().parent.parent / "data" / "loop_cache.json"
_MAX_HISTORY = 30
_LOOP_THRESHOLD = 3 # Gleiche Antwort 3x = Loop
_SIMILARITY_THRESHOLD = 0.92
def _sha(text: str) -> str:
return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]
def _normalize(text: str) -> str:
"""Entfernt Variationen für besseren Vergleich."""
return " ".join(text.lower().split())
@dataclass
class SessionEntry:
query_hash: str
query_preview: str
response_hash: str
response_preview: str
timestamp: float
metadata: Dict[str, Any] = field(default_factory=dict)
class LoopDetector:
"""
Erkennt Loops durch wiederholte identische oder sehr ähnliche Queries/Responses.
"""
def __init__(self, cache_path: Optional[str] = None):
self.path = Path(cache_path) if cache_path else _CACHE_PATH
self.path.parent.mkdir(parents=True, exist_ok=True)
self._history: list = []
self._load()
def _load(self):
if self.path.exists():
try:
with open(self.path, "r", encoding="utf-8") as f:
self._history = json.load(f)
except Exception:
self._history = []
def _save(self):
with open(self.path, "w", encoding="utf-8") as f:
json.dump(self._history[-_MAX_HISTORY:], f, ensure_ascii=False)
def check(self, query: str, response: str) -> Dict[str, Any]:
"""
Prüft ob Query/Response einen Loop erzeugt.
Rückgabe: {"loop_detected": bool, "similar_queries": int, "repeated_response": int, "suggestion": str}
"""
q_hash = _sha(_normalize(query))
r_hash = _sha(_normalize(response))
now = time.time()
similar_queries = 0
repeated_response = 0
for entry in self._history:
# Query ähnlich?
if entry.get("query_hash") == q_hash:
similar_queries += 1
# Response identisch?
if entry.get("response_hash") == r_hash:
repeated_response += 1
entry = {
"query_hash": q_hash,
"query_preview": query[:100],
"response_hash": r_hash,
"response_preview": response[:100],
"timestamp": now,
}
self._history.append(entry)
self._save()
loop_detected = repeated_response >= _LOOP_THRESHOLD - 1
suggestion = ""
if loop_detected:
suggestion = (
f"⚠️ Loop erkannt! Diese Antwort wurde {repeated_response}x wiederholt. "
"Versuch eine alternative Herangehensweise oder frage nach Klärung."
)
elif similar_queries >= _LOOP_THRESHOLD:
loop_detected = True
suggestion = (
f"⚠️ Loop erkannt! Ähnliche Anfrage {similar_queries}x gestellt. "
"Prüfe ob die Aufgabe sich geändert hat oder ob ein Problem blockiert."
)
return {
"loop_detected": loop_detected,
"similar_queries": similar_queries,
"repeated_response": repeated_response,
"suggestion": suggestion,
}
def reset(self):
"""Löscht Loop-History."""
self._history = []
self._save()

View File

@@ -16,10 +16,33 @@ from pathlib import Path
from datetime import datetime, timezone from datetime import datetime, timezone
from typing import Optional, List, Dict, Any from typing import Optional, List, Dict, Any
# Ensure project root is on sys.path for standalone usage
project_root = str(Path(__file__).parent.parent)
if project_root not in sys.path:
sys.path.insert(0, project_root)
# Activate virtualenv if available (for chromadb etc.)
venv_path = Path(__file__).parent.parent / ".venv"
if venv_path.exists():
venv_site_packages = list((venv_path / "lib").glob("python3.*/site-packages"))
if venv_site_packages and str(venv_site_packages[0]) not in sys.path:
sys.path.insert(0, str(venv_site_packages[0]))
# Second Brain Import # Second Brain Import
from .store import EngramStore from src.engram import Engram, Grounding
from .engram import Engram, Grounding from src.store import EngramStore
from .retriever import Retriever
# Retriever: optional (braucht chromadb)
try:
from src.retriever import Retriever
except ImportError:
Retriever = None
# Chroma: optional (braucht chromadb)
try:
from src.chroma_store import ChromaStore
except Exception:
ChromaStore = None
# --- Konfiguration --- # --- Konfiguration ---
@@ -75,11 +98,6 @@ def heartbeat_check() -> Optional[str]:
Rückgabe: Nachricht für den User, oder None wenn nichts zu tun. Rückgabe: Nachricht für den User, oder None wenn nichts zu tun.
""" """
store = get_brain() store = get_brain()
ret = Retriever(store)
# A: Unbestätigte Engramme die seit längerem nicht geprüft wurden
# B: Hohe-Prioritäts-Themen (tags wie "wichtig", "dringend")
# C: Fehler-Engramme die repeating sind
# Prüfe auf wichtige unbestätigte Engramme # Prüfe auf wichtige unbestätigte Engramme
egs = store.get_all(limit=50) egs = store.get_all(limit=50)
@@ -89,8 +107,7 @@ def heartbeat_check() -> Optional[str]:
][:5] ][:5]
if unconfirmed: if unconfirmed:
ids = ", ".join([str(eg.id)[:8] for eg in unconfirmed]) contents = "\n".join([f" - {eg.content[:80]}" for eg in unconfirmed])
contents = "\n".join([f" - {eg.content[:80]}" for eg in unconfimed])
return ( return (
f"🧠 Second Brain Heartbeat\n" f"🧠 Second Brain Heartbeat\n"
f"Unbestätigte Engramme mit gutem Confidence-Score:\n{contents}\n" f"Unbestätigte Engramme mit gutem Confidence-Score:\n{contents}\n"
@@ -195,8 +212,39 @@ def enrich_context(topic: str, limit: int = 3) -> str:
# memory_context in das Prompt einbauen # memory_context in das Prompt einbauen
""" """
store = get_brain() store = get_brain()
ret = Retriever(store)
results = ret.retrieve(topic, limit=limit, min_confidence=0.3) # Versuche Hybrid-Retrieval (FTS + optional Vector), fallback auf Textsuche
if Retriever:
chroma = None
if ChromaStore:
try:
chroma = ChromaStore(path=str(Path(__file__).parent.parent / "data" / "chroma"))
except Exception:
chroma = None
ret = Retriever(store, chroma=chroma)
try:
results = ret.hybrid_retrieve(topic, limit=limit * 3, min_confidence=0.3)
except Exception:
results = ret.retrieve(topic, limit=limit * 3, min_confidence=0.3)
# confirmed-first ranking
def _rank(r):
eg = r["engram"]
confirmed = 1 if getattr(eg.correctness, "confirmed", False) else 0
return (confirmed, float(r.get("score", 0.0)))
results.sort(key=_rank, reverse=True)
# If we have confirmed results, show only confirmed up to limit
confirmed_only = [r for r in results if r["engram"].correctness.confirmed]
if confirmed_only:
results = confirmed_only[:limit]
else:
results = results[:limit]
else:
results_raw = store.search_text(topic, limit=limit)
results = [{"engram": eg, "score": 0.5} for eg in results_raw]
if not results: if not results:
return "" return ""

165
src/proactive_search.py Normal file
View File

@@ -0,0 +1,165 @@
#!/usr/bin/env python3
"""
proactive_search.py - Proaktive Websuche für Second Brain.
Sucht relevante Themen, speichert Ergebnisse als Engramme.
Stoppt wenn neue Aufgaben erkannt werden.
"""
import sys
import json
from pathlib import Path
from datetime import datetime, timezone, timedelta
from typing import List, Dict, Any, Optional
sys.path.insert(0, str(Path(__file__).resolve().parent))
from src.store import EngramStore
from src.engram import Engram, Grounding
from src.retriever import Retriever
from src.embedder import encode
from src.chroma_store import ChromaStore
DB_PATH = Path(__file__).resolve().parent.parent / "data" / "brain.sqlite"
CHROMA_PATH = Path(__file__).resolve().parent.parent / "data" / "chroma"
# Themen die relevant sind für den Benutzer
INTEREST_TOPICS = [
"OpenClaw AI Agent",
"Künstliche Intelligenz Trends 2025",
"Second Brain Memory System",
"Automation DIY Projects",
"Smart Home IoT",
"Raspberry Pi Projects",
"Deutschland Tech News",
"AI Agent Frameworks",
"Workflow Automation",
]
def get_store():
return EngramStore(str(DB_PATH))
def load_state() -> Dict[str, Any]:
"""Lädt den Such-Zustand."""
state_path = Path(__file__).resolve().parent.parent / "data" / "search_state.json"
if state_path.exists():
with open(state_path, "r", encoding="utf-8") as f:
return json.load(f)
return {
"last_search": None,
"searched_topics": [],
"new_tasks_detected": False,
"paused_until": None,
}
def save_state(state: Dict[str, Any]):
state_path = Path(__file__).resolve().parent.parent / "data" / "search_state.json"
with open(state_path, "w", encoding="utf-8") as f:
json.dump(state, f, ensure_ascii=False)
def check_for_new_tasks(store: EngramStore) -> bool:
"""Prüft ob in letzten 2h neue Aufgaben-Artige Engramme erstellt wurden."""
now = datetime.now(timezone.utc)
recent = now - timedelta(hours=2)
egs = store.get_all(limit=1000)
for eg in egs:
created_str = eg.metadata.get("created", "")
if not created_str:
continue
try:
eg_time = datetime.fromisoformat(created_str)
if eg_time.tzinfo is None:
eg_time = eg_time.replace(tzinfo=timezone.utc)
if eg_time > recent:
tags = eg.metadata.get("tags", [])
if "task" in tags or "aufgabe" in tags or "todo" in tags:
return True
except Exception:
pass
return False
def try_web_search(topic: str) -> Optional[List[Dict[str, str]]]:
"""Web-Suche via OpenClaw."""
try:
import subprocess
result = subprocess.run(
["python3", "-c", f"""
import sys
sys.path.insert(0, '/root/.openclaw/workspace/second-brain/src')
from src.retriever import Retriever
from src.store import EngramStore
store = EngramStore('data/brain.sqlite')
ret = Retriever(store)
results = ret.retrieve('{topic}')
print('FOUND ' + str(len(results)))
"""],
capture_output=True,
text=True,
timeout=30,
cwd="/root/.openclaw/workspace/second-brain",
)
# Actually do web search
print(f"[search] Would search: {topic}")
return None # Placeholder: real search would be here
except Exception as e:
print(f"[search] Error: {e}")
return None
def run_proactive_search():
"""Haupt-Funktion für proaktive Suche."""
store = get_store()
state = load_state()
now = datetime.now(timezone.utc)
# Check: Neue Aufgaben?
if check_for_new_tasks(store):
state["new_tasks_detected"] = True
state["paused_until"] = (now + timedelta(hours=4)).isoformat()
save_state(state)
print("🛑 Neue Aufgaben erkannt. Suche pausiert für 4h.")
return
# Check: Pausiert?
if state.get("paused_until"):
paused = datetime.fromisoformat(state["paused_until"])
if now < paused:
print(f"⏸️ Suche pausiert bis {state['paused_until']}")
return
else:
state["paused_until"] = None
state["new_tasks_detected"] = False
# Thema auswählen (Round-Robin)
searched = set(state.get("searched_topics", []))
remaining = [t for t in INTEREST_TOPICS if t not in searched]
if not remaining:
remaining = INTEREST_TOPICS
searched = set()
topic = remaining[0]
print(f"🔍 Suche: {topic}")
# Als Engramm speichern (als "Suchanfrage", nicht als Faktum)
eg = Engram.create(
content=f"Proaktive Web-Suche: {topic}\nStatus: Geplant",
source="agent",
tags=["proactive", "search", "planned"],
confidence=0.3,
grounding=Grounding.ASSUMPTION,
)
store.save(eg)
state["last_search"] = now.isoformat()
state["searched_topics"] = list(searched | {topic})
save_state(state)
print(f"✅ Such-Engramm gespeichert: {eg.id}")
if __name__ == "__main__":
run_proactive_search()

View File

@@ -7,12 +7,10 @@ Phase 2: + Embedding + Fusion.
from typing import List, Dict, Any, Optional from typing import List, Dict, Any, Optional
from .engram import Engram from .engram import Engram
from .store import EngramStore from .store import EngramStore
from .chroma_store import ChromaStore
from .embedder import encode
class Retriever: class Retriever:
def __init__(self, store: EngramStore, chroma: Optional[ChromaStore] = None): def __init__(self, store: EngramStore, chroma: Optional[object] = None):
self.store = store self.store = store
self.chroma = chroma self.chroma = chroma
@@ -50,7 +48,6 @@ class Retriever:
if not self.chroma: if not self.chroma:
return [] return []
chroma_results = self.chroma.query(query, top_k=limit * 3) chroma_results = self.chroma.query(query, top_k=limit * 3)
eids = [r["id"] for r in chroma_results]
results = [] results = []
for r in chroma_results: for r in chroma_results:
eg = self.store.get(r["id"]) eg = self.store.get(r["id"])

View File

@@ -127,6 +127,14 @@ class EngramStore:
).fetchall() ).fetchall()
return [self._row_to_engram(r) for r in rows] return [self._row_to_engram(r) for r in rows]
def get_modified_since(self, iso_ts: str, limit: int = 5000) -> List[Engram]:
"""Gibt Engramme zurück, deren `modified_at` nach `iso_ts` liegt."""
rows = self._conn.execute(
"SELECT * FROM engrams WHERE modified_at > ? ORDER BY modified_at ASC LIMIT ?",
(iso_ts, limit),
).fetchall()
return [self._row_to_engram(r) for r in rows]
def delete(self, engram_id: str) -> bool: def delete(self, engram_id: str) -> bool:
"""Löscht ein Engramm und alle Verknüpfungen.""" """Löscht ein Engramm und alle Verknüpfungen."""
rowid = self._conn.execute( rowid = self._conn.execute(
@@ -239,6 +247,13 @@ class EngramStore:
"links": json.loads(row["links_json"]), "links": json.loads(row["links_json"]),
"hierarchy": json.loads(row["hierarchy_json"]), "hierarchy": json.loads(row["hierarchy_json"]),
} }
# Keep Engram metadata timestamps aligned with DB columns so downstream
# consumers (e.g. vector indexing watermarks) can rely on them.
try:
d["metadata"]["created"] = row["created_at"]
d["metadata"]["modified"] = row["modified_at"]
except Exception:
pass
emb = row["embedding_json"] emb = row["embedding_json"]
if emb: if emb:
d["embedding"] = json.loads(emb) d["embedding"] = json.loads(emb)

316
static/style.css Normal file
View File

@@ -0,0 +1,316 @@
/* ─── Reset & Base ────────────────────────────────────────────────────────── */
* { margin: 0; padding: 0; box-sizing: border-box; }
html { font-size: 14px; }
body {
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
background: #0f0f12;
color: #e8e8ee;
-webkit-font-smoothing: antialiased;
overscroll-behavior-y: contain;
}
.app {
max-width: 480px;
margin: 0 auto;
min-height: 100vh;
background: #141419;
}
/* ─── Stats Bar ───────────────────────────────────────────────────────────── */
.stats-bar {
display: flex;
justify-content: space-around;
padding: 10px 8px;
background: linear-gradient(180deg, #1a1a22 0%, #131318 100%);
border-bottom: 1px solid #252530;
position: sticky;
top: 0;
z-index: 50;
}
.stat {
text-align: center;
min-width: 60px;
}
.stat-num {
display: block;
font-size: 1.3rem;
font-weight: 700;
color: #6c8af5;
line-height: 1.2;
}
.stat-label {
display: block;
font-size: 0.65rem;
color: #888899;
text-transform: uppercase;
letter-spacing: 0.5px;
margin-top: 2px;
}
/* ─── Search ──────────────────────────────────────────────────────────────── */
.search-box {
display: flex;
gap: 8px;
padding: 10px 12px;
background: #141419;
}
#searchInput {
flex: 1;
background: #1e1e28;
border: 1px solid #2a2a3a;
border-radius: 10px;
padding: 10px 14px;
color: #e8e8ee;
font-size: 0.95rem;
outline: none;
}
#searchInput:focus { border-color: #6c8af5; }
#filterSelect {
background: #1e1e28;
border: 1px solid #2a2a3a;
border-radius: 10px;
padding: 10px;
color: #e8e8ee;
font-size: 0.85rem;
outline: none;
}
/* ─── New Engram ──────────────────────────────────────────────────────────── */
.new-engram {
padding: 0 12px 8px;
display: flex;
flex-direction: column;
gap: 6px;
}
.new-engram textarea {
background: #1e1e28;
border: 1px solid #2a2a3a;
border-radius: 10px;
padding: 10px;
color: #e8e8ee;
font-size: 0.9rem;
resize: vertical;
min-height: 50px;
}
.new-engram input {
background: #1e1e28;
border: 1px solid #2a2a3a;
border-radius: 10px;
padding: 8px 10px;
color: #e8e8ee;
font-size: 0.85rem;
}
.new-engram button {
background: #3a7d3a;
border: none;
border-radius: 10px;
padding: 10px;
color: #fff;
font-weight: 600;
font-size: 0.95rem;
}
/* ─── Cards ───────────────────────────────────────────────────────────────── */
.cards {
padding: 4px 12px;
display: flex;
flex-direction: column;
gap: 10px;
}
.card {
background: #1a1a24;
border: 1px solid #252533;
border-radius: 14px;
overflow: hidden;
transition: transform 0.15s ease, border-color 0.2s ease;
touch-action: manipulation;
}
.card:active { transform: scale(0.985); }
.card.confirmed { border-left: 4px solid #3a7d3a; }
.card.rejected { border-left: 4px solid #8a3a3a; }
.card-header {
display: flex;
align-items: center;
gap: 8px;
padding: 10px 12px 6px;
border-bottom: 1px solid #20202a;
}
.conf-badge {
font-size: 0.75rem;
font-weight: 700;
color: #fff;
padding: 2px 8px;
border-radius: 20px;
min-width: 36px;
text-align: center;
}
.tags {
flex: 1;
display: flex;
flex-wrap: wrap;
gap: 4px;
}
.tag {
background: #2a2a3a;
color: #8a9aff;
font-size: 0.68rem;
padding: 2px 8px;
border-radius: 12px;
text-transform: uppercase;
letter-spacing: 0.3px;
}
.date {
font-size: 0.65rem;
color: #666677;
white-space: nowrap;
}
.card-body {
padding: 10px 12px;
font-size: 0.9rem;
line-height: 1.45;
color: #ccccdd;
cursor: pointer;
}
.card-footer {
display: flex;
align-items: center;
gap: 8px;
padding: 8px 12px 10px;
border-top: 1px solid #20202a;
}
.reason-input {
flex: 1;
background: #14141a;
border: 1px solid #2a2a3a;
border-radius: 8px;
padding: 6px 10px;
color: #b0b0c0;
font-size: 0.8rem;
}
.actions {
display: flex;
gap: 6px;
}
.actions button {
width: 34px;
height: 34px;
border-radius: 50%;
border: none;
font-size: 1rem;
cursor: pointer;
display: flex;
align-items: center;
justify-content: center;
transition: transform 0.1s;
}
.actions button:active { transform: scale(0.9); }
.btn-ok { background: #2a5a2a; }
.btn-no { background: #5a2a2a; }
.btn-archive { background: #2a2a4a; }
/* ─── Pagination ──────────────────────────────────────────────────────────── */
.pagination {
display: flex;
justify-content: center;
align-items: center;
gap: 16px;
padding: 14px;
}
.pagination button {
background: #252535;
border: none;
border-radius: 10px;
width: 40px;
height: 40px;
color: #e8e8ee;
font-size: 1.1rem;
}
.pagination button:disabled {
opacity: 0.3;
}
#pageNum {
font-weight: 700;
color: #6c8af5;
}
/* ─── Footer ──────────────────────────────────────────────────────────────── */
.footer {
display: flex;
justify-content: space-between;
align-items: center;
padding: 10px 16px 20px;
color: #555566;
font-size: 0.75rem;
}
.refresh-btn {
background: #252535;
border: none;
border-radius: 50%;
width: 36px;
height: 36px;
color: #8888aa;
font-size: 1.2rem;
}
/* ─── Modal ───────────────────────────────────────────────────────────────── */
.modal {
display: none;
position: fixed;
inset: 0;
background: rgba(0,0,0,0.85);
z-index: 100;
overflow-y: auto;
padding: 20px 16px;
}
.modal.open { display: block; }
.modal-content {
background: #1a1a24;
border: 1px solid #333344;
border-radius: 16px;
padding: 20px 16px;
max-width: 480px;
margin: 0 auto;
position: relative;
}
.close-btn {
position: absolute;
top: 10px;
right: 14px;
background: none;
border: none;
color: #8888aa;
font-size: 1.5rem;
cursor: pointer;
}
.history {
list-style: none;
font-size: 0.8rem;
color: #9999aa;
}
.history li {
padding: 4px 0;
border-bottom: 1px solid #20202a;
}
.detail-content {
background: #14141a;
border-radius: 10px;
padding: 12px;
font-size: 0.88rem;
line-height: 1.5;
margin: 8px 0;
max-height: 40vh;
overflow-y: auto;
}
/* ─── Scrollbar ───────────────────────────────────────────────────────────── */
::-webkit-scrollbar { width: 4px; }
::-webkit-scrollbar-track { background: transparent; }
::-webkit-scrollbar-thumb { background: #333344; border-radius: 4px; }
/* ─── Touch fix ───────────────────────────────────────────────────────────── */
@media (pointer: coarse) {
button, .card { -webkit-tap-highlight-color: transparent; }
}

246
templates/dashboard.html Normal file
View File

@@ -0,0 +1,246 @@
<!DOCTYPE html>
<html lang="de">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width; initial-scale=1.0; maximum-scale=1.0; user-scalable=no">
<title>🧠 Second Brain</title>
<link rel="stylesheet" href="/static/style.css">
</head>
<body>
<div class="app">
<!-- Stats Header -->
<header class="stats-bar" id="statsBar">
<div class="stat"><span class="stat-num" id="statTotal">-</span><span class="stat-label">Total</span></div>
<div class="stat"><span class="stat-num" id="statConfirmed">-</span><span class="stat-label">OK</span></div>
<div class="stat"><span class="stat-num" id="statPending">-</span><span class="stat-label">Pending</span></div>
<div class="stat"><span class="stat-num" id="statErrors">-</span><span class="stat-label">Err</span></div>
</header>
<!-- Search -->
<div class="search-box">
<input type="text" id="searchInput" placeholder="🔍 Suche..." />
<select id="filterSelect">
<option value="all">Alle</option>
<option value="pending">Pending</option>
<option value="confirmed">Confirmed</option>
<option value="errors">Errors</option>
</select>
</div>
<!-- New Engram -->
<div class="new-engram">
<textarea id="newContent" placeholder="Neues Engramm..."></textarea>
<input type="text" id="newTags" placeholder="Tags (comma sep)" />
<button onclick="createEngram()"> Speichern</button>
</div>
<!-- Cards -->
<div class="cards" id="cards"></div>
<!-- Pagination -->
<div class="pagination" id="pagination">
<button id="btnPrev" onclick="prevPage()"></button>
<span id="pageNum">1</span>
<button id="btnNext" onclick="nextPage()"></button>
</div>
<div class="footer">
<span id="lastUpdate">--:--</span>
<button onclick="manualRefresh()" class="refresh-btn"></button>
</div>
</div>
<!-- Detail Modal -->
<div class="modal" id="detailModal">
<div class="modal-content">
<button class="close-btn" onclick="closeModal()">×</button>
<div id="modalBody"></div>
</div>
</div>
<script>
// ─── State ──────────────────────────────────────────────────────────────────
let state = {
items: [],
offset: 0,
limit: 10,
filter: 'all',
search: '',
autoRefresh: true,
};
// ─── Fetch ──────────────────────────────────────────────────────────────────
async function api(path, opts = {}) {
const r = await fetch(path, opts);
if (!r.ok) throw new Error((await r.json()).error || r.statusText);
return r.json();
}
async function loadStats() {
const s = await api('/api/stats');
document.getElementById('statTotal').textContent = s.total;
document.getElementById('statConfirmed').textContent = s.confirmed;
document.getElementById('statPending').textContent = s.pending;
document.getElementById('statErrors').textContent = s.errors;
}
async function loadCards() {
let url = `/api/engrams?limit=${state.limit}&offset=${state.offset}`;
if (state.search) url += `&search=${encodeURIComponent(state.search)}`;
if (state.filter === 'confirmed') url += '&confirmed=1';
if (state.filter === 'pending') url += '&confirmed=0';
if (state.filter === 'errors') url += '&tag=error';
const data = await api(url);
state.items = data.items;
renderCards();
document.getElementById('pageNum').textContent = Math.floor(state.offset / state.limit) + 1;
document.getElementById('btnPrev').disabled = state.offset === 0;
document.getElementById('btnNext').disabled = data.items.length < state.limit;
}
function renderCards() {
const el = document.getElementById('cards');
el.innerHTML = state.items.map(item => `
<div class="card ${item.confirmed ? 'confirmed' : ''} ${item.rejections > 0 ? 'rejected' : ''}" data-id="${item.id}">
<div class="card-header">
<span class="conf-badge" style="background:hsl(${item.confidence*120},70%,40%)">${Math.round(item.confidence*100)}%</span>
<span class="tags">${item.tags.map(t => '<span class="tag">'+t+'</span>').join('')}</span>
<span class="date">${fmtDate(item.created)}</span>
</div>
<div class="card-body" onclick="showDetail('${item.id}')">
${escapeHtml(item.content.substring(0, 200))}${item.content.length>200?'...':''}
</div>
<div class="card-footer">
<input type="text" class="reason-input" placeholder="Grund (optional)" id="reason-${item.id}"/>
<div class="actions">
<button class="btn-ok" onclick="confirm('${item.id}', event)">✅</button>
<button class="btn-no" onclick="reject('${item.id}', event)">❌</button>
<button class="btn-archive" onclick="refresh('${item.id}', event)">🔄</button>
</div>
</div>
</div>
`).join('');
}
function fmtDate(iso) {
const d = new Date(iso);
return `${d.getDate().toString().padStart(2,'0')}.${(d.getMonth()+1).toString().padStart(2,'0')} ${d.getHours().toString().padStart(2,'0')}:${d.getMinutes().toString().padStart(2,'0')}`;
}
function escapeHtml(t) {
const d = document.createElement('div');
d.textContent = t;
return d.innerHTML;
}
// ─── Actions ────────────────────────────────────────────────────────────────
async function confirm(id, ev) {
ev.stopPropagation();
const reason = document.getElementById('reason-'+id).value;
await api(`/api/engrams/${id}/confirm`, {
method: 'POST',
headers: {'Content-Type': 'application/x-www-form-urlencoded'},
body: new URLSearchParams({reason})
});
await loadCards(); await loadStats();
}
async function reject(id, ev) {
ev.stopPropagation();
const reason = document.getElementById('reason-'+id).value;
await api(`/api/engrams/${id}/reject`, {
method: 'POST',
headers: {'Content-Type': 'application/x-www-form-urlencoded'},
body: new URLSearchParams({reason})
});
await loadCards(); await loadStats();
}
async function refresh(id, ev) {
ev.stopPropagation();
await api(`/api/engrams/${id}/refresh`, {method: 'POST'});
await loadCards();
}
async function createEngram() {
const content = document.getElementById('newContent').value;
const tags = document.getElementById('newTags').value;
if (!content.trim()) return;
await api('/api/engrams', {
method: 'POST',
headers: {'Content-Type': 'application/x-www-form-urlencoded'},
body: new URLSearchParams({content, tags})
});
document.getElementById('newContent').value = '';
document.getElementById('newTags').value = '';
await loadCards(); await loadStats();
}
async function showDetail(id) {
const item = await api(`/api/engrams/${id}`);
const body = document.getElementById('modalBody');
body.innerHTML = `
<h3>Engramm ${item.id.substring(0,8)}</h3>
<p><b>Confidence:</b> ${Math.round(item.confidence*100)}%</p>
<p><b>Confirmed:</b> ${item.confirmed ? '✅' : '❌'}</p>
<p><b>Tags:</b> ${item.tags.map(t => '<span class="tag">'+t+'</span>').join(' ')}</p>
<p><b>Content:</b></p>
<div class="detail-content">${escapeHtml(item.content)}</div>
<p><b>History:</b></p>
<ul class="history">
${(item.review_history || []).map(h => `<li>${fmtDate(h.at)}${h.action} (${h.note})</li>`).join('')}
</ul>
<p><b>Links:</b> ${item.links?.join(', ') || 'none'}</p>
`;
document.getElementById('detailModal').classList.add('open');
}
function closeModal() {
document.getElementById('detailModal').classList.remove('open');
}
// ─── Pagination ─────────────────────────────────────────────────────────────
function nextPage() {
state.offset += state.limit;
loadCards();
}
function prevPage() {
state.offset = Math.max(0, state.offset - state.limit);
loadCards();
}
function manualRefresh() {
loadCards(); loadStats();
}
// ─── Search ─────────────────────────────────────────────────────────────────
document.getElementById('searchInput').addEventListener('input', (e) => {
state.search = e.target.value;
state.offset = 0;
loadCards();
});
document.getElementById('filterSelect').addEventListener('change', (e) => {
state.filter = e.target.value;
state.offset = 0;
loadCards();
});
// ─── Auto Refresh ───────────────────────────────────────────────────────────
setInterval(() => {
if (!state.autoRefresh) return;
loadStats();
loadCards();
const now = new Date();
document.getElementById('lastUpdate').textContent =
`${now.getHours().toString().padStart(2,'0')}:${now.getMinutes().toString().padStart(2,'0')}`;
}, 5000);
// ─── Init ───────────────────────────────────────────────────────────────────
loadStats();
loadCards();
</script>
</body>
</html>

View File

@@ -4,16 +4,11 @@
import sys import sys
import os import os
import tempfile import tempfile
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src")) sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
try:
from src.engram import Engram, Grounding, Correctness from src.engram import Engram, Grounding, Correctness
from src.store import EngramStore from src.store import EngramStore
from src.retriever import Retriever from src.retriever import Retriever
except ImportError:
from engram import Engram, Grounding, Correctness
from store import EngramStore
from retriever import Retriever
def test_engram_creation(): def test_engram_creation():