18 Commits

Author SHA1 Message Date
f10a5b9f19 docs: add dashboard UI/UX design plan 2026-05-29 10:50:23 +02:00
6232f25cc9 fix(fastapi): remove duplicate confirm/reject routes
- api_confirm and api_reject were defined twice on same paths
- FastAPI only registers first definition, causing silent 404s
- Kept api_confirm_engram and api_reject_engram (use _update_correctness)
- Removed duplicate direct DB implementations
- Fixes dashboard confirm/reject buttons not working
2026-05-27 18:36:03 +02:00
6b0cf5889f fix(store): escape FTS5 special characters in search_text()
- FTS5 crashes on dots (IP addresses) and hyphens (dates)
- Add regex to strip non-alphanumeric chars before FTS5 MATCH
- Fixes: fts5 syntax error near '.' and no such column: 05

Files changed: src/store.py
2026-05-27 17:54:51 +02:00
021fd0e328 feat(dashboard): pending queue + confirm/reject endpoints 2026-05-27 01:17:32 +02:00
d52e3a7f74 feat(ingest): transcript direct to DB 2026-05-27 01:14:42 +02:00
1635ee8b03 fix(verify): scan all engrams 2026-05-27 01:11:59 +02:00
f8ac0af869 chore(timers): ingest memory every 5 min 2026-05-27 01:11:17 +02:00
9dd5e49e2a chore(timers): process pending every 5 min 2026-05-27 01:08:08 +02:00
b158b19208 feat(ingest): tail session transcript into memory 2026-05-27 01:04:47 +02:00
095e6a33f8 feat(dashboard): realtime status + graph render 2026-05-27 00:22:14 +02:00
e5061b317f feat(dashboard): add status+graph views 2026-05-27 00:11:44 +02:00
ec8870ea40 feat(verify): add pending external verifier 2026-05-27 00:05:51 +02:00
8f47151a48 docs(systemd): fix paths + add dashboard smoke + vendor units 2026-05-26 23:47:07 +02:00
83b85cb760 Add FastAPI dashboard MVP 2026-05-26 21:02:39 +02:00
e1640071e4 Second-brain 2.0: hybrid retrieval, obsidian bridge, vector watermark, tests 2026-05-26 19:27:12 +02:00
29bc45d623 feat(systemd): Dashboard-Service, brain_rules, 18 Engramme bewertet, Cron persistent
Neu:
- systemd: secondbrain-dashboard.service (Port 8501, autostart)
- cron_rules.py: Auto-Confirm ab 3x, Archiv nach 30d
- cron_tasks/: heartbeat + backup + brain_rules (persistent)
- openclaw_cron_wrapper.py: subprocess-Isolation (kein SessionTakeover)
- chat_autosave.py: Auto-Save von Chat + Kontext-Anreicherung

Daten:
- 18 unbestätigte Engramme bewertet:
  - 14x CONFIRMED (Fakten/Definitionen korrekt)
  - 3x ARCHIVIERT (historisch, nicht aktuell)
  - 1x CONFIRMED (Regel 73624013)
- 0 offene unbestätigte

Closes Gitea-Issue: #9
2026-05-25 22:35:44 +02:00
a5d5b2f2ec fix(cron): Session-Takeover-Workaround, mobil-Dashboard, Import-Fix standalone
- feat(isolation): Cron-Wrapper (subprocess, /workspace/cron_tasks)
- fix(dashboard): mobil-optimierte Karten statt Tabelle
- fix(imports): sys.path + venv auto-activation
- chore: Lasse flüchtiges /tmp hinter mir

Closes-Bug: Session-Takeover bei isolated Cron
Engramme-bestaetigt: Standalone-Import-OK, Wrapper-getestet
2026-05-25 22:08:52 +02:00
4e0f5e7e9a feat(active): Proaktive Suche (Cron 4h), Aufgaben-Tracking, Heartbeat-Integration, Stop-Logik 2026-05-25 11:17:05 +02:00
54 changed files with 3928 additions and 97 deletions

View File

@@ -8,7 +8,28 @@ An embeddable, offline-first memory system for AI agents with correctness tracki
- **ChromaDB Vector Store** (`src/chroma_store.py`) — Semantic similarity search - **ChromaDB Vector Store** (`src/chroma_store.py`) — Semantic similarity search
- **Neural Confidence Scorer** (`src/neural_scorer.py`) — PyTorch RL net, trains on confirm/reject feedback - **Neural Confidence Scorer** (`src/neural_scorer.py`) — PyTorch RL net, trains on confirm/reject feedback
- **Hybrid Retrieval** (`src/retriever.py`) — Keyword + Semantic + Neural fusion - **Hybrid Retrieval** (`src/retriever.py`) — Keyword + Semantic + Neural fusion
- **Streamlit Dashboard** (`src/app_dashboard.py`) — Search, confirm/reject, neural training UI - **FastAPI Dashboard** (`fastapi_app.py`) — Lightweight web UI (search + confirm/reject) and JSON API
- **Streamlit Dashboard** (`src/app_dashboard.py`) — (optional) richer UI for neural training, etc.
- **Graph Visualization** (`src/graph_view.py`) — Interactive Cytoscape.js graph with confidence colors - **Graph Visualization** (`src/graph_view.py`) — Interactive Cytoscape.js graph with confidence colors
## Architecture ## Architecture
## Obsidian
Setup and timers: `second-brain/docs/OBSIDIAN.md`
## Quickstart (Dashboard)
Install minimal dashboard deps:
`python3 -m pip install -r requirements-dashboard.txt`
Run:
`SECOND_BRAIN_WORKSPACE="$(pwd)" python3 fastapi_app.py`
Then open: `http://localhost:8501/`
Port is configurable via `SECOND_BRAIN_PORT` (or `PORT`), e.g.:
`SECOND_BRAIN_WORKSPACE="$(pwd)" SECOND_BRAIN_PORT=8502 python3 fastapi_app.py`

94
RUNBOOK.md Normal file
View File

@@ -0,0 +1,94 @@
# Second-Brain 2.0 (Grundversion) — Runbook
This is the operational quick-reference for the shipped systemd timers/services and the optional FastAPI dashboard backend.
Repository root (on host): `/root/.openclaw/workspace/second-brain`
## Systemd units (cron jobs)
Unit files are shipped in `systemd/` (this repo). Install them into `/etc/systemd/system/` (symlink or copy), then reload:
```bash
sudo ln -sf /root/.openclaw/workspace/second-brain/systemd/openclaw-secondbrain-*.service /etc/systemd/system/
sudo ln -sf /root/.openclaw/workspace/second-brain/systemd/openclaw-secondbrain-*.timer /etc/systemd/system/
sudo ln -sf /root/.openclaw/workspace/second-brain/systemd/openclaw-memory-archive.* /etc/systemd/system/
sudo systemctl daemon-reload
```
Optional (verification hardening):
```bash
sudo ln -sf /root/.openclaw/workspace/second-brain/systemd/openclaw-secondbrain-verify-pending.* /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now openclaw-secondbrain-verify-pending.timer
```
Enable timers:
```bash
sudo systemctl enable --now openclaw-secondbrain-ingest-memory.timer
sudo systemctl enable --now openclaw-secondbrain-index-vectors.timer
sudo systemctl enable --now openclaw-secondbrain-review.timer
sudo systemctl enable --now openclaw-secondbrain-backup.timer
sudo systemctl enable --now openclaw-secondbrain-heartbeat.timer
sudo systemctl enable --now openclaw-secondbrain-proactive-search.timer
sudo systemctl enable --now openclaw-memory-archive.timer
# Optional (Obsidian coupling)
sudo systemctl enable --now openclaw-secondbrain-ingest-obsidian.timer
sudo systemctl enable --now openclaw-secondbrain-export-obsidian.timer
```
Verify scheduling:
```bash
sudo systemctl list-timers --all | grep -E 'openclaw-(secondbrain|memory-archive)' || true
```
Run a job once:
```bash
sudo systemctl start openclaw-secondbrain-ingest-memory.service
sudo systemctl status openclaw-secondbrain-ingest-memory.service --no-pager
sudo journalctl -u openclaw-secondbrain-ingest-memory.service -n 200 --no-pager
```
Wrapper logs:
```bash
tail -n 200 /root/.openclaw/workspace/cron_wrapper.log
```
## FastAPI dashboard (manual start)
FastAPI entrypoint:
```bash
cd /root/.openclaw/workspace
python3 -m pip install -r second-brain/requirements-dashboard.txt
SECOND_BRAIN_WORKSPACE="/root/.openclaw/workspace/second-brain" python3 second-brain/fastapi_app.py
```
Default port is `8501` (same as Streamlit default). You can override via `SECOND_BRAIN_PORT` (or `PORT`) when starting manually.
Endpoint smoke tests:
```bash
curl -fsS http://127.0.0.1:8501/api/stats
curl -fsS "http://127.0.0.1:8501/api/engrams?limit=1&offset=0"
curl -fsS "http://127.0.0.1:8501/api/search?q=test&limit=1"
```
## DB quick check
```bash
python3 - <<'PY'
import sqlite3
db="/root/.openclaw/workspace/second-brain/data/brain.sqlite"
con=sqlite3.connect(db)
cur=con.cursor()
print(cur.execute("PRAGMA integrity_check").fetchone()[0])
print("engrams:", cur.execute("SELECT COUNT(*) FROM engrams").fetchone()[0])
con.close()
PY
```

119
chat_autosave.py Normal file
View File

@@ -0,0 +1,119 @@
#!/usr/bin/env python3
"""
Chat-Auto-Save: Wertvolle User-Nachrichten → Engramm.
Wird am Ende jeder Main-Session-Antwort aufgerufen.
"""
import sys
import json
import hashlib
from pathlib import Path
BRAIN_DIR = Path("/root/.openclaw/workspace/second-brain")
sys.path.insert(0, str(BRAIN_DIR))
from src.engram import Engram, Grounding
from src.store import EngramStore
DB_PATH = BRAIN_DIR / "data" / "brain.sqlite"
def _hash(text: str) -> str:
return hashlib.sha256(text.encode("utf-8")).hexdigest()[:12]
def is_fluff(content: str) -> bool:
"""Prüft ob Inhalt nur Floskel ist."""
lower = content.lower().strip().rstrip(".?!")
short_fluff = [
"hallo", "hi", "hey", "guten tag", "guten morgen", "guten abend",
"danke", "ok", "okay", "ja", "nein", "bitte", "gerne", "tschüss",
"bis später", "bis morgen", "alles klar", "in ordnung",
]
if lower in short_fluff:
return True
if len(content) < 10 and all(c in " ?,!.;:-" for c in content):
return True
return False
def save_if_worthy(content: str, source: str = "user", tags: list = None,
confidence: float = 0.7, session_id: str = None,
reasoning: str = None) -> dict:
"""
Speichert Nachricht als Engramm wenn sie Wert hat.
Wird in jeder Antwort aufgerufen.
"""
if is_fluff(content):
return {"saved": False, "reason": "fluff"}
store = EngramStore(str(DB_PATH))
content_hash = _hash(content)
recent = store.get_all(limit=200)
for eg in recent:
if _hash(eg.content) == content_hash:
return {"saved": False, "reason": "duplicate", "id": str(eg.id)}
eg = Engram.create(
content=content,
source=source,
tags=tags or ["auto-save", "chat"],
session_id=session_id,
confidence=confidence,
grounding=Grounding.ASSUMPTION,
)
store.save(eg)
return {
"saved": True,
"id": str(eg.id),
"confidence": eg.compute_confidence(),
"first8": str(eg.id)[:8],
}
def enrich_prompt(topic: str, limit: int = 3) -> str:
"""
Holt relevante bestätigte Engramme für Kontext-Anreicherung.
Wird VOR jeder Antwort aufgerufen.
"""
store = EngramStore(str(DB_PATH))
recent = store.get_all(limit=100)
# Einfache Text-Suche (kein FTS wegen Satzzeichen)
topic_lower = topic.lower()
matches = []
for eg in recent:
if eg.correctness.confirmed and topic_lower in eg.content.lower():
matches.append(eg)
elif len(matches) < limit and any(t in topic_lower for t in [t.lower() for t in eg.metadata.get("tags", [])]):
matches.append(eg)
if len(matches) >= limit:
break
if not matches:
return ""
lines = ["\n📚 Relevantes Wissen:"]
for eg in matches[:limit]:
lines.append(f" • [{eg.compute_confidence():.0%}] {eg.content[:120]}")
return "\n".join(lines)
def check_pending(session_id: str = None) -> list:
"""Gibt unbestätigte Engramme zurück."""
store = EngramStore(str(DB_PATH))
egs = store.get_all(limit=50)
pending = [eg for eg in egs if not eg.correctness.confirmed]
return pending
if __name__ == "__main__":
import sys
if len(sys.argv) > 1:
result = save_if_worthy(sys.argv[1])
print(json.dumps(result, indent=2))
else:
print("Usage: python3 chat_autosave.py 'Nachricht'")

View File

@@ -0,0 +1,22 @@
#!/usr/bin/env python3
"""Backup-Task für Second Brain - isoliert, persistent."""
import json, os, sys
from pathlib import Path
from datetime import datetime, timezone
BRAIN_DIR = Path("/root/.openclaw/workspace/second-brain")
sys.path.insert(0, str(BRAIN_DIR))
from src.store import EngramStore
def main():
brain_db = os.environ.get("BRAIN_DB", str(BRAIN_DIR / "data" / "brain.sqlite"))
store = EngramStore(brain_db)
ts = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S")
backup_path = Path(brain_db).parent / f"backup_{ts}.jsonl"
count = store.export_jsonl(str(backup_path))
result = {"timestamp": datetime.now(timezone.utc).isoformat(), "backup_path": str(backup_path), "count": count, "success": True}
print(f"BACKUP: {count} Engramme -> {backup_path}")
return 0
if __name__ == "__main__":
sys.exit(main())

53
cron_tasks/brain_rules.py Normal file
View File

@@ -0,0 +1,53 @@
#!/usr/bin/env python3
"""
Brain-Regeln - Automatische Bestaetigungs- und Archivierungslogik.
Wird von Cron und Agent aufgerufen.
"""
import sys
sys.path.insert(0, "/root/.openclaw/workspace/second-brain")
from src.engram import Engram, Grounding
from src.store import EngramStore
DB = "/root/.openclaw/workspace/second-brain/data/brain.sqlite"
def apply_rules():
store = EngramStore(DB)
egs = store.get_all(limit=1000)
actions = []
for eg in egs:
conf = eg.compute_confidence()
age_days = eg._age_days(eg.metadata.get("created", ""))
correct = eg.correctness
# Regel 1: Triple-Confirm → Auto-Verifiziert
if not correct.confirmed and correct.confirmations >= 3:
correct.confirmed = True
correct.confirmations += 1
store.save(eg)
actions.append(f"Auto-Confirm: {str(eg.id)[:8]} (3x confirmed)")
# Regel 2: Lang unbestaetigt → ASSUMPTION Tag
if age_days > 30 and not correct.confirmed and "archiviert" not in eg.metadata.get("tags", []):
eg.metadata.setdefault("tags", []).append("archiviert")
eg.metadata["archivgrund"] = f"Unbestaetigt seit {age_days} Tagen"
store.save(eg)
actions.append(f"Archiviert: {str(eg.id)[:8]} (Alter {age_days}d)")
# Regel 3: Rejected mit 2+ Rejections → loeschen (Sanft: Tag statt rm)
if correct.rejections >= 2:
eg.metadata.setdefault("tags", []).append("deleted")
store.save(eg)
actions.append(f"Deleted-Tag: {str(eg.id)[:8]} ({correct.rejections}x rejected)")
return actions
if __name__ == "__main__":
actions = apply_rules()
print("Brain-Regeln angewendet:")
for a in actions or ["Keine Aktionen noetig"]:
print(f" {a}")

View File

@@ -0,0 +1,47 @@
#!/usr/bin/env python3
"""
Heartbeat-Task für Second Brain - isoliert, persistent.
"""
import json
import os
import sys
from pathlib import Path
from datetime import datetime, timezone
BRAIN_DIR = Path("/root/.openclaw/workspace/second-brain")
sys.path.insert(0, str(BRAIN_DIR))
from src.engram import Engram, Grounding
from src.store import EngramStore
def main():
output_file = os.environ.get("CRON_OUTPUT_FILE", "/tmp/heartbeat_result.json")
brain_db = os.environ.get("BRAIN_DB", str(BRAIN_DIR / "data" / "brain.sqlite"))
store = EngramStore(brain_db)
egs = store.get_all(limit=50)
unconfirmed = [eg for eg in egs if not eg.correctness.confirmed and eg.compute_confidence() > 0.5][:5]
errors = store.search_tag("error", limit=5)
result = {
"timestamp": datetime.now(timezone.utc).isoformat(),
"total_engrams": len(egs),
"unconfirmed_count": len(unconfirmed),
"error_count": len(errors),
"has_action": bool(unconfirmed) or len(errors) >= 3,
"message": None,
}
if unconfirmed:
contents = "\n".join([f" - {eg.content[:80]}" for eg in unconfirmed])
result["message"] = f"🧠 Unbestätigte Engramme:\n{contents}"
elif len(errors) >= 3:
result["message"] = f"⚠️ {len(errors)} Fehler-Engramme gespeichert."
Path(output_file).write_text(json.dumps(result, indent=2))
print(f"HEARTBEAT: {result['unconfirmed_count']} unconfirmed, {result['error_count']} errors")
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,160 @@
#!/usr/bin/env python3
"""
Ingest OpenClaw session transcript JSONL directly into the Second-Brain DB.
State is tracked with byte offsets per transcript file.
Sources are configured via workspace/memory/session_sources.json.
"""
from __future__ import annotations
import json
import os
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Dict, Iterable, List
from uuid import NAMESPACE_URL, uuid5
WORKSPACE = Path("/root/.openclaw/workspace")
MEMORY_DIR = WORKSPACE / "memory"
SOURCES_PATH = MEMORY_DIR / "session_sources.json"
STATE_PATH = MEMORY_DIR / "session_db_ingest_state.json"
BRAIN_DIR = WORKSPACE / "second-brain"
DB_PATH = Path(os.environ.get("BRAIN_DB", str(BRAIN_DIR / "data" / "brain.sqlite")))
import sys
sys.path.insert(0, str(BRAIN_DIR))
from src.engram import Engram, Grounding # type: ignore
from src.store import EngramStore # type: ignore
def _now() -> str:
return datetime.now(timezone.utc).isoformat()
def _load_json(path: Path, default: Any) -> Any:
try:
if not path.exists():
return default
return json.loads(path.read_text(encoding="utf-8"))
except Exception:
return default
def _save_json(path: Path, payload: Any) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(payload, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
def _extract_text(content: Any) -> str:
if isinstance(content, str):
return content.strip()
if isinstance(content, list):
parts: List[str] = []
for c in content:
if isinstance(c, dict) and c.get("type") == "text" and isinstance(c.get("text"), str):
parts.append(c["text"])
elif isinstance(c, str):
parts.append(c)
return "\n".join(p.strip() for p in parts if p and p.strip()).strip()
if isinstance(content, dict) and isinstance(content.get("text"), str):
return content["text"].strip()
return str(content).strip()
@dataclass
class Source:
label: str
transcript_path: Path
def _load_sources() -> List[Source]:
payload = _load_json(SOURCES_PATH, {"sources": []})
sources: List[Source] = []
for item in payload.get("sources", []) if isinstance(payload, dict) else []:
if not isinstance(item, dict):
continue
label = str(item.get("label") or "session")
p = item.get("path")
if not isinstance(p, str) or not p:
continue
sources.append(Source(label=label, transcript_path=Path(p)))
return sources
def _iter_new_lines(path: Path, *, start_offset: int) -> Iterable[tuple[int, str]]:
with open(path, "rb") as f:
f.seek(max(0, int(start_offset)))
while True:
raw = f.readline()
if not raw:
break
line = raw.decode("utf-8", errors="ignore").strip()
if not line:
continue
yield (f.tell(), line)
def run() -> Dict[str, Any]:
if not DB_PATH.exists():
return {"success": False, "time": _now(), "error": f"db missing: {DB_PATH}"}
sources = _load_sources()
state = _load_json(STATE_PATH, {"offsets": {}})
offsets: Dict[str, int] = state.get("offsets", {}) if isinstance(state, dict) else {}
store = EngramStore(str(DB_PATH))
out = {"success": True, "time": _now(), "sources": len(sources), "messages_saved": 0, "messages_skipped": 0, "errors": []}
for src in sources:
try:
if not src.transcript_path.exists():
continue
key = str(src.transcript_path)
start = int(offsets.get(key, 0))
for new_off, line in _iter_new_lines(src.transcript_path, start_offset=start):
offsets[key] = new_off
obj = json.loads(line)
if not isinstance(obj, dict) or obj.get("type") != "message":
continue
msg = obj.get("message") if isinstance(obj.get("message"), dict) else {}
role = str(msg.get("role") or "unknown")
content = _extract_text(msg.get("content"))
if len(content.strip()) < 5:
continue
mid = str(obj.get("id") or msg.get("id") or msg.get("messageId") or msg.get("message_id") or "")
if not mid:
mid = str(uuid5(NAMESPACE_URL, f"openclaw-transcript:{src.label}:{role}:{content[:200]}"))
eid = str(uuid5(NAMESPACE_URL, f"openclaw-transcript:{src.label}:{mid}"))
if store.get(eid):
out["messages_skipped"] += 1
continue
eg = Engram.create(
content=f"[transcript:{src.label}] [{role}] [{mid}]\n\n{content}"[:4000],
source="session",
tags=["session", "openclaw", "transcript", f"role:{role}"],
session_id=src.label,
confidence=0.55,
grounding=Grounding.ASSUMPTION,
)
eg.id = uuid5(NAMESPACE_URL, eid)
eg.metadata["source"] = "session"
eg.metadata["session_id"] = src.label
eg.metadata["role"] = role
eg.metadata["message_id"] = mid
store.save(eg)
out["messages_saved"] += 1
except Exception as e:
out["success"] = False
out["errors"].append(f"{src.transcript_path}: {e}")
_save_json(STATE_PATH, {"offsets": offsets, "updated_at": out["time"]})
return out
if __name__ == "__main__":
print(json.dumps(run(), ensure_ascii=False, indent=2))

View File

@@ -0,0 +1,213 @@
#!/usr/bin/env python3
"""
Tail OpenClaw session transcript JSONL files and append new messages into
workspace/memory/YYYY-MM-DD.md so the existing ingest_memory pipeline can pick
them up.
Why: when chat_autosave hooks are missed/aborted, the "memory/*.md -> DB" ingest
doesn't see the latest conversation. This bridges transcript -> memory.
Safety: read-only access to transcript files; state stored in workspace/memory/.
"""
from __future__ import annotations
import json
import os
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Dict, Iterable, List, Optional
try:
from zoneinfo import ZoneInfo # py3.9+
except Exception: # pragma: no cover
ZoneInfo = None # type: ignore
WORKSPACE = Path("/root/.openclaw/workspace")
MEMORY_DIR = WORKSPACE / "memory"
SOURCES_PATH = MEMORY_DIR / "session_sources.json"
STATE_PATH = MEMORY_DIR / "session_ingest_state.json"
def _local_tz():
tz = os.environ.get("TZ") or "Europe/Berlin"
if ZoneInfo:
try:
return ZoneInfo(tz)
except Exception:
return timezone.utc
return timezone.utc
def _load_json(path: Path, default: Any) -> Any:
try:
if not path.exists():
return default
return json.loads(path.read_text(encoding="utf-8"))
except Exception:
return default
def _save_json(path: Path, payload: Any) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(payload, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
def _iso_to_dt(ts: str) -> Optional[datetime]:
try:
if ts.endswith("Z"):
ts = ts[:-1] + "+00:00"
return datetime.fromisoformat(ts)
except Exception:
return None
def _ms_to_dt(ms: int) -> datetime:
return datetime.fromtimestamp(ms / 1000, tz=timezone.utc)
def _extract_text(content: Any) -> str:
if isinstance(content, str):
return content.strip()
if isinstance(content, list):
parts: List[str] = []
for c in content:
if isinstance(c, dict) and c.get("type") == "text" and isinstance(c.get("text"), str):
parts.append(c["text"])
elif isinstance(c, str):
parts.append(c)
return "\n".join(p.strip() for p in parts if p and p.strip()).strip()
if isinstance(content, dict):
if isinstance(content.get("text"), str):
return content["text"].strip()
return str(content).strip()
@dataclass
class Source:
label: str
transcript_path: Path
def _load_sources() -> List[Source]:
"""
Sources file format:
{
"sources": [
{ "label": "telegram:263887248", "path": "/root/.openclaw/agents/main/sessions/<id>.jsonl" }
]
}
"""
payload = _load_json(SOURCES_PATH, {"sources": []})
sources: List[Source] = []
for item in payload.get("sources", []) if isinstance(payload, dict) else []:
if not isinstance(item, dict):
continue
label = str(item.get("label") or "session")
p = item.get("path")
if not isinstance(p, str) or not p:
continue
sources.append(Source(label=label, transcript_path=Path(p)))
return sources
def _memory_path_for(dt: datetime) -> Path:
tz = _local_tz()
local = dt.astimezone(tz)
MEMORY_DIR.mkdir(parents=True, exist_ok=True)
return MEMORY_DIR / f"{local.date().isoformat()}.md"
def _append_memory(dt: datetime, label: str, role: str, text: str) -> None:
if not text.strip():
return
tz = _local_tz()
local = dt.astimezone(tz)
mem = _memory_path_for(dt)
if not mem.exists():
mem.write_text(f"# {local.date().isoformat()}\n\n", encoding="utf-8")
header = f"## {local.strftime('%H:%M:%S')} - {label} ({role})"
body = text.strip()
mem.write_text(mem.read_text(encoding="utf-8") + f"{header}\n\n{body}\n\n", encoding="utf-8")
def _iter_new_lines(path: Path, *, start_offset: int) -> Iterable[tuple[int, str]]:
with open(path, "rb") as f:
f.seek(max(0, int(start_offset)))
while True:
raw = f.readline()
if not raw:
break
try:
line = raw.decode("utf-8", errors="ignore").strip()
except Exception:
line = ""
if not line:
continue
yield (f.tell(), line)
def run() -> Dict[str, Any]:
sources = _load_sources()
state = _load_json(STATE_PATH, {"offsets": {}})
offsets: Dict[str, int] = state.get("offsets", {}) if isinstance(state, dict) else {}
out = {
"success": True,
"time": datetime.now(timezone.utc).isoformat(),
"sources": len(sources),
"messages_appended": 0,
"errors": [],
}
for src in sources:
try:
if not src.transcript_path.exists():
continue
key = str(src.transcript_path)
start = int(offsets.get(key, 0))
for new_off, line in _iter_new_lines(src.transcript_path, start_offset=start):
try:
obj = json.loads(line)
except Exception:
offsets[key] = new_off
continue
if not isinstance(obj, dict) or obj.get("type") != "message":
offsets[key] = new_off
continue
msg = obj.get("message") if isinstance(obj.get("message"), dict) else {}
role = str(msg.get("role") or "unknown")
content = msg.get("content")
text = _extract_text(content)
dt = None
if isinstance(msg.get("timestamp"), (int, float)):
dt = _ms_to_dt(int(msg["timestamp"]))
elif isinstance(obj.get("timestamp"), str):
dt = _iso_to_dt(obj["timestamp"])
if dt is None:
dt = datetime.now(timezone.utc)
if len(text.strip()) < 5:
offsets[key] = new_off
continue
_append_memory(dt, src.label, role, text)
out["messages_appended"] += 1
offsets[key] = new_off
except Exception as e:
out["success"] = False
out["errors"].append(f"{src.transcript_path}: {e}")
_save_json(STATE_PATH, {"offsets": offsets, "updated_at": out["time"]})
return out
if __name__ == "__main__":
res = run()
print(json.dumps(res, ensure_ascii=False, indent=2))

View File

@@ -0,0 +1,169 @@
#!/usr/bin/env python3
"""
Verify pending (unconfirmed) engrams using lightweight external checks.
Policy (conservative):
- `openclaw-memory` is treated as internal ground-truth and is auto-confirmed
by the review job (see `cron_tasks/review_brain.py` in the workspace runtime).
- For `source=web`, confirm if the grounded URL responds with HTTP 2xx, reject on
4xx/5xx, and keep pending on timeouts/unknown.
- Reject obvious low-signal placeholders (e.g. session summary stubs).
"""
import json
import os
import sys
from pathlib import Path
from datetime import datetime, timezone
from typing import Any, Optional
WORKSPACE = Path(os.environ.get("SECOND_BRAIN_WORKSPACE", "/root/.openclaw/workspace/second-brain"))
DB_PATH = Path(os.environ.get("BRAIN_DB", str(WORKSPACE / "data" / "brain.sqlite"))).resolve()
sys.path.insert(0, str(WORKSPACE))
from src.store import EngramStore
from src.engram import ReviewEntry
OUTPUT_FILE = os.environ.get("CRON_OUTPUT_FILE", "/tmp/verify_pending_external.json")
def _now() -> str:
return datetime.now(timezone.utc).isoformat()
def _get_url(meta: dict[str, Any]) -> Optional[str]:
url = meta.get("url")
if isinstance(url, str) and url.startswith(("http://", "https://")):
return url
grounding = meta.get("grounding")
if isinstance(grounding, dict):
g_url = grounding.get("url")
if isinstance(g_url, str) and g_url.startswith(("http://", "https://")):
return g_url
return None
def _http_status(url: str, timeout_s: float = 6.0) -> Optional[int]:
try:
import urllib.request
req = urllib.request.Request(
url,
method="GET",
headers={"User-Agent": "openclaw-secondbrain/verify_pending_external"},
)
with urllib.request.urlopen(req, timeout=timeout_s) as resp:
return int(getattr(resp, "status", 200))
except Exception:
return None
def main() -> int:
if not DB_PATH.exists():
out = {"success": False, "error": f"db missing: {DB_PATH}", "time": _now()}
Path(OUTPUT_FILE).write_text(json.dumps(out, indent=2))
print(out["error"])
return 1
store = EngramStore(str(DB_PATH))
all_egs = []
offset = 0
while True:
batch = store.get_all(limit=2000, offset=offset)
if not batch:
break
all_egs.extend(batch)
offset += len(batch)
pending = [
eg
for eg in all_egs
if (not eg.correctness.confirmed and eg.correctness.rejections == 0)
]
confirmed = 0
rejected = 0
still_pending = 0
checked = 0
for eg in pending:
checked += 1
src = eg.metadata.get("source")
content = (eg.content or "").strip()
if src == "session" and (
content.startswith("Session Summary (sess_") or content.startswith("Please remember ")
):
eg.correctness.rejections += 1
eg.correctness.last_reviewed = _now()
eg.correctness.review_history.append(
ReviewEntry(
by="verify-pending",
action="reject",
at=_now(),
note="Auto-reject: session placeholder",
)
)
store.save(eg)
rejected += 1
continue
if src == "web":
url = _get_url(eg.metadata)
if not url:
still_pending += 1
continue
status = _http_status(url)
if status is None:
still_pending += 1
continue
if 200 <= status < 300:
eg.correctness.confirmed = True
eg.correctness.confirmations += 1
eg.correctness.last_reviewed = _now()
eg.correctness.review_history.append(
ReviewEntry(
by="verify-pending",
action="confirm",
at=_now(),
note=f"Auto-confirm: web url ok ({status}) {url}",
)
)
store.save(eg)
confirmed += 1
else:
eg.correctness.rejections += 1
eg.correctness.last_reviewed = _now()
eg.correctness.review_history.append(
ReviewEntry(
by="verify-pending",
action="reject",
at=_now(),
note=f"Auto-reject: web url status={status} {url}",
)
)
store.save(eg)
rejected += 1
continue
still_pending += 1
out = {
"success": True,
"time": _now(),
"total": len(all_egs),
"pending_before": len(pending),
"checked": checked,
"confirmed": confirmed,
"rejected": rejected,
"still_pending": still_pending,
}
Path(OUTPUT_FILE).write_text(json.dumps(out, indent=2))
print(
f"VERIFY: pending_before={out['pending_before']} confirmed={confirmed} rejected={rejected} still_pending={still_pending}"
)
return 0
if __name__ == "__main__":
raise SystemExit(main())

75
docs/OBSIDIAN.md Normal file
View File

@@ -0,0 +1,75 @@
# Obsidian Coupling (Second-Brain 2.0)
This integrates an Obsidian vault with Second-Brain via two cron tasks:
- `cron_tasks/ingest_obsidian.py` (vault → Second-Brain)
- `cron_tasks/export_obsidian.py` (Second-Brain → vault)
All settings live in `second-brain/data/obsidian_config.json`.
## 1) Install / Sync the vault to the server
You need a local folder on the server that contains an Obsidian vault (it must contain a `.obsidian/` directory), e.g.:
- `/srv/obsidian/MyVault`
- `/data/obsidian/MyVault`
- `/root/Obsidian/MyVault`
How you sync it is up to you (Syncthing, rsync, SMB mount, etc.).
## 2) Set `vault_path` in config (auto or manual)
### Auto-discover (only writes if unambiguous)
```bash
python3 second-brain/scripts/discover_obsidian_vault.py
python3 second-brain/scripts/discover_obsidian_vault.py --write
```
If multiple vaults are detected, it prints them and refuses to write.
### Manual
Edit `second-brain/data/obsidian_config.json` and set:
- `vault_path` to the vault directory (the parent of `.obsidian/`)
## 3) Enable ingest/export
In `second-brain/data/obsidian_config.json`:
- Set `enabled.ingest` to `true` to ingest vault markdown into Second-Brain
- Set `enabled.export` to `true` to export Second-Brain engrams into the vault
## 4) Enable timers (systemd)
This repo ships unit files in `systemd/`:
- `systemd/openclaw-secondbrain-ingest-obsidian.service`
- `systemd/openclaw-secondbrain-ingest-obsidian.timer`
- `systemd/openclaw-secondbrain-export-obsidian.service`
- `systemd/openclaw-secondbrain-export-obsidian.timer`
Install them (copy or symlink) to `/etc/systemd/system/`, then:
```bash
sudo systemctl daemon-reload
sudo systemctl enable --now openclaw-secondbrain-ingest-obsidian.timer
sudo systemctl enable --now openclaw-secondbrain-export-obsidian.timer
```
## 5) Verify
Run once manually:
```bash
python3 openclaw_cron_wrapper.py ingest_obsidian
python3 openclaw_cron_wrapper.py export_obsidian
```
What to expect:
- If `vault_path` is missing/invalid, both tasks **skip** safely (no writes to random paths).
- Ingest creates/updates `second-brain/data/obsidian_ingest_state.json`.
- Export writes markdown files to `<vault_path>/<export.subdir>/` (default: `SecondBrain/`) and tracks state in `second-brain/data/obsidian_export_state.json`.

113
docs/RELEASE_CHECKLIST.md Normal file
View File

@@ -0,0 +1,113 @@
# Second-Brain 2.0 — Release Checklist (Grundversion + FastAPI)
Scope: OpenClaw workspace at `/root/.openclaw/workspace` with `second-brain/` and the shipped `second-brain/systemd/` unit files.
This checklist is copy/paste friendly. Run as a user with `sudo` (systemd commands need it).
## 0) Preconditions (one-time per host)
### Repo + Python sanity
```bash
cd /root/.openclaw/workspace
test -d second-brain || { echo "missing: second-brain/"; exit 1; }
python3 --version
```
### Ensure systemd units are installed (copy/symlink)
```bash
ls -la /etc/systemd/system/openclaw-secondbrain-*.timer /etc/systemd/system/openclaw-secondbrain-*.service 2>/dev/null || true
ls -la /etc/systemd/system/openclaw-memory-archive.* 2>/dev/null || true
```
If missing, install them (symlink is fine; copy is fine too):
```bash
sudo ln -sf /root/.openclaw/workspace/second-brain/systemd/openclaw-secondbrain-*.service /etc/systemd/system/
sudo ln -sf /root/.openclaw/workspace/second-brain/systemd/openclaw-secondbrain-*.timer /etc/systemd/system/
sudo ln -sf /root/.openclaw/workspace/second-brain/systemd/openclaw-memory-archive.* /etc/systemd/system/
sudo ln -sf /root/.openclaw/workspace/second-brain/systemd/openclaw-secondbrain-dashboard.service /etc/systemd/system/
sudo systemctl daemon-reload
```
Optional (verification hardening):
```bash
sudo ln -sf /root/.openclaw/workspace/second-brain/systemd/openclaw-secondbrain-verify-pending.* /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now openclaw-secondbrain-verify-pending.timer
```
### Enable timers
```bash
sudo systemctl enable --now openclaw-secondbrain-backup.timer
sudo systemctl enable --now openclaw-secondbrain-ingest-memory.timer
sudo systemctl enable --now openclaw-secondbrain-index-vectors.timer
sudo systemctl enable --now openclaw-secondbrain-review.timer
sudo systemctl enable --now openclaw-secondbrain-heartbeat.timer
sudo systemctl enable --now openclaw-secondbrain-proactive-search.timer
sudo systemctl enable --now openclaw-memory-archive.timer
```
## 1) Release QA — systemd status + timers
### Verify timers are active and scheduled
```bash
sudo systemctl list-timers --all | grep -E 'openclaw-(secondbrain|memory-archive)' || true
sudo systemctl --failed --no-pager || true
```
### Verify the oneshot services can run successfully (manual trigger)
```bash
sudo systemctl start openclaw-secondbrain-ingest-memory.service
sudo systemctl start openclaw-secondbrain-index-vectors.service
sudo systemctl start openclaw-secondbrain-review.service
sudo systemctl start openclaw-secondbrain-backup.service
sudo systemctl start openclaw-secondbrain-heartbeat.service
sudo systemctl start openclaw-secondbrain-proactive-search.service
sudo systemctl start openclaw-memory-archive.service
```
Logs:
```bash
tail -n 200 /root/.openclaw/workspace/cron_wrapper.log
sudo journalctl -u openclaw-secondbrain-review.service -n 200 --no-pager
```
## 2) Release QA — data + DB invariants
```bash
ls -la /root/.openclaw/workspace/second-brain/data/brain.sqlite
python3 - <<'PY'
import sqlite3
db="/root/.openclaw/workspace/second-brain/data/brain.sqlite"
con=sqlite3.connect(db)
cur=con.cursor()
print("integrity_check:", cur.execute("PRAGMA integrity_check").fetchone()[0])
print("engrams:", cur.execute("SELECT COUNT(*) FROM engrams").fetchone()[0])
con.close()
PY
```
## 3) Release QA — FastAPI (HTTP endpoints + logs)
FastAPI service:
```bash
sudo systemctl enable --now openclaw-secondbrain-dashboard.service
sudo systemctl status openclaw-secondbrain-dashboard.service --no-pager
```
Endpoint checks:
```bash
curl -fsS http://127.0.0.1:8501/healthz && echo
curl -fsS http://127.0.0.1:8501/api/config
curl -fsS http://127.0.0.1:8501/api/stats
curl -fsS "http://127.0.0.1:8501/api/engrams?limit=1&offset=0"
```

View File

@@ -0,0 +1,215 @@
# Second-Brain Dashboard UI/UX Design Plan (mobile-first)
Basis: `second-brain/templates/dashboard.html`, `second-brain/static/style.css` (aktueller Stand) + neue Graph-Controls aus `/tmp/second-brain-staging/`.
## 1) Konkreter Design-Plan
### Ziele (UX)
- **Schnelles Triaging unterwegs:** Pending/Errors sofort sichtbar, 1-Hand-Bedienung.
- **Konsistentes Design-System:** einheitliche Buttons/Inputs/Panels statt Einzellösungen.
- **Graph als Diagnose-Tool:** klare Controls, Legende, nachvollziehbares Feedback (Loading/Empty/Errors).
### Farbschema (Dark, high-contrast, "indigo + emerald")
- **Background:** sehr dunkel (nahe #0f1117) für weniger Blendung.
- **Surface (Cards/Panels):** abgestufte Flächen (Surface-1, Surface-2) für Hierarchie.
- **Primary:** Indigo/Blue für interaktive Elemente und Highlights (bisher #6c8af5 bleibt als Basis).
- **Success:** Emerald für Confirm/OK (bisher #3a7d3a → etwas heller/satter).
- **Danger:** Red für Errors/Reject.
- **Warning:** Amber für Pending/Match/Attention.
- **Text:** fast-weiß, sekundär gedimmt.
### Typografie
- Systemfont-Stack (schnell, gut lesbar): `ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, Arial`.
- Skala (mobile-first):
- `text-xs` 12px (Labels, Meta)
- `text-sm` 1314px (secondary)
- `text-md` 1516px (Body)
- `text-lg` 1820px (Titles)
- **Ziffern:** optional `font-variant-numeric: tabular-nums;` für Stats.
### Spacing/Rhythm
- 4px Grid; Standard-Gaps: 8/12/16.
- Container-Padding: 12px (mobile), 1620px (>=768px).
- Border-radius: 1216px (Cards/Modals), 10px (Inputs/Buttons), 999px (Pills).
### Komponenten (konkret)
#### A) Tabs
- Tabs bleiben als 3 Buttons, aber:
- **Active State** deutlicher (background + border + subtle glow).
- **Touch target** min. 44px Höhe.
- Sticky bleibt, aber mit leichter **Backdrop-Blurring** (wenn möglich) oder solid surface.
#### B) Search
- Search-Row wird als **kompakte Toolbar** gestaltet:
- Input + Filter in einer Zeile.
- Optional Quick-Chips darunter: `All / Pending / Confirmed / Errors` (klickbar) als Alternative zum Select.
- Clear-Button (×) im Input (per CSS `::-webkit-search-cancel-button` oder eigener Button) für Mobile.
#### C) Cards
- Karte als 3 Zonen: Header (badges/tags/date), Body (content), Footer (actions).
- Status wird stärker codiert:
- left-border + kleine **Status-Pill** (OK/Pending/Error) mit eindeutiger Farbe.
- Body: bessere Lesbarkeit via `line-height: 1.55` und max-height/clamp optional.
#### D) Modal
- Modal als **Bottom Sheet** auf Mobile (>=50vh) + klassisches Center-Modal auf Desktop.
- Close-Button größer + Tap-Area.
- Inhalt in Tabs/Sections (History/Meta/Content) optional später.
#### E) Graph + Controls (aus Staging)
- Controls als **Control Bar** oberhalb Canvas:
- Primary: Physics toggle.
- Secondary: Fit, Reset, Reload.
- Text labels kurz (z.B. `Physics` statt `Physics: off`, state als Badge).
- Canvas passt sich an Viewport an:
- `width: min(100%, 560px)`; Height: `min(65vh, 560px)` (CSS statt fester HTML Attribute, wenn möglich).
- Legende als einklappbares Panel (`Details`/`summary`) oder leichtes Panel unter Canvas.
#### F) Status Panels
- Status-View nutzt vorhandene `.panel`/`.kv-*`:
- Gruppen: System, Storage, Jobs, Insights, Pending Queue.
- Jede Gruppe als Panel mit klarer Title Row.
- Kritische Werte (Errors/Pending/Queue) farblich markieren.
## 2) CSS-Variablen (Theme-Tokens) + Mapping
### Token-Vorschlag (`:root`)
```css
:root {
/* typography */
--font-sans: ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, Arial, sans-serif;
/* colors */
--bg: #0f1117;
--surface-1: #14151d;
--surface-2: #1a1b26;
--border: #2a2d3a;
--text: #e8e8ee;
--text-muted: #8b90a3;
--text-dim: #5c6276;
--primary: #6c8af5; /* existing */
--primary-2: #8aa1ff;
--success: #2fbf71;
--danger: #ef4444;
--warning: #f7d154; /* existing-ish */
/* shadows */
--shadow-1: 0 1px 0 rgba(0,0,0,.25), 0 8px 24px rgba(0,0,0,.35);
/* radius */
--r-sm: 10px;
--r-md: 14px;
--r-lg: 16px;
/* spacing */
--s-1: 4px;
--s-2: 8px;
--s-3: 12px;
--s-4: 16px;
--s-5: 20px;
/* control sizes */
--tap: 44px;
}
```
### Mapping auf existierende Selektoren
- `body`/`.app`
- `background: var(--bg)` statt `#141419`
- `font-family: var(--font-sans)`
- `color: var(--text)`
- `.stats-bar`, `.tabs-bar`, `.search-box`
- `background: var(--surface-1)` (Stats ggf. Gradient bleibt, aber auf Tokens)
- `border-bottom: 1px solid var(--border)`
- `.panel`, `.card`, `.modal-content`, `.graph-legend`
- `background: var(--surface-2)`
- `border: 1px solid var(--border)`
- `border-radius: var(--r-md)`
- `.stat-num`, `#pageNum`, `.tab-btn.active`, `#searchInput:focus`
- `color/border-color: var(--primary)`
- `.muted`, `.stat-label`, `.kv-key`, `.date`
- `color: var(--text-muted)` bzw. `var(--text-dim)`
- Buttons
- vereinheitlichen über `.btn` + Modifier: `.btn.primary`, `.btn.danger`, `.btn.success`, `.btn.ghost`
- `min-height: var(--tap)` für Touch
## 3) 510 priorisierte UI-Änderungen (mit Begründung)
1) **Design Tokens (CSS vars) einführen** → reduziert Farbmix, erleichtert spätere Themes/Anpassungen.
2) **Einheitliche Button-Komponente (`.btn`)** (inkl. `:active`, `:disabled`, `min-height`) → bessere Touch-UX, konsistente Interaktion.
3) **Graph-Controls + Legende aus Staging in den Main-Template-Stand ziehen** → Graph wird tatsächlich bedienbar/selbsterklärend.
4) **Responsive Graph-Canvas (CSS gesteuert)** statt fixer `width/height` → bessere Nutzung auf Phones, weniger Scroll.
5) **Search als Toolbar + Clear-Action** → schnelleres Filtern unterwegs, weniger Friktion.
6) **Modal als Bottom-Sheet auf Mobile** → angenehmer für längeren Content + History, weniger „winziges Fenster“.
7) **Status/Health Werte farblich akzentuieren** (pending/errors/warn) → schnelleres Erkennen von Problemen.
8) **Cards: Status-Pill + typografische Lesbarkeit** (line-height, spacing) → weniger „Textblock“, bessere Scanbarkeit.
9) **Accessibility-Basics**: Focus-Rings, Kontrast, `prefers-reduced-motion` → weniger „invisible focus“ und bessere Bedienbarkeit.
10) **Top-level Layout Max-Width für Desktop** (z.B. 560720px) → verhindert „zu breite“ Zeilen.
## 4) Optionale Patch-Vorschläge (Diff-Snippets, NICHT anwenden)
> Hinweis: Snippets sind bewusst klein gehalten. Gesamt < 120 Zeilen.
### Snippet A — Tokens + Button-System (style.css)
```diff
--- a/second-brain/static/style.css
+++ b/second-brain/static/style.css
@@
+:root {
+ --font-sans: ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, Arial, sans-serif;
+ --bg:#0f1117; --surface-1:#14151d; --surface-2:#1a1b26; --border:#2a2d3a;
+ --text:#e8e8ee; --text-muted:#8b90a3; --text-dim:#5c6276;
+ --primary:#6c8af5; --success:#2fbf71; --danger:#ef4444; --warning:#f7d154;
+ --r-sm:10px; --r-md:14px; --r-lg:16px;
+ --s-2:8px; --s-3:12px; --s-4:16px;
+ --tap:44px;
+}
+
+body { font-family: var(--font-sans); background: var(--bg); color: var(--text); }
+
+.btn{
+ min-height: var(--tap);
+ background: #1e1e28;
+ border: 1px solid var(--border);
+ border-radius: var(--r-sm);
+ padding: 8px 12px;
+ color: #cfd3ff;
+ font-weight: 700;
+}
+.btn.primary{ border-color: var(--primary); box-shadow: 0 0 0 1px rgba(108,138,245,0.18) inset; }
+.btn.success{ background: rgba(47,191,113,.18); border-color: rgba(47,191,113,.35); }
+.btn.danger{ background: rgba(239,68,68,.16); border-color: rgba(239,68,68,.35); }
+.btn:active{ transform: scale(.98); }
+.btn:disabled{ opacity: .45; }
```
### Snippet B — Graph Controls + Legend übernehmen (dashboard.html)
```diff
--- a/second-brain/templates/dashboard.html
+++ b/second-brain/templates/dashboard.html
@@
- <div class="graph" id="graph" style="display:none;">
- <canvas id="graphCanvas" width="440" height="520"></canvas>
- <div class="muted small" id="graphHint">Lade Graph…</div>
- </div>
+ <div class="graph" id="graph" style="display:none;">
+ <div class="graph-controls">
+ <button class="btn primary" id="btnGraphPhysics" onclick="toggleGraphPhysics()">Physics</button>
+ <button class="btn" onclick="resetGraphView()">Reset</button>
+ <button class="btn" onclick="fitGraphView()">Fit</button>
+ <button class="btn" onclick="reloadGraph()">Reload</button>
+ </div>
+ <canvas id="graphCanvas" width="440" height="520"></canvas>
+ <div class="graph-hint muted small" id="graphHint">Lade Graph…</div>
+ <div class="graph-legend">
+ <div><strong>Graph</strong>: Zoom (Wheel/Pinch), Pan (Drag). Klick auf Engram öffnet Details, Klick auf Tag setzt Suche.</div>
+ <div class="legend-row"><span class="legend-dot engram"></span> Engram</div>
+ <div class="legend-row"><span class="legend-dot tag"></span> Tag</div>
+ <div class="legend-row"><span class="legend-dot match"></span> Match</div>
+ </div>
+ </div>
```

838
fastapi_app.py Normal file
View File

@@ -0,0 +1,838 @@
#!/usr/bin/env python3
"""
Second Brain FastAPI Dashboard
Goals:
- "Release-ready" defaults (no hardcoded absolute paths)
- Minimal config via env vars
- Serves the existing static dashboard (templates/dashboard.html + static/)
"""
import json
import os
import sqlite3
import subprocess
from datetime import datetime, timezone
from pathlib import Path
from uuid import uuid4
from urllib.parse import urlparse
from fastapi import FastAPI, Form, Query, Request
from fastapi.responses import HTMLResponse, JSONResponse, PlainTextResponse, StreamingResponse
from fastapi.staticfiles import StaticFiles
# ─── Config ──────────────────────────────────────────────────────────────────
REPO_ROOT = Path(__file__).resolve().parent
WORKSPACE = Path(os.environ.get("SECOND_BRAIN_WORKSPACE", str(REPO_ROOT))).resolve()
DB_PATH = Path(os.environ.get("SECOND_BRAIN_DB_PATH", str(WORKSPACE / "data" / "brain.sqlite"))).resolve()
PORT = int(os.environ.get("SECOND_BRAIN_PORT", os.environ.get("PORT", "8501")))
HOST = os.environ.get("SECOND_BRAIN_HOST", "0.0.0.0")
def create_app() -> FastAPI:
app = FastAPI(title="Second Brain Dashboard")
static_dir = WORKSPACE / "static"
if static_dir.is_dir():
app.mount("/static", StaticFiles(directory=str(static_dir)), name="static")
return app
app = create_app()
# ─── Helpers ─────────────────────────────────────────────────────────────────
def get_db():
if not DB_PATH.exists():
raise FileNotFoundError(f"DB not found: {DB_PATH}")
conn = sqlite3.connect(str(DB_PATH))
conn.row_factory = sqlite3.Row
return conn
def parse_engram(row: sqlite3.Row) -> dict:
meta = json.loads(row["metadata_json"] or "{}")
correctness = json.loads(row["correctness_json"] or "{}")
return {
"id": row["id"],
"content": row["content"],
"confidence": meta.get("confidence", 0.0),
"confirmed": correctness.get("confirmed", False),
"confirmations": correctness.get("confirmations", 0),
"rejections": correctness.get("rejections", 0),
"tags": meta.get("tags", []),
"created": meta.get("created", row["created_at"]),
"modified": meta.get("modified", row["modified_at"]),
"last_reviewed": correctness.get("last_reviewed"),
"review_history": correctness.get("review_history", []),
"source": meta.get("source", "unknown"),
"access_count": meta.get("access_count", 0),
"grounding": meta.get("grounding", 0),
}
def _now_iso() -> str:
return datetime.now(timezone.utc).isoformat()
def _update_correctness(engram_id: str, *, action: str, reason: str | None = None) -> dict:
"""
Update correctness_json for an engram. action: confirm|reject
"""
conn = get_db()
c = conn.cursor()
row = c.execute("SELECT correctness_json FROM engrams WHERE id = ?", (engram_id,)).fetchone()
if not row:
conn.close()
raise FileNotFoundError(f"Engram not found: {engram_id}")
corr = json.loads(row["correctness_json"] or "{}")
corr.setdefault("confirmed", False)
corr.setdefault("confirmations", 0)
corr.setdefault("rejections", 0)
corr.setdefault("review_history", [])
corr["last_reviewed"] = _now_iso()
entry = {
"by": "dashboard",
"action": action,
"at": corr["last_reviewed"],
"note": (reason or "").strip(),
}
try:
corr["review_history"].append(entry)
except Exception:
corr["review_history"] = [entry]
if action == "confirm":
corr["confirmed"] = True
corr["confirmations"] = int(corr.get("confirmations", 0) or 0) + 1
elif action == "reject":
corr["rejections"] = int(corr.get("rejections", 0) or 0) + 1
c.execute(
"UPDATE engrams SET correctness_json = ?, modified_at = ? WHERE id = ?",
(json.dumps(corr, ensure_ascii=False), corr["last_reviewed"], engram_id),
)
conn.commit()
conn.close()
return {"ok": True}
def _bump_access(engram_id: str) -> dict:
conn = get_db()
c = conn.cursor()
row = c.execute("SELECT metadata_json FROM engrams WHERE id = ?", (engram_id,)).fetchone()
if not row:
conn.close()
raise FileNotFoundError(f"Engram not found: {engram_id}")
meta = json.loads(row["metadata_json"] or "{}")
meta["access_count"] = int(meta.get("access_count", 0) or 0) + 1
meta["last_accessed"] = _now_iso()
c.execute(
"UPDATE engrams SET metadata_json = ?, modified_at = ? WHERE id = ?",
(json.dumps(meta, ensure_ascii=False), meta["last_accessed"], engram_id),
)
conn.commit()
conn.close()
return {"ok": True}
def _safe_json_extract_tags(meta_json: str) -> list[str]:
try:
d = json.loads(meta_json or "{}")
tags = d.get("tags") or []
return [t for t in tags if isinstance(t, str)]
except Exception:
return []
def _host_from_meta(meta_json: str) -> str | None:
try:
d = json.loads(meta_json or "{}")
grounding = d.get("grounding")
url = d.get("url")
if isinstance(grounding, dict) and isinstance(grounding.get("url"), str):
url = grounding.get("url")
if not isinstance(url, str):
return None
parsed = urlparse(url)
return parsed.hostname
except Exception:
return None
def _systemd_unit_state(unit: str) -> dict:
"""
Best-effort systemd status snapshot for a known unit.
Never raises; returns minimal fields.
"""
try:
out = subprocess.check_output(
["systemctl", "show", unit, "--no-page", "--property=ActiveState,SubState,Result,ExecMainStatus,ExecMainCode,ExecMainStartTimestamp,ExecMainExitTimestamp"],
text=True,
stderr=subprocess.STDOUT,
timeout=2,
)
kv = {}
for line in out.splitlines():
if "=" in line:
k, v = line.split("=", 1)
kv[k] = v
return {
"unit": unit,
"active": kv.get("ActiveState"),
"sub": kv.get("SubState"),
"result": kv.get("Result"),
"exit_status": kv.get("ExecMainStatus"),
"start_ts": kv.get("ExecMainStartTimestamp"),
"exit_ts": kv.get("ExecMainExitTimestamp"),
}
except Exception as e:
return {"unit": unit, "error": str(e)}
def _dir_size_bytes(path: Path) -> int:
total = 0
try:
for p in path.rglob("*"):
try:
if p.is_file():
total += p.stat().st_size
except Exception:
pass
except Exception:
pass
return total
# ─── API Endpoints ───────────────────────────────────────────────────────────
@app.get("/healthz", response_class=PlainTextResponse)
def healthz():
return "ok"
@app.get("/api/config")
def api_config():
return {
"workspace": str(WORKSPACE),
"db_path": str(DB_PATH),
}
@app.get("/api/db_info")
def api_db_info():
if not DB_PATH.exists():
raise FileNotFoundError(f"DB not found: {DB_PATH}")
st = DB_PATH.stat()
return {
"db_path": str(DB_PATH),
"size_bytes": st.st_size,
"mtime": datetime.fromtimestamp(st.st_mtime, tz=timezone.utc).isoformat(),
}
@app.get("/api/storage_stats")
def api_storage_stats():
conn = get_db()
c = conn.cursor()
total = c.execute("SELECT COUNT(*) FROM engrams").fetchone()[0]
confirmed = c.execute(
"SELECT COUNT(*) FROM engrams WHERE json_extract(correctness_json, '$.confirmed') = 1"
).fetchone()[0]
sources = {
r[0]: r[1]
for r in c.execute(
"SELECT json_extract(metadata_json, '$.source') AS src, COUNT(*) FROM engrams GROUP BY src ORDER BY COUNT(*) DESC"
).fetchall()
if r[0] is not None
}
conn.close()
chroma_dir = WORKSPACE / "data" / "chroma"
emb_cache_dir = WORKSPACE / "data" / "embedding_cache"
vec_state_path = WORKSPACE / "data" / "vector_index_state.json"
vec_state = {}
if vec_state_path.exists():
try:
vec_state = json.loads(vec_state_path.read_text())
except Exception:
vec_state = {}
obsidian_cfg_path = WORKSPACE / "data" / "obsidian_config.json"
obsidian_cfg = None
if obsidian_cfg_path.exists():
try:
obsidian_cfg = json.loads(obsidian_cfg_path.read_text())
except Exception:
obsidian_cfg = {"raw": obsidian_cfg_path.read_text()[:2000]}
backup_files = sorted((WORKSPACE / "data").glob("backup_*.jsonl"))
return {
"sql": {
"total_engrams": total,
"confirmed": confirmed,
"pending": total - confirmed,
"by_source": sources,
},
"vector": {
"chroma_dir": str(chroma_dir),
"chroma_size_bytes": _dir_size_bytes(chroma_dir) if chroma_dir.exists() else 0,
"embedding_cache_dir": str(emb_cache_dir),
"embedding_cache_files": len(list(emb_cache_dir.glob("*.json"))) if emb_cache_dir.exists() else 0,
"vector_state": vec_state,
},
"obsidian": {
"config_path": str(obsidian_cfg_path),
"configured": bool(obsidian_cfg),
"config": obsidian_cfg,
},
"backups": {
"count": len(backup_files),
"latest": str(backup_files[-1]) if backup_files else None,
},
}
@app.get("/api/jobs")
def api_jobs():
# Known units that influence "freshness" of the brain.
units = [
"openclaw-secondbrain-ingest-memory.service",
"openclaw-secondbrain-index-vectors.service",
"openclaw-secondbrain-review.service",
"openclaw-secondbrain-heartbeat.service",
"openclaw-secondbrain-verify-pending.service",
]
return {"items": [_systemd_unit_state(u) for u in units]}
@app.get("/api/insights")
def api_insights(limit: int = Query(8, ge=1, le=50)):
conn = get_db()
c = conn.cursor()
rows = c.execute(
"SELECT id, metadata_json, correctness_json, created_at, modified_at FROM engrams ORDER BY created_at DESC LIMIT 2000"
).fetchall()
total = c.execute("SELECT COUNT(*) FROM engrams").fetchone()[0]
confirmed = c.execute(
"SELECT COUNT(*) FROM engrams WHERE json_extract(correctness_json, '$.confirmed') = 1"
).fetchone()[0]
pending = total - confirmed
tag_counts: dict[str, int] = {}
source_counts: dict[str, int] = {}
host_counts: dict[str, int] = {}
active: list[dict] = []
forgotten: list[dict] = []
for r in rows:
meta = json.loads(r["metadata_json"] or "{}")
src = meta.get("source", "unknown")
source_counts[src] = source_counts.get(src, 0) + 1
for t in (meta.get("tags") or []):
if isinstance(t, str):
tag_counts[t] = tag_counts.get(t, 0) + 1
host = _host_from_meta(r["metadata_json"])
if host:
host_counts[host] = host_counts.get(host, 0) + 1
access_count = int(meta.get("access_count", 0) or 0)
created = meta.get("created", r["created_at"])
if access_count >= 5 and len(active) < limit:
active.append(
{
"id": r["id"],
"access_count": access_count,
"source": src,
"created": created,
}
)
if access_count == 0 and len(forgotten) < limit:
forgotten.append(
{
"id": r["id"],
"access_count": access_count,
"source": src,
"created": created,
}
)
def top_k(d: dict[str, int]) -> list[dict]:
return [
{"key": k, "count": v}
for k, v in sorted(d.items(), key=lambda kv: kv[1], reverse=True)[:limit]
]
conn.close()
return {
"total": total,
"confirmed": confirmed,
"pending": pending,
"top_tags": top_k(tag_counts),
"top_sources": top_k(source_counts),
"top_hosts": top_k(host_counts),
"active_engrams": active,
"forgotten_engrams": forgotten,
}
@app.get("/api/graph")
def api_graph(limit_nodes: int = Query(200, ge=50, le=1000)):
"""
Returns a lightweight graph view:
- Nodes: engrams + tag:<tag> + host:<hostname>
- Edges: engram->tag and engram->host plus explicit engrams_links edges.
"""
conn = get_db()
c = conn.cursor()
rows = c.execute("SELECT id, metadata_json FROM engrams ORDER BY created_at DESC LIMIT 1000").fetchall()
link_rows = c.execute("SELECT from_id, to_id FROM engrams_links ORDER BY rowid DESC LIMIT 2000").fetchall()
conn.close()
nodes: dict[str, dict] = {}
edges: list[dict] = []
def add_node(nid: str, kind: str, label: str | None = None, weight: float | None = None):
if nid not in nodes:
nodes[nid] = {"id": nid, "kind": kind}
if label is not None:
nodes[nid]["label"] = label
if weight is not None:
nodes[nid]["weight"] = weight
for r in rows:
eid = r["id"]
add_node(eid, "engram", label=eid[:8])
for t in _safe_json_extract_tags(r["metadata_json"]):
tid = f"tag:{t}"
add_node(tid, "tag", label=t)
edges.append({"from": eid, "to": tid, "kind": "has_tag"})
host = _host_from_meta(r["metadata_json"])
if host:
hid = f"host:{host}"
add_node(hid, "host", label=host)
edges.append({"from": eid, "to": hid, "kind": "grounded_at"})
for fr, to in link_rows:
add_node(fr, "engram")
add_node(to, "engram")
edges.append({"from": fr, "to": to, "kind": "link"})
# Trim nodes to keep payload bounded (prefer engrams and connected tags/hosts)
if len(nodes) > limit_nodes:
# Keep a balanced subset: many engrams plus the most-connected non-engrams.
kept: dict[str, dict] = {}
engram_budget = int(limit_nodes * 0.7)
# 1) Keep newest engrams first (they appear first in `rows` loop insertion order)
for r in rows:
eid = r["id"]
if eid in nodes:
kept[eid] = nodes[eid]
if len(kept) >= engram_budget:
break
# 2) Rank remaining nodes by degree within current edge set
degree: dict[str, int] = {}
for e in edges:
degree[e["from"]] = degree.get(e["from"], 0) + 1
degree[e["to"]] = degree.get(e["to"], 0) + 1
remaining = [nid for nid in nodes.keys() if nid not in kept]
remaining.sort(key=lambda nid: degree.get(nid, 0), reverse=True)
for nid in remaining:
kept[nid] = nodes[nid]
if len(kept) >= limit_nodes:
break
nodes = kept
edges = [e for e in edges if e["from"] in nodes and e["to"] in nodes]
return {"nodes": list(nodes.values()), "edges": edges}
@app.get("/api/events")
def api_events():
"""
Server-Sent Events stream for lightweight real-time UI refresh.
"""
import time
def gen():
while True:
payload = {
"ts": datetime.now(timezone.utc).isoformat(),
"stats": api_stats(),
"storage": api_storage_stats(),
"jobs": api_jobs(),
"insights": api_insights(limit=8),
}
yield f"data: {json.dumps(payload, ensure_ascii=False)}\n\n"
time.sleep(5)
return StreamingResponse(gen(), media_type="text/event-stream")
@app.exception_handler(FileNotFoundError)
def handle_file_not_found(request: Request, exc: FileNotFoundError):
return JSONResponse(
status_code=503,
content={
"error": str(exc),
"hint": "Set SECOND_BRAIN_DB_PATH or SECOND_BRAIN_WORKSPACE to a valid location.",
},
)
@app.exception_handler(sqlite3.Error)
def handle_sqlite_error(request: Request, exc: sqlite3.Error):
return JSONResponse(status_code=500, content={"error": f"sqlite error: {exc}"})
@app.get("/api/stats")
def api_stats():
conn = get_db()
c = conn.cursor()
total = c.execute("SELECT COUNT(*) FROM engrams").fetchone()[0]
confirmed = c.execute(
"SELECT COUNT(*) FROM engrams WHERE json_extract(correctness_json, '$.confirmed') = 1"
).fetchone()[0]
pending = total - confirmed
errors = c.execute(
"SELECT COUNT(*) FROM engrams WHERE json_extract(metadata_json, '$.tags') LIKE '%error%'"
).fetchone()[0]
avg_conf = c.execute(
"SELECT AVG(json_extract(metadata_json, '$.confidence')) FROM engrams"
).fetchone()[0] or 0.0
conn.close()
return {
"total": total,
"confirmed": confirmed,
"pending": pending,
"errors": errors,
"avg_confidence": round(avg_conf, 2),
}
@app.get("/api/engrams")
def api_engrams(
limit: int = Query(20, ge=1, le=100),
offset: int = Query(0, ge=0),
tag: str = Query(None),
confirmed: bool = Query(None),
search: str = Query(None),
min_confidence: float = Query(0.0),
):
conn = get_db()
c = conn.cursor()
where_clauses = ["json_extract(metadata_json, '$.confidence') >= ?"]
params = [min_confidence]
if tag:
where_clauses.append("json_extract(metadata_json, '$.tags') LIKE ?")
params.append(f'%"{tag}"%')
if confirmed is not None:
where_clauses.append(
f"json_extract(correctness_json, '$.confirmed') = {int(confirmed)}"
)
if search:
# Use FTS
try:
ids = [
r[0] for r in c.execute(
"SELECT rowid FROM engrams_fts WHERE content MATCH ? LIMIT 200",
(search,)
).fetchall()
]
if ids:
placeholders = ",".join("?" * len(ids))
where_clauses.append(f"id IN ({placeholders})")
params.extend(ids)
else:
# Full-text fallback on content
where_clauses.append("content LIKE ?")
params.append(f"%{search}%")
except Exception:
where_clauses.append("content LIKE ?")
params.append(f"%{search}%")
where_sql = " AND ".join(where_clauses)
rows = c.execute(
f"""
SELECT * FROM engrams
WHERE {where_sql}
ORDER BY created_at DESC
LIMIT ? OFFSET ?
""",
params + [limit, offset],
).fetchall()
result = [parse_engram(r) for r in rows]
conn.close()
return {"items": result, "limit": limit, "offset": offset}
@app.get("/api/engrams/{engram_id}")
def api_engram_detail(engram_id: str):
conn = get_db()
c = conn.cursor()
row = c.execute("SELECT * FROM engrams WHERE id = ?", (engram_id,)).fetchone()
if not row:
conn.close()
return JSONResponse({"error": "Not found"}, status_code=404)
# Links
links = c.execute(
"SELECT to_id FROM engrams_links WHERE from_id = ?", (engram_id,)
).fetchall()
result = parse_engram(row)
result["links"] = [r[0] for r in links]
conn.close()
return result
@app.get("/api/pending")
def api_pending(
limit: int = Query(20, ge=1, le=200),
offset: int = Query(0, ge=0),
source: str | None = Query(None),
):
conn = get_db()
c = conn.cursor()
where = ["json_extract(correctness_json, '$.confirmed') = 0"]
params: list = []
if source:
where.append("json_extract(metadata_json, '$.source') = ?")
params.append(source)
rows = c.execute(
f"""
SELECT * FROM engrams
WHERE {' AND '.join(where)}
ORDER BY created_at DESC
LIMIT ? OFFSET ?
""",
params + [limit, offset],
).fetchall()
items = [parse_engram(r) for r in rows]
conn.close()
return {"items": items, "limit": limit, "offset": offset}
@app.post("/api/engrams")
def api_create_engram(content: str = Form(...), tags: str = Form("")):
content = (content or "").strip()
if not content:
return JSONResponse({"error": "content required"}, status_code=400)
tag_list = [t.strip() for t in (tags or "").split(",") if t.strip()]
now = _now_iso()
engram_id = str(uuid4())
meta = {
"source": "user",
"confidence": 0.7,
"created": now,
"modified": now,
"access_count": 0,
"last_accessed": now,
"tags": tag_list,
"session_id": None,
"agent_id": None,
"grounding": 0,
}
corr = {
"confirmed": False,
"confirmations": 0,
"rejections": 0,
"last_reviewed": None,
"review_history": [],
}
conn = get_db()
c = conn.cursor()
c.execute(
"""
INSERT INTO engrams (id, content, metadata_json, correctness_json, links_json, hierarchy_json, embedding_json, created_at, modified_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
engram_id,
content,
json.dumps(meta, ensure_ascii=False),
json.dumps(corr, ensure_ascii=False),
"[]",
"{}",
None,
now,
now,
),
)
conn.commit()
conn.close()
return {"id": engram_id}
@app.post("/api/engrams/{engram_id}/confirm")
def api_confirm_engram(engram_id: str, reason: str = Form("")):
try:
return _update_correctness(engram_id, action="confirm", reason=reason)
except FileNotFoundError as e:
return JSONResponse({"error": str(e)}, status_code=404)
@app.post("/api/engrams/{engram_id}/reject")
def api_reject_engram(engram_id: str, reason: str = Form("")):
try:
return _update_correctness(engram_id, action="reject", reason=reason)
except FileNotFoundError as e:
return JSONResponse({"error": str(e)}, status_code=404)
@app.post("/api/engrams/{engram_id}/refresh")
def api_refresh_engram(engram_id: str):
try:
return _bump_access(engram_id)
except FileNotFoundError as e:
return JSONResponse({"error": str(e)}, status_code=404)
@app.post("/api/engrams/{engram_id}/refresh")
def api_refresh(engram_id: str):
conn = get_db()
c = conn.cursor()
row = c.execute(
"SELECT metadata_json, correctness_json FROM engrams WHERE id = ?", (engram_id,)
).fetchone()
if not row:
conn.close()
return JSONResponse({"error": "Not found"}, status_code=404)
meta = json.loads(row["metadata_json"] or "{}")
correctness = json.loads(row["correctness_json"] or "{}")
# Simple heuristic: confidence based on confirmations vs rejections
conf = 0.5
conf += 0.1 * correctness.get("confirmations", 0)
conf -= 0.15 * correctness.get("rejections", 0)
conf = max(0.1, min(1.0, conf))
meta["confidence"] = round(conf, 2)
meta["modified"] = datetime.now(timezone.utc).isoformat()
c.execute(
"UPDATE engrams SET metadata_json = ?, modified_at = ? WHERE id = ?",
(json.dumps(meta), datetime.now(timezone.utc).isoformat(), engram_id),
)
conn.commit()
conn.close()
return {"success": True, "new_confidence": round(conf, 2)}
@app.post("/api/engrams")
def api_create_engram(content: str = Form(...), tags: str = Form(""), source: str = Form("web")):
engram_id = f"web-{datetime.now(timezone.utc).strftime('%Y%m%d-%H%M%S-%f')[:20]}"
now = datetime.now(timezone.utc).isoformat()
meta = {
"source": source,
"confidence": 0.5,
"created": now,
"modified": now,
"access_count": 0,
"last_accessed": now,
"tags": [t.strip() for t in tags.split(",") if t.strip()] or ["web"],
"session_id": None,
"agent_id": None,
"grounding": 0,
"hash": "",
}
correctness = {
"confirmed": False,
"confirmations": 0,
"rejections": 0,
"last_reviewed": None,
"review_history": [],
}
conn = get_db()
c = conn.cursor()
c.execute(
"""
INSERT INTO engrams (id, content, metadata_json, correctness_json, links_json, hierarchy_json, created_at, modified_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
""",
(engram_id, content, json.dumps(meta), json.dumps(correctness), "[]", '{"parent": null, "children": [], "depth": 0}', now, now),
)
conn.commit()
conn.close()
return {"success": True, "engram_id": engram_id}
@app.get("/api/pending")
def api_pending(limit: int = Query(20, ge=1, le=100), offset: int = Query(0, ge=0)):
conn = get_db()
c = conn.cursor()
rows = c.execute(
"""
SELECT * FROM engrams
WHERE json_extract(correctness_json, '$.confirmed') = 0
ORDER BY created_at DESC
LIMIT ? OFFSET ?
""",
(limit, offset),
).fetchall()
result = [parse_engram(r) for r in rows]
conn.close()
return {"items": result, "limit": limit, "offset": offset}
@app.get("/api/search")
def api_search(
q: str = Query(..., min_length=1),
min_confidence: float = Query(0.0),
limit: int = Query(20, ge=1, le=100),
):
conn = get_db()
c = conn.cursor()
try:
ids = [
r[0] for r in c.execute(
"SELECT rowid FROM engrams_fts WHERE content MATCH ? LIMIT 200",
(q,)
).fetchall()
]
if ids:
placeholders = ",".join("?" * len(ids))
rows = c.execute(
f"""
SELECT * FROM engrams
WHERE id IN ({placeholders})
AND json_extract(metadata_json, '$.confidence') >= ?
ORDER BY created_at DESC
LIMIT ?
""",
ids + [min_confidence, limit],
).fetchall()
else:
rows = []
except Exception:
rows = c.execute(
"""
SELECT * FROM engrams
WHERE content LIKE ? AND json_extract(metadata_json, '$.confidence') >= ?
ORDER BY created_at DESC
LIMIT ?
""",
(f"%{q}%", min_confidence, limit),
).fetchall()
result = [parse_engram(r) for r in rows]
conn.close()
return {"items": result, "query": q}
# ─── Frontend ────────────────────────────────────────────────────────────────
@app.get("/", response_class=HTMLResponse)
def dashboard(request: Request):
with open(WORKSPACE / "templates" / "dashboard.html", "r", encoding="utf-8") as f:
html = f.read()
return HTMLResponse(content=html)
# ─── Main ────────────────────────────────────────────────────────────────────
if __name__ == "__main__":
import uvicorn
uvicorn.run("fastapi_app:app", host=HOST, port=PORT)

102
openclaw_cron_wrapper.py Normal file
View File

@@ -0,0 +1,102 @@
#!/usr/bin/env python3
"""
OpenClaw Cron Isolation Wrapper - Updatesicherer Workaround.
Persistent: Tasks und Logs liegen im Workspace, nicht in /tmp.
"""
import os
import sys
import json
import subprocess
import tempfile
from pathlib import Path
from datetime import datetime, timezone
# --- Konfiguration (persistent) ---
WORKSPACE = Path("/root/.openclaw/workspace")
CRON_TASKS_DIR = WORKSPACE / "cron_tasks"
LOG_FILE = WORKSPACE / "cron_wrapper.log"
BRAIN_DIR = WORKSPACE / "second-brain"
def log(msg: str):
ts = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
line = f"[{ts}] {msg}\n"
with open(LOG_FILE, "a") as f:
f.write(line)
print(line.strip())
def run_isolated(task_name: str, task_args: dict = None) -> dict:
"""
Führt einen Task in echt isolierter Umgebung aus.
Kein Zugriff auf Session-Files, nur stdout/stderr.
"""
CRON_TASKS_DIR.mkdir(parents=True, exist_ok=True)
task_script = CRON_TASKS_DIR / f"{task_name}.py"
if not task_script.exists():
return {"success": False, "error": f"Task nicht gefunden: {task_script}"}
# Temp-Verzeichnis für Output (flüchtig ist OK, Ergebnis kommt via stdout)
temp_dir = tempfile.mkdtemp(prefix=f"cron_{task_name}_")
output_file = Path(temp_dir) / "output.json"
# Saubere Env: Keine OpenClaw-Session-Variablen
env = os.environ.copy()
for key in list(env.keys()):
if "OPENCLAW" in key.upper() or "SESSION" in key.upper():
del env[key]
env["CRON_TASK_NAME"] = task_name
env["CRON_OUTPUT_FILE"] = str(output_file)
env["BRAIN_DB"] = str(BRAIN_DIR / "data" / "brain.sqlite")
try:
result = subprocess.run(
[sys.executable, str(task_script)] + ([json.dumps(task_args)] if task_args else []),
capture_output=True,
text=True,
timeout=300,
cwd=str(temp_dir),
env=env,
)
stdout = result.stdout.strip()
stderr = result.stderr.strip()
output_data = {}
if output_file.exists():
try:
output_data = json.loads(output_file.read_text())
except Exception:
output_data = {"raw": output_file.read_text()}
return {
"success": result.returncode == 0,
"returncode": result.returncode,
"stdout": stdout[-2000:] if stdout else "",
"stderr": stderr[-1000:] if stderr else "",
"output": output_data,
}
except subprocess.TimeoutExpired:
return {"success": False, "error": "Timeout nach 300s"}
except Exception as e:
return {"success": False, "error": str(e)}
def main():
import argparse
parser = argparse.ArgumentParser(description="OpenClaw Cron Isolation Wrapper")
parser.add_argument("task", help="Task-Name aus cron_tasks/")
parser.add_argument("--args", help="JSON-Args für den Task")
args = parser.parse_args()
task_args = json.loads(args.args) if args.args else None
result = run_isolated(args.task, task_args)
print(json.dumps(result, indent=2, default=str))
return 0 if result["success"] else 1
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,4 @@
fastapi>=0.110
uvicorn[standard]>=0.23
jinja2>=3.1
python-multipart>=0.0.9

View File

@@ -0,0 +1,159 @@
#!/usr/bin/env python3
"""
Auto-discover an Obsidian vault on this server and (optionally) write it into:
second-brain/data/obsidian_config.json
Safety:
- Only writes when exactly one vault is detected (unambiguous).
- A "vault" is a directory that contains a `.obsidian/` folder.
"""
from __future__ import annotations
import argparse
import json
import os
from pathlib import Path
from typing import Iterable
WORKSPACE = Path("/root/.openclaw/workspace")
BRAIN_DIR = WORKSPACE / "second-brain"
CONFIG_PATH = BRAIN_DIR / "data" / "obsidian_config.json"
def _iter_common_candidates() -> Iterable[Path]:
env = os.environ.get("OBSIDIAN_VAULT_PATH")
if env:
yield Path(env).expanduser()
home = Path.home()
for p in [
home / "Obsidian",
home / "ObsidianVault",
home / "Vault",
home / "Vaults",
home / "Documents" / "Obsidian",
home / "Documents" / "Vaults",
home / "Syncthing" / "Obsidian",
Path("/srv/obsidian"),
Path("/srv/Obsidian"),
Path("/data/obsidian"),
Path("/data/Obsidian"),
WORKSPACE / "obsidian",
WORKSPACE / "vault",
WORKSPACE / "vaults",
]:
yield p
def _is_vault_dir(p: Path) -> bool:
try:
return p.exists() and p.is_dir() and (p / ".obsidian").exists() and (p / ".obsidian").is_dir()
except Exception:
return False
def _bounded_find_obsidian_dirs(root: Path, *, max_depth: int) -> list[Path]:
"""
Find `.obsidian` directories below root, limited by depth to keep runtime bounded.
"""
results: list[Path] = []
try:
root = root.resolve()
except Exception:
return results
if not root.exists() or not root.is_dir():
return results
def depth_of(path: Path) -> int:
try:
return len(path.relative_to(root).parts)
except Exception:
return 9999
# Breadth-first-ish scan with pruning
queue = [root]
while queue:
current = queue.pop(0)
if depth_of(current) > max_depth:
continue
try:
entries = list(current.iterdir())
except Exception:
continue
for e in entries:
name = e.name
if name in (".git", "node_modules", "__pycache__", ".cache", ".venv", "venv", "tmp", "proc", "sys", "dev"):
continue
if name.startswith(".") and name not in (".obsidian",):
continue
if name == ".obsidian" and e.is_dir():
results.append(e)
continue
if e.is_dir() and not e.is_symlink():
queue.append(e)
return results
def discover(*, roots: list[Path], max_depth: int) -> list[Path]:
vaults: set[Path] = set()
for p in _iter_common_candidates():
if _is_vault_dir(p):
vaults.add(p.resolve())
for root in roots:
for obsidian_dir in _bounded_find_obsidian_dirs(root, max_depth=max_depth):
vaults.add(obsidian_dir.parent.resolve())
return sorted(vaults)
def main() -> int:
ap = argparse.ArgumentParser(description="Discover Obsidian vault and optionally write config")
ap.add_argument("--write", action="store_true", help="Write detected vault_path into obsidian_config.json")
ap.add_argument(
"--roots",
nargs="*",
default=[str(Path.home()), "/srv", "/data", "/mnt", str(WORKSPACE)],
help="Roots to scan (bounded). Default: home,/srv,/data,/mnt,workspace",
)
ap.add_argument("--max-depth", type=int, default=4, help="Max directory depth to scan under each root")
args = ap.parse_args()
roots = [Path(r).expanduser() for r in args.roots]
vaults = discover(roots=roots, max_depth=int(args.max_depth))
if not vaults:
print("No Obsidian vault found (no `.obsidian/` directories detected).")
return 1
if len(vaults) > 1:
print("Multiple Obsidian vaults found; refusing to write config:")
for v in vaults:
print(f"- {v}")
return 2
vault = vaults[0]
print(f"Detected Obsidian vault: {vault}")
if not args.write:
return 0
if not CONFIG_PATH.exists():
raise SystemExit(f"Missing config file: {CONFIG_PATH}")
cfg = json.loads(CONFIG_PATH.read_text(encoding="utf-8"))
cfg["vault_path"] = str(vault)
CONFIG_PATH.write_text(json.dumps(cfg, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
print(f"Wrote vault_path to: {CONFIG_PATH}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

20
scripts/smoke_dashboard.sh Executable file
View File

@@ -0,0 +1,20 @@
#!/usr/bin/env bash
set -euo pipefail
HOST="${SECOND_BRAIN_HOST:-127.0.0.1}"
PORT="${SECOND_BRAIN_PORT:-${PORT:-8501}}"
BASE_URL="http://${HOST}:${PORT}"
if command -v systemctl >/dev/null 2>&1; then
if ! systemctl is-active --quiet openclaw-secondbrain-dashboard.service; then
echo "ERROR: openclaw-secondbrain-dashboard.service is not active" >&2
exit 1
fi
fi
curl -fsS "${BASE_URL}/healthz" >/dev/null
stats_json="$(curl -fsS "${BASE_URL}/api/stats")"
python3 -c 'import json,sys; json.load(sys.stdin)' <<<"$stats_json"
echo "OK: dashboard smoke test passed (${BASE_URL})"

View File

@@ -1,8 +1,14 @@
"""Second Brain - Gedächtnissystem für OpenClaw.""" """Second Brain - Gedächtnissystem für OpenClaw."""
from .engram import Engram, Grounding, Correctness, ReviewEntry try:
from .store import EngramStore from .engram import Engram, Grounding, Correctness, ReviewEntry
from .retriever import Retriever from .store import EngramStore
from .retriever import Retriever
except ImportError:
# Fallback: ChromaDB optional, SQLite-core funktioniert immer
from .engram import Engram, Grounding, Correctness, ReviewEntry
from .store import EngramStore
Retriever = None
__version__ = "0.1.0" __version__ = "0.1.0"
__all__ = ["Engram", "Grounding", "Correctness", "ReviewEntry", "EngramStore", "Retriever"] __all__ = ["Engram", "Grounding", "Correctness", "ReviewEntry", "EngramStore", "Retriever"]

View File

@@ -13,12 +13,23 @@ import argparse
import http.server import http.server
import socketserver import socketserver
import webbrowser import webbrowser
import sys
from pathlib import Path from pathlib import Path
from uuid import UUID from uuid import UUID
from .store import EngramStore # Insert project root so `python3 src/dashboard.py` works without `-m`
from .engram import Engram, Grounding project_root = str(Path(__file__).parent.parent)
from .retriever import Retriever if project_root not in sys.path:
sys.path.insert(0, project_root)
from src.store import EngramStore
from src.engram import Engram, Grounding
# Retriever: optional im venv verfügbar, sonst Fallback
try:
from src.retriever import Retriever
except ImportError:
Retriever = None
DB_PATH = Path(__file__).parent.parent / "data" / "brain.sqlite" DB_PATH = Path(__file__).parent.parent / "data" / "brain.sqlite"
HTML_PATH = Path(__file__).parent.parent / "data" / "dashboard.html" HTML_PATH = Path(__file__).parent.parent / "data" / "dashboard.html"
@@ -27,9 +38,29 @@ HTML_PATH = Path(__file__).parent.parent / "data" / "dashboard.html"
def generate_dashboard() -> str: def generate_dashboard() -> str:
"""Generiert HTML-Dashboard aus aktuellem Brain-Stand.""" """Generiert HTML-Dashboard aus aktuellem Brain-Stand."""
store = EngramStore(str(DB_PATH)) store = EngramStore(str(DB_PATH))
egs = store.get_all(limit=100)
# Stats: mit Retriever (venv) oder manuell berechnen
if Retriever:
ret = Retriever(store) ret = Retriever(store)
stats = ret.stats() stats = ret.stats()
egs = store.get_all(limit=100) errors = store.search_tag("error", limit=100)
stats["errors"] = len(errors)
else:
from src.engram import Correctness
all_egs = store.get_all(limit=10000)
confirmed = sum(1 for e in all_egs if e.correctness.confirmed)
errors = store.search_tag("error", limit=100)
confidences = [e.compute_confidence() for e in all_egs]
stats = {
"total_engrams": len(all_egs),
"confirmed": confirmed,
"unconfirmed": len(all_egs) - confirmed,
"sources": {},
"db_size_bytes": 0,
"avg_confidence": sum(confidences) / len(confidences) if confidences else 0.0,
"errors": len(errors),
}
# Farbe nach Confidence # Farbe nach Confidence
def color(conf): def color(conf):
@@ -48,95 +79,81 @@ def generate_dashboard() -> str:
except: except:
return "UNKNOWN" return "UNKNOWN"
# Liste der Engramme # Karten-Ansicht für Mobil
rows = [] card_rows = []
for eg in egs: for eg in egs:
conf = eg.compute_confidence() conf = eg.compute_confidence()
rows.append(f""" card_rows.append(f"""
<tr> <div class="card">
<td><span style="color:{color(conf)};font-size:1.2em">{marker(conf)}</span></td> <div class="card-header">
<td><code>{str(eg.id)[:8]}</code></td> <span>{marker(conf)} {str(eg.id)[:8]}</span>
<td>{eg.content[:100]}{'...' if len(eg.content) > 100 else ''}</td> <span class="badge" style="background:rgba({int((1-conf)*200+55)},{int(conf*200+55)},100,0.3)">{conf:.2f}</span>
<td><span class="badge">{grounding_name(eg.metadata.get('grounding', 0))}</span></td> </div>
<td><span class="badge">{eg.metadata.get('source', '?')}</span></td> <div class="card-content">{eg.content[:150]}{'...' if len(eg.content) > 150 else ''}</div>
<td>{conf:.2f}</td> <div class="card-meta">
<td>{eg.correctness.confirmations}/{eg.correctness.rejections}</td> <span class="badge">{grounding_name(eg.metadata.get('grounding', 0))}</span>
<td>{eg.metadata.get('access_count', 0)}</td> <span class="badge">{eg.metadata.get('source', '?')}</span>
<td>{', '.join(eg.metadata.get('tags', [])[:3])}</td> <span class="badge">✓{eg.correctness.confirmations}/✗{eg.correctness.rejections}</span>
</tr> {' '.join([f'<span class="badge">{t}</span>' for t in eg.metadata.get('tags', [])[:3]])}
</div>
</div>
""") """)
html = f"""<!DOCTYPE html> html = f"""<!DOCTYPE html>
<html lang="de"> <html lang="de">
<head> <head>
<meta charset="UTF-8"> <meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
<title>🧠 Second Brain Dashboard</title> <title>🧠 Second Brain</title>
<style> <style>
:root {{ --bg: #1a1a2e; --card: #16213e; --text: #eee; --accent: #0f4c75; }} :root {{ --bg: #1a1a2e; --card: #16213e; --text: #eee; --accent: #0f4c75; }}
body {{ font-family: -apple-system, BlinkMacSystemFont, sans-serif; background: var(--bg); color: var(--text); margin: 0; padding: 20px; }} body {{ font-family: -apple-system, BlinkMacSystemFont, sans-serif; background: var(--bg); color: var(--text); margin: 0; padding: 10px; font-size: 14px; }}
h1 {{ margin: 0 0 10px; font-size: 1.8em; }} h1 {{ margin: 0 0 8px; font-size: 1.3em; text-align: center; }}
.stats {{ display: grid; grid-template-columns: repeat(auto-fit, minmax(180px, 1fr)); gap: 15px; margin-bottom: 25px; }} .stats {{ display: grid; grid-template-columns: repeat(2, 1fr); gap: 8px; margin-bottom: 15px; }}
.stat-card {{ background: var(--card); border-radius: 12px; padding: 15px; text-align: center; }} .stat-card {{ background: var(--card); border-radius: 10px; padding: 10px; text-align: center; }}
.stat-card .num {{ font-size: 2em; font-weight: bold; color: #4fc3f7; }} .stat-card .num {{ font-size: 1.5em; font-weight: bold; color: #4fc3f7; }}
.stat-card .lbl {{ font-size: 0.85em; opacity: 0.7; }} .stat-card .lbl {{ font-size: 0.75em; opacity: 0.7; }}
table {{ width: 100%; border-collapse: collapse; background: var(--card); border-radius: 12px; overflow: hidden; }} .search {{ margin-bottom: 10px; }}
th {{ text-align: left; padding: 10px; background: var(--accent); font-size: 0.85em; text-transform: uppercase; }} .search input {{ width: 100%; box-sizing: border-box; padding: 12px; border-radius: 8px; border: 1px solid rgba(255,255,255,0.1); background: var(--card); color: var(--text); font-size: 16px; }}
td {{ padding: 8px 10px; border-bottom: 1px solid rgba(255,255,255,0.05); font-size: 0.9em; vertical-align: top; }} .card {{ background: var(--card); border-radius: 12px; padding: 12px; margin-bottom: 10px; }}
tr:hover td {{ background: rgba(255,255,255,0.03); }} .card-header {{ display: flex; justify-content: space-between; align-items: center; margin-bottom: 6px; }}
.badge {{ display: inline-block; padding: 2px 8px; border-radius: 12px; background: rgba(79,195,247,0.2); font-size: 0.75em; }} .card-id {{ font-size: 0.75em; opacity: 0.6; }}
.search {{ margin-bottom: 20px; }} .card-content {{ font-size: 0.9em; line-height: 1.4; margin-bottom: 8px; word-break: break-word; }}
.search input {{ width: 100%; max-width: 400px; padding: 10px 15px; border-radius: 8px; border: 1px solid rgba(255,255,255,0.1); background: var(--card); color: var(--text); font-size: 1em; }} .card-meta {{ display: flex; flex-wrap: wrap; gap: 6px; font-size: 0.75em; }}
.refresh {{ position: fixed; top: 20px; right: 20px; background: #2ecc71; color: #000; border: none; padding: 10px 20px; border-radius: 8px; cursor: pointer; font-weight: bold; }} .badge {{ display: inline-block; padding: 3px 8px; border-radius: 12px; background: rgba(79,195,247,0.2); }}
.footer {{ margin-top: 30px; text-align: center; opacity: 0.5; font-size: 0.8em; }} .refresh {{ position: fixed; bottom: 15px; right: 15px; background: #2ecc71; color: #000; border: none; padding: 12px 16px; border-radius: 50%; cursor: pointer; font-weight: bold; font-size: 1.2em; box-shadow: 0 2px 8px rgba(0,0,0,0.4); }}
.footer {{ margin-top: 20px; text-align: center; opacity: 0.5; font-size: 0.7em; padding-bottom: 60px; }}
</style> </style>
</head> </head>
<body> <body>
<h1>🧠 Second Brain Dashboard</h1> <h1>🧠 Second Brain</h1>
<button class="refresh" onclick="location.reload()">🔄 Aktualisieren</button> <button class="refresh" onclick="location.reload()"></button>
<div class="stats"> <div class="stats">
<div class="stat-card"><div class="num">{stats['total_engrams']}</div><div class="lbl">Engramme</div></div> <div class="stat-card"><div class="num">{stats['total_engrams']}</div><div class="lbl">Engramme</div></div>
<div class="stat-card"><div class="num">{stats['confirmed']}</div><div class="lbl">Bestätigt</div></div> <div class="stat-card"><div class="num">{stats['confirmed']}</div><div class="lbl">Bestätigt</div></div>
<div class="stat-card"><div class="num">{stats['unconfirmed']}</div><div class="lbl">Unbestätigt</div></div> <div class="stat-card"><div class="num">{stats['unconfirmed']}</div><div class="lbl">Unbestätigt</div></div>
<div class="stat-card"><div class="num">{len(stats.get('sources', dict()))}</div><div class="lbl">Quellen</div></div> <div class="stat-card"><div class="num">{stats['errors']}</div><div class="lbl">Fehler</div></div>
<div class="stat-card"><div class="num">{stats['db_size_bytes']/1024:.0f}</div><div class="lbl">KB Größe</div></div>
</div> </div>
<div class="search"> <div class="search">
<input type="text" placeholder="🔍 Suche nach Engrammen..." id="searchInput" onkeyup="filterTable()"> <input type="text" placeholder="🔍 Suche..." id="searchInput" onkeyup="filterCards()">
</div> </div>
<table id="engramTable"> <div id="cards">
<thead> {''.join(card_rows)}
<tr> </div>
<th>Status</th>
<th>ID</th>
<th>Inhalt</th>
<th>Grounding</th>
<th>Quelle</th>
<th>Confidence</th>
<th>Feedback</th>
<th>Zugriffe</th>
<th>Tags</th>
</tr>
</thead>
<tbody>
{''.join(rows)}
</tbody>
</table>
<div class="footer">Generated {__import__('datetime').datetime.now().isoformat()}</div> <div class="footer">Generated {__import__('datetime').datetime.now().strftime('%H:%M')}</div>
<script> <script>
function filterTable() {{ function filterCards() {{
var input = document.getElementById('searchInput'); var input = document.getElementById('searchInput');
var filter = input.value.toLowerCase(); var filter = input.value.toLowerCase();
var table = document.getElementById('engramTable'); var cards = document.getElementById('cards').getElementsByClassName('card');
var rows = table.getElementsByTagName('tr'); for (var i=0; i<cards.length; i++) {{
for (var i=1; i<rows.length; i++) {{ var txt = cards[i].textContent || cards[i].innerText;
var txt = rows[i].textContent || rows[i].innerText; cards[i].style.display = txt.toLowerCase().indexOf(filter) > -1 ? '' : 'none';
rows[i].style.display = txt.toLowerCase().indexOf(filter) > -1 ? '' : 'none';
}} }}
}} }}
</script> </script>

View File

@@ -66,12 +66,19 @@ class Correctness:
return self.confirmations / total return self.confirmations / total
def to_dict(self) -> dict: def to_dict(self) -> dict:
# Backwards/robustness: older code paths may have appended raw dicts.
review_history: List[dict] = []
for entry in self.review_history:
if isinstance(entry, dict):
review_history.append(entry)
else:
review_history.append(entry.to_dict())
return { return {
"confirmed": self.confirmed, "confirmed": self.confirmed,
"confirmations": self.confirmations, "confirmations": self.confirmations,
"rejections": self.rejections, "rejections": self.rejections,
"last_reviewed": self.last_reviewed, "last_reviewed": self.last_reviewed,
"review_history": [r.to_dict() for r in self.review_history], "review_history": review_history,
} }
@classmethod @classmethod

View File

@@ -16,10 +16,33 @@ from pathlib import Path
from datetime import datetime, timezone from datetime import datetime, timezone
from typing import Optional, List, Dict, Any from typing import Optional, List, Dict, Any
# Ensure project root is on sys.path for standalone usage
project_root = str(Path(__file__).parent.parent)
if project_root not in sys.path:
sys.path.insert(0, project_root)
# Activate virtualenv if available (for chromadb etc.)
venv_path = Path(__file__).parent.parent / ".venv"
if venv_path.exists():
venv_site_packages = list((venv_path / "lib").glob("python3.*/site-packages"))
if venv_site_packages and str(venv_site_packages[0]) not in sys.path:
sys.path.insert(0, str(venv_site_packages[0]))
# Second Brain Import # Second Brain Import
from .store import EngramStore from src.engram import Engram, Grounding
from .engram import Engram, Grounding from src.store import EngramStore
from .retriever import Retriever
# Retriever: optional (braucht chromadb)
try:
from src.retriever import Retriever
except ImportError:
Retriever = None
# Chroma: optional (braucht chromadb)
try:
from src.chroma_store import ChromaStore
except Exception:
ChromaStore = None
# --- Konfiguration --- # --- Konfiguration ---
@@ -75,11 +98,6 @@ def heartbeat_check() -> Optional[str]:
Rückgabe: Nachricht für den User, oder None wenn nichts zu tun. Rückgabe: Nachricht für den User, oder None wenn nichts zu tun.
""" """
store = get_brain() store = get_brain()
ret = Retriever(store)
# A: Unbestätigte Engramme die seit längerem nicht geprüft wurden
# B: Hohe-Prioritäts-Themen (tags wie "wichtig", "dringend")
# C: Fehler-Engramme die repeating sind
# Prüfe auf wichtige unbestätigte Engramme # Prüfe auf wichtige unbestätigte Engramme
egs = store.get_all(limit=50) egs = store.get_all(limit=50)
@@ -89,8 +107,7 @@ def heartbeat_check() -> Optional[str]:
][:5] ][:5]
if unconfirmed: if unconfirmed:
ids = ", ".join([str(eg.id)[:8] for eg in unconfirmed]) contents = "\n".join([f" - {eg.content[:80]}" for eg in unconfirmed])
contents = "\n".join([f" - {eg.content[:80]}" for eg in unconfimed])
return ( return (
f"🧠 Second Brain Heartbeat\n" f"🧠 Second Brain Heartbeat\n"
f"Unbestätigte Engramme mit gutem Confidence-Score:\n{contents}\n" f"Unbestätigte Engramme mit gutem Confidence-Score:\n{contents}\n"
@@ -195,8 +212,39 @@ def enrich_context(topic: str, limit: int = 3) -> str:
# memory_context in das Prompt einbauen # memory_context in das Prompt einbauen
""" """
store = get_brain() store = get_brain()
ret = Retriever(store)
results = ret.retrieve(topic, limit=limit, min_confidence=0.3) # Versuche Hybrid-Retrieval (FTS + optional Vector), fallback auf Textsuche
if Retriever:
chroma = None
if ChromaStore:
try:
chroma = ChromaStore(path=str(Path(__file__).parent.parent / "data" / "chroma"))
except Exception:
chroma = None
ret = Retriever(store, chroma=chroma)
try:
results = ret.hybrid_retrieve(topic, limit=limit * 3, min_confidence=0.3)
except Exception:
results = ret.retrieve(topic, limit=limit * 3, min_confidence=0.3)
# confirmed-first ranking
def _rank(r):
eg = r["engram"]
confirmed = 1 if getattr(eg.correctness, "confirmed", False) else 0
return (confirmed, float(r.get("score", 0.0)))
results.sort(key=_rank, reverse=True)
# If we have confirmed results, show only confirmed up to limit
confirmed_only = [r for r in results if r["engram"].correctness.confirmed]
if confirmed_only:
results = confirmed_only[:limit]
else:
results = results[:limit]
else:
results_raw = store.search_text(topic, limit=limit)
results = [{"engram": eg, "score": 0.5} for eg in results_raw]
if not results: if not results:
return "" return ""

165
src/proactive_search.py Normal file
View File

@@ -0,0 +1,165 @@
#!/usr/bin/env python3
"""
proactive_search.py - Proaktive Websuche für Second Brain.
Sucht relevante Themen, speichert Ergebnisse als Engramme.
Stoppt wenn neue Aufgaben erkannt werden.
"""
import sys
import json
from pathlib import Path
from datetime import datetime, timezone, timedelta
from typing import List, Dict, Any, Optional
sys.path.insert(0, str(Path(__file__).resolve().parent))
from src.store import EngramStore
from src.engram import Engram, Grounding
from src.retriever import Retriever
from src.embedder import encode
from src.chroma_store import ChromaStore
DB_PATH = Path(__file__).resolve().parent.parent / "data" / "brain.sqlite"
CHROMA_PATH = Path(__file__).resolve().parent.parent / "data" / "chroma"
# Themen die relevant sind für den Benutzer
INTEREST_TOPICS = [
"OpenClaw AI Agent",
"Künstliche Intelligenz Trends 2025",
"Second Brain Memory System",
"Automation DIY Projects",
"Smart Home IoT",
"Raspberry Pi Projects",
"Deutschland Tech News",
"AI Agent Frameworks",
"Workflow Automation",
]
def get_store():
return EngramStore(str(DB_PATH))
def load_state() -> Dict[str, Any]:
"""Lädt den Such-Zustand."""
state_path = Path(__file__).resolve().parent.parent / "data" / "search_state.json"
if state_path.exists():
with open(state_path, "r", encoding="utf-8") as f:
return json.load(f)
return {
"last_search": None,
"searched_topics": [],
"new_tasks_detected": False,
"paused_until": None,
}
def save_state(state: Dict[str, Any]):
state_path = Path(__file__).resolve().parent.parent / "data" / "search_state.json"
with open(state_path, "w", encoding="utf-8") as f:
json.dump(state, f, ensure_ascii=False)
def check_for_new_tasks(store: EngramStore) -> bool:
"""Prüft ob in letzten 2h neue Aufgaben-Artige Engramme erstellt wurden."""
now = datetime.now(timezone.utc)
recent = now - timedelta(hours=2)
egs = store.get_all(limit=1000)
for eg in egs:
created_str = eg.metadata.get("created", "")
if not created_str:
continue
try:
eg_time = datetime.fromisoformat(created_str)
if eg_time.tzinfo is None:
eg_time = eg_time.replace(tzinfo=timezone.utc)
if eg_time > recent:
tags = eg.metadata.get("tags", [])
if "task" in tags or "aufgabe" in tags or "todo" in tags:
return True
except Exception:
pass
return False
def try_web_search(topic: str) -> Optional[List[Dict[str, str]]]:
"""Web-Suche via OpenClaw."""
try:
import subprocess
result = subprocess.run(
["python3", "-c", f"""
import sys
sys.path.insert(0, '/root/.openclaw/workspace/second-brain/src')
from src.retriever import Retriever
from src.store import EngramStore
store = EngramStore('data/brain.sqlite')
ret = Retriever(store)
results = ret.retrieve('{topic}')
print('FOUND ' + str(len(results)))
"""],
capture_output=True,
text=True,
timeout=30,
cwd="/root/.openclaw/workspace/second-brain",
)
# Actually do web search
print(f"[search] Would search: {topic}")
return None # Placeholder: real search would be here
except Exception as e:
print(f"[search] Error: {e}")
return None
def run_proactive_search():
"""Haupt-Funktion für proaktive Suche."""
store = get_store()
state = load_state()
now = datetime.now(timezone.utc)
# Check: Neue Aufgaben?
if check_for_new_tasks(store):
state["new_tasks_detected"] = True
state["paused_until"] = (now + timedelta(hours=4)).isoformat()
save_state(state)
print("🛑 Neue Aufgaben erkannt. Suche pausiert für 4h.")
return
# Check: Pausiert?
if state.get("paused_until"):
paused = datetime.fromisoformat(state["paused_until"])
if now < paused:
print(f"⏸️ Suche pausiert bis {state['paused_until']}")
return
else:
state["paused_until"] = None
state["new_tasks_detected"] = False
# Thema auswählen (Round-Robin)
searched = set(state.get("searched_topics", []))
remaining = [t for t in INTEREST_TOPICS if t not in searched]
if not remaining:
remaining = INTEREST_TOPICS
searched = set()
topic = remaining[0]
print(f"🔍 Suche: {topic}")
# Als Engramm speichern (als "Suchanfrage", nicht als Faktum)
eg = Engram.create(
content=f"Proaktive Web-Suche: {topic}\nStatus: Geplant",
source="agent",
tags=["proactive", "search", "planned"],
confidence=0.3,
grounding=Grounding.ASSUMPTION,
)
store.save(eg)
state["last_search"] = now.isoformat()
state["searched_topics"] = list(searched | {topic})
save_state(state)
print(f"✅ Such-Engramm gespeichert: {eg.id}")
if __name__ == "__main__":
run_proactive_search()

View File

@@ -7,12 +7,10 @@ Phase 2: + Embedding + Fusion.
from typing import List, Dict, Any, Optional from typing import List, Dict, Any, Optional
from .engram import Engram from .engram import Engram
from .store import EngramStore from .store import EngramStore
from .chroma_store import ChromaStore
from .embedder import encode
class Retriever: class Retriever:
def __init__(self, store: EngramStore, chroma: Optional[ChromaStore] = None): def __init__(self, store: EngramStore, chroma: Optional[object] = None):
self.store = store self.store = store
self.chroma = chroma self.chroma = chroma
@@ -50,7 +48,6 @@ class Retriever:
if not self.chroma: if not self.chroma:
return [] return []
chroma_results = self.chroma.query(query, top_k=limit * 3) chroma_results = self.chroma.query(query, top_k=limit * 3)
eids = [r["id"] for r in chroma_results]
results = [] results = []
for r in chroma_results: for r in chroma_results:
eg = self.store.get(r["id"]) eg = self.store.get(r["id"])

View File

@@ -6,6 +6,7 @@ Keine externen Abhängigkeiten außer sqlite3 (stdlib).
import json import json
import sqlite3 import sqlite3
import os import os
import re
from pathlib import Path from pathlib import Path
from typing import List, Optional, Dict, Any from typing import List, Optional, Dict, Any
from uuid import UUID from uuid import UUID
@@ -127,6 +128,14 @@ class EngramStore:
).fetchall() ).fetchall()
return [self._row_to_engram(r) for r in rows] return [self._row_to_engram(r) for r in rows]
def get_modified_since(self, iso_ts: str, limit: int = 5000) -> List[Engram]:
"""Gibt Engramme zurück, deren `modified_at` nach `iso_ts` liegt."""
rows = self._conn.execute(
"SELECT * FROM engrams WHERE modified_at > ? ORDER BY modified_at ASC LIMIT ?",
(iso_ts, limit),
).fetchall()
return [self._row_to_engram(r) for r in rows]
def delete(self, engram_id: str) -> bool: def delete(self, engram_id: str) -> bool:
"""Löscht ein Engramm und alle Verknüpfungen.""" """Löscht ein Engramm und alle Verknüpfungen."""
rowid = self._conn.execute( rowid = self._conn.execute(
@@ -150,7 +159,12 @@ class EngramStore:
def search_text(self, query: str, limit: int = 10) -> List[Engram]: def search_text(self, query: str, limit: int = 10) -> List[Engram]:
"""Full-Text-Suche über Engramm-Inhalt via SQLite FTS5 (OR-Verknüpfung).""" """Full-Text-Suche über Engramm-Inhalt via SQLite FTS5 (OR-Verknüpfung)."""
# FTS5-Syntax: Wörter mit OR verbinden für bessere Ergebnisse # FTS5-Syntax: Wörter mit OR verbinden für bessere Ergebnisse
words = [w.strip() for w in query.replace("'", "''").split() if w.strip()] words = []
for word in query.split():
# Nur alphanumerische Zeichen als FTS5-Tokens akzeptieren
clean_word = re.sub(r'[^a-zA-Z0-9]+', '', word)
if clean_word:
words.append(clean_word)
safe_query = " OR ".join(words) if len(words) > 1 else (words[0] if words else "*") safe_query = " OR ".join(words) if len(words) > 1 else (words[0] if words else "*")
sql = """ sql = """
SELECT e.* FROM engrams e SELECT e.* FROM engrams e
@@ -239,6 +253,13 @@ class EngramStore:
"links": json.loads(row["links_json"]), "links": json.loads(row["links_json"]),
"hierarchy": json.loads(row["hierarchy_json"]), "hierarchy": json.loads(row["hierarchy_json"]),
} }
# Keep Engram metadata timestamps aligned with DB columns so downstream
# consumers (e.g. vector indexing watermarks) can rely on them.
try:
d["metadata"]["created"] = row["created_at"]
d["metadata"]["modified"] = row["modified_at"]
except Exception:
pass
emb = row["embedding_json"] emb = row["embedding_json"]
if emb: if emb:
d["embedding"] = json.loads(emb) d["embedding"] = json.loads(emb)

401
static/style.css Normal file
View File

@@ -0,0 +1,401 @@
/* ─── Reset & Base ────────────────────────────────────────────────────────── */
* { margin: 0; padding: 0; box-sizing: border-box; }
html { font-size: 14px; }
body {
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
background: #0f0f12;
color: #e8e8ee;
-webkit-font-smoothing: antialiased;
overscroll-behavior-y: contain;
}
.app {
max-width: 480px;
margin: 0 auto;
min-height: 100vh;
background: #141419;
}
/* ─── Stats Bar ───────────────────────────────────────────────────────────── */
.stats-bar {
display: flex;
justify-content: space-around;
padding: 10px 8px;
background: linear-gradient(180deg, #1a1a22 0%, #131318 100%);
border-bottom: 1px solid #252530;
position: sticky;
top: 0;
z-index: 50;
}
.tabs-bar{
display:flex;
gap:8px;
padding:8px 12px 10px;
background:#141419;
border-bottom:1px solid #252530;
position: sticky;
top: 52px;
z-index: 45;
}
.tabs-bar .tab-btn{
flex:1;
background:#1e1e28;
border:1px solid #2a2a3a;
border-radius: 12px;
padding:10px 10px;
color:#cfd3ff;
font-weight:700;
font-size:0.82rem;
}
.tabs-bar .tab-btn.active{
border-color:#6c8af5;
box-shadow:0 0 0 1px rgba(108,138,245,0.22) inset;
}
.stat {
text-align: center;
min-width: 60px;
}
.stat-num {
display: block;
font-size: 1.3rem;
font-weight: 700;
color: #6c8af5;
line-height: 1.2;
}
.stat-label {
display: block;
font-size: 0.65rem;
color: #888899;
text-transform: uppercase;
letter-spacing: 0.5px;
margin-top: 2px;
}
/* ─── Search ──────────────────────────────────────────────────────────────── */
.search-box {
display: flex;
gap: 8px;
padding: 10px 12px;
background: #141419;
}
/* tab buttons styled via .tabs-bar */
/* ─── Panels (Graph/Status) ──────────────────────────────────────────────── */
.panel {
margin: 8px 12px;
background: #1a1a24;
border: 1px solid #252533;
border-radius: 14px;
padding: 10px 12px;
}
.panel-title {
font-size: 0.8rem;
text-transform: uppercase;
letter-spacing: 0.5px;
color: #9aa3d9;
margin-bottom: 8px;
}
.kv-row {
display: flex;
gap: 10px;
padding: 6px 0;
border-bottom: 1px solid #20202a;
}
.kv-row:last-child { border-bottom: none; }
.kv-key {
width: 110px;
color: #888899;
font-size: 0.78rem;
}
.kv-val {
flex: 1;
color: #e8e8ee;
font-size: 0.85rem;
word-break: break-word;
}
.pill {
display: inline-block;
margin: 2px 4px 2px 0;
padding: 2px 8px;
border-radius: 999px;
background: #2a2a3a;
color: #8a9aff;
font-size: 0.72rem;
}
.muted {
color: #888899;
font-size: 0.8rem;
margin-top: 6px;
}
.small { font-size: 0.75rem; }
/* Graph canvas */
#graphCanvas{
display:block;
margin: 8px auto 0;
background:#12121a;
border:1px solid #252533;
border-radius: 14px;
}
#searchInput {
flex: 1;
background: #1e1e28;
border: 1px solid #2a2a3a;
border-radius: 10px;
padding: 10px 14px;
color: #e8e8ee;
font-size: 0.95rem;
outline: none;
}
#searchInput:focus { border-color: #6c8af5; }
#filterSelect {
background: #1e1e28;
border: 1px solid #2a2a3a;
border-radius: 10px;
padding: 10px;
color: #e8e8ee;
font-size: 0.85rem;
outline: none;
}
/* ─── New Engram ──────────────────────────────────────────────────────────── */
.new-engram {
padding: 0 12px 8px;
display: flex;
flex-direction: column;
gap: 6px;
}
.new-engram textarea {
background: #1e1e28;
border: 1px solid #2a2a3a;
border-radius: 10px;
padding: 10px;
color: #e8e8ee;
font-size: 0.9rem;
resize: vertical;
min-height: 50px;
}
.new-engram input {
background: #1e1e28;
border: 1px solid #2a2a3a;
border-radius: 10px;
padding: 8px 10px;
color: #e8e8ee;
font-size: 0.85rem;
}
.new-engram button {
background: #3a7d3a;
border: none;
border-radius: 10px;
padding: 10px;
color: #fff;
font-weight: 600;
font-size: 0.95rem;
}
/* ─── Cards ───────────────────────────────────────────────────────────────── */
.cards {
padding: 4px 12px;
display: flex;
flex-direction: column;
gap: 10px;
}
.card {
background: #1a1a24;
border: 1px solid #252533;
border-radius: 14px;
overflow: hidden;
transition: transform 0.15s ease, border-color 0.2s ease;
touch-action: manipulation;
}
.card:active { transform: scale(0.985); }
.card.confirmed { border-left: 4px solid #3a7d3a; }
.card.rejected { border-left: 4px solid #8a3a3a; }
.card-header {
display: flex;
align-items: center;
gap: 8px;
padding: 10px 12px 6px;
border-bottom: 1px solid #20202a;
}
.conf-badge {
font-size: 0.75rem;
font-weight: 700;
color: #fff;
padding: 2px 8px;
border-radius: 20px;
min-width: 36px;
text-align: center;
}
.tags {
flex: 1;
display: flex;
flex-wrap: wrap;
gap: 4px;
}
.tag {
background: #2a2a3a;
color: #8a9aff;
font-size: 0.68rem;
padding: 2px 8px;
border-radius: 12px;
text-transform: uppercase;
letter-spacing: 0.3px;
}
.date {
font-size: 0.65rem;
color: #666677;
white-space: nowrap;
}
.card-body {
padding: 10px 12px;
font-size: 0.9rem;
line-height: 1.45;
color: #ccccdd;
cursor: pointer;
}
.card-footer {
display: flex;
align-items: center;
gap: 8px;
padding: 8px 12px 10px;
border-top: 1px solid #20202a;
}
.reason-input {
flex: 1;
background: #14141a;
border: 1px solid #2a2a3a;
border-radius: 8px;
padding: 6px 10px;
color: #b0b0c0;
font-size: 0.8rem;
}
.actions {
display: flex;
gap: 6px;
}
.actions button {
width: 34px;
height: 34px;
border-radius: 50%;
border: none;
font-size: 1rem;
cursor: pointer;
display: flex;
align-items: center;
justify-content: center;
transition: transform 0.1s;
}
.actions button:active { transform: scale(0.9); }
.btn-ok { background: #2a5a2a; }
.btn-no { background: #5a2a2a; }
.btn-archive { background: #2a2a4a; }
/* ─── Pagination ──────────────────────────────────────────────────────────── */
.pagination {
display: flex;
justify-content: center;
align-items: center;
gap: 16px;
padding: 14px;
}
.pagination button {
background: #252535;
border: none;
border-radius: 10px;
width: 40px;
height: 40px;
color: #e8e8ee;
font-size: 1.1rem;
}
.pagination button:disabled {
opacity: 0.3;
}
#pageNum {
font-weight: 700;
color: #6c8af5;
}
/* ─── Footer ──────────────────────────────────────────────────────────────── */
.footer {
display: flex;
justify-content: space-between;
align-items: center;
padding: 10px 16px 20px;
color: #555566;
font-size: 0.75rem;
}
.refresh-btn {
background: #252535;
border: none;
border-radius: 50%;
width: 36px;
height: 36px;
color: #8888aa;
font-size: 1.2rem;
}
/* ─── Modal ───────────────────────────────────────────────────────────────── */
.modal {
display: none;
position: fixed;
inset: 0;
background: rgba(0,0,0,0.85);
z-index: 100;
overflow-y: auto;
padding: 20px 16px;
}
.modal.open { display: block; }
.modal-content {
background: #1a1a24;
border: 1px solid #333344;
border-radius: 16px;
padding: 20px 16px;
max-width: 480px;
margin: 0 auto;
position: relative;
}
.close-btn {
position: absolute;
top: 10px;
right: 14px;
background: none;
border: none;
color: #8888aa;
font-size: 1.5rem;
cursor: pointer;
}
.history {
list-style: none;
font-size: 0.8rem;
color: #9999aa;
}
.history li {
padding: 4px 0;
border-bottom: 1px solid #20202a;
}
.detail-content {
background: #14141a;
border-radius: 10px;
padding: 12px;
font-size: 0.88rem;
line-height: 1.5;
margin: 8px 0;
max-height: 40vh;
overflow-y: auto;
}
/* ─── Scrollbar ───────────────────────────────────────────────────────────── */
::-webkit-scrollbar { width: 4px; }
::-webkit-scrollbar-track { background: transparent; }
::-webkit-scrollbar-thumb { background: #333344; border-radius: 4px; }
/* ─── Touch fix ───────────────────────────────────────────────────────────── */
@media (pointer: coarse) {
button, .card { -webkit-tap-highlight-color: transparent; }
}

View File

@@ -0,0 +1,9 @@
[Unit]
Description=OpenClaw archive memory/*.md older than 7 days
OnFailure=openclaw-secondbrain-notify@%n.service
[Service]
Type=oneshot
WorkingDirectory=/root/.openclaw/workspace
ExecStart=/bin/bash -lc 'flock -n /tmp/%n.lock /usr/bin/python3 /root/.openclaw/workspace/openclaw_cron_wrapper.py archive_memory_md'

View File

@@ -0,0 +1,10 @@
[Unit]
Description=OpenClaw archive memory/*.md daily
[Timer]
OnCalendar=*-*-* 03:30:00
Persistent=true
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,8 @@
[Unit]
Description=OpenClaw Second-Brain backup_secondbrain
OnFailure=openclaw-secondbrain-notify@%n.service
[Service]
Type=oneshot
WorkingDirectory=/root/.openclaw/workspace
ExecStart=/bin/bash -lc 'flock -n /tmp/%n.lock /usr/bin/python3 /root/.openclaw/workspace/openclaw_cron_wrapper.py backup_secondbrain'

View File

@@ -0,0 +1,10 @@
[Unit]
Description=OpenClaw Second-Brain backup_secondbrain (daily 02:00)
[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,15 @@
[Unit]
Description=OpenClaw Second-Brain Dashboard (FastAPI)
After=network.target
[Service]
Type=simple
WorkingDirectory=/root/.openclaw/workspace/second-brain
Environment=SECOND_BRAIN_WORKSPACE=/root/.openclaw/workspace/second-brain
Environment=SECOND_BRAIN_PORT=8501
ExecStart=/root/.openclaw/workspace/second-brain/.venv/bin/uvicorn fastapi_app:app --host 0.0.0.0 --port 8501
Restart=always
RestartSec=2
[Install]
WantedBy=multi-user.target

View File

@@ -0,0 +1,8 @@
[Unit]
Description=OpenClaw Second-Brain export_obsidian
OnFailure=openclaw-secondbrain-notify@%n.service
[Service]
Type=oneshot
WorkingDirectory=/root/.openclaw/workspace
ExecStart=/bin/bash -lc 'flock -n /tmp/%n.lock /usr/bin/python3 /root/.openclaw/workspace/openclaw_cron_wrapper.py export_obsidian'

View File

@@ -0,0 +1,10 @@
[Unit]
Description=OpenClaw Second-Brain export_obsidian (hourly)
[Timer]
OnCalendar=hourly
Persistent=true
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,8 @@
[Unit]
Description=OpenClaw Second-Brain heartbeat_secondbrain
OnFailure=openclaw-secondbrain-notify@%n.service
[Service]
Type=oneshot
WorkingDirectory=/root/.openclaw/workspace
ExecStart=/bin/bash -lc 'flock -n /tmp/%n.lock /usr/bin/python3 /root/.openclaw/workspace/openclaw_cron_wrapper.py heartbeat_secondbrain'

View File

@@ -0,0 +1,11 @@
[Unit]
Description=OpenClaw Second-Brain heartbeat_secondbrain (every 6h)
[Timer]
OnCalendar=*-*-* 00,06,12,18:00:00
RandomizedDelaySec=300
Persistent=true
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,10 @@
[Unit]
Description=OpenClaw Second-Brain index_vectors
OnFailure=openclaw-secondbrain-notify@%n.service
[Service]
Type=oneshot
WorkingDirectory=/root/.openclaw/workspace
Environment=HF_HOME=/root/.openclaw/workspace/second-brain/data/hf_cache
Environment=SENTENCE_TRANSFORMERS_HOME=/root/.openclaw/workspace/second-brain/data/st_cache
ExecStart=/bin/bash -lc 'flock -n /tmp/%n.lock /usr/bin/python3 /root/.openclaw/workspace/openclaw_cron_wrapper.py index_vectors'

View File

@@ -0,0 +1,10 @@
[Unit]
Description=OpenClaw Second-Brain index_vectors (every 30 min)
[Timer]
OnCalendar=*:0/30
Persistent=true
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,8 @@
[Unit]
Description=OpenClaw Second-Brain ingest_memory
OnFailure=openclaw-secondbrain-notify@%n.service
[Service]
Type=oneshot
WorkingDirectory=/root/.openclaw/workspace
ExecStart=/bin/bash -lc 'flock -n /tmp/%n.lock /usr/bin/python3 /root/.openclaw/workspace/openclaw_cron_wrapper.py ingest_memory'

View File

@@ -0,0 +1,9 @@
[Unit]
Description=OpenClaw Second-Brain ingest_memory (every 5 min)
[Timer]
OnCalendar=*:0/5
Persistent=true
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,8 @@
[Unit]
Description=OpenClaw Second-Brain ingest_obsidian
OnFailure=openclaw-secondbrain-notify@%n.service
[Service]
Type=oneshot
WorkingDirectory=/root/.openclaw/workspace
ExecStart=/bin/bash -lc 'flock -n /tmp/%n.lock /usr/bin/python3 /root/.openclaw/workspace/openclaw_cron_wrapper.py ingest_obsidian'

View File

@@ -0,0 +1,10 @@
[Unit]
Description=OpenClaw Second-Brain ingest_obsidian (every 15 min)
[Timer]
OnCalendar=*:0/15
Persistent=true
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,9 @@
[Unit]
Description=OpenClaw Second-Brain ingest transcript -> DB
OnFailure=openclaw-secondbrain-notify@%n.service
[Service]
Type=oneshot
WorkingDirectory=/root/.openclaw/workspace
ExecStart=/bin/bash -lc 'flock -n /tmp/%n.lock /usr/bin/python3 /root/.openclaw/workspace/openclaw_cron_wrapper.py ingest_transcript_to_db'

View File

@@ -0,0 +1,12 @@
[Unit]
Description=OpenClaw Second-Brain ingest transcript -> DB (every 5 min)
[Timer]
OnBootSec=90s
OnUnitActiveSec=300s
Unit=openclaw-secondbrain-ingest-transcript-to-db.service
Persistent=true
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,8 @@
[Unit]
Description=OpenClaw Second-Brain ingest transcript -> memory/*.md
OnFailure=openclaw-secondbrain-notify@%n.service
[Service]
Type=oneshot
WorkingDirectory=/root/.openclaw/workspace
ExecStart=/bin/bash -lc 'flock -n /tmp/%n.lock /usr/bin/python3 /root/.openclaw/workspace/second-brain/cron_tasks/ingest_transcript_to_memory.py'

View File

@@ -0,0 +1,11 @@
[Unit]
Description=OpenClaw Second-Brain ingest transcript -> memory (every 1 min)
[Timer]
OnBootSec=30s
OnUnitActiveSec=60s
Unit=openclaw-secondbrain-ingest-transcript-to-memory.service
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,8 @@
[Unit]
Description=OpenClaw Second-Brain failure notify (%i)
[Service]
Type=oneshot
WorkingDirectory=/root/.openclaw/workspace
ExecStart=/bin/bash -lc '/root/.openclaw/workspace/notify-telegram.sh "❌ Second-Brain job failed: %i. Check: journalctl -u %i -n 50 --no-pager"'

View File

@@ -0,0 +1,10 @@
[Unit]
Description=OpenClaw Second-Brain proactive_search_wrapper
Wants=network-online.target
After=network-online.target
OnFailure=openclaw-secondbrain-notify@%n.service
[Service]
Type=oneshot
WorkingDirectory=/root/.openclaw/workspace
ExecStart=/bin/bash -lc 'flock -n /tmp/%n.lock /usr/bin/python3 /root/.openclaw/workspace/openclaw_cron_wrapper.py proactive_search_wrapper'

View File

@@ -0,0 +1,11 @@
[Unit]
Description=OpenClaw Second-Brain proactive_search_wrapper (every 4h)
[Timer]
OnCalendar=*-*-* 01,05,09,13,17,21:10:00
RandomizedDelaySec=600
Persistent=true
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,8 @@
[Unit]
Description=OpenClaw Second-Brain review_brain
OnFailure=openclaw-secondbrain-notify@%n.service
[Service]
Type=oneshot
WorkingDirectory=/root/.openclaw/workspace
ExecStart=/bin/bash -lc 'flock -n /tmp/%n.lock /usr/bin/python3 /root/.openclaw/workspace/openclaw_cron_wrapper.py review_brain'

View File

@@ -0,0 +1,9 @@
[Unit]
Description=OpenClaw Second-Brain review_brain (every 5 min)
[Timer]
OnCalendar=*:0/5
Persistent=true
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,13 @@
[Unit]
Description=OpenClaw Second-Brain task (%i)
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
WorkingDirectory=/root/.openclaw/workspace
ExecStart=/usr/bin/python3 /root/.openclaw/workspace/openclaw_cron_wrapper.py %i
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=6

View File

@@ -0,0 +1,9 @@
[Unit]
Description=OpenClaw Second-Brain verify_pending_external
OnFailure=openclaw-secondbrain-notify@%n.service
[Service]
Type=oneshot
WorkingDirectory=/root/.openclaw/workspace
ExecStart=/bin/bash -lc 'flock -n /tmp/%n.lock /usr/bin/python3 /root/.openclaw/workspace/openclaw_cron_wrapper.py verify_pending_external'

View File

@@ -0,0 +1,10 @@
[Unit]
Description=OpenClaw Second-Brain periodic verify_pending_external
[Timer]
OnBootSec=2min
OnUnitActiveSec=5min
Persistent=true
[Install]
WantedBy=timers.target

488
templates/dashboard.html Normal file
View File

@@ -0,0 +1,488 @@
<!DOCTYPE html>
<html lang="de">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width; initial-scale=1.0; maximum-scale=1.0; user-scalable=no">
<title>🧠 Second Brain</title>
<link rel="stylesheet" href="/static/style.css">
</head>
<body>
<div class="app">
<!-- Stats Header -->
<header class="stats-bar" id="statsBar">
<div class="stat"><span class="stat-num" id="statTotal">-</span><span class="stat-label">Total</span></div>
<div class="stat"><span class="stat-num" id="statConfirmed">-</span><span class="stat-label">OK</span></div>
<div class="stat"><span class="stat-num" id="statPending">-</span><span class="stat-label">Pending</span></div>
<div class="stat"><span class="stat-num" id="statErrors">-</span><span class="stat-label">Err</span></div>
</header>
<div class="tabs-bar">
<button class="tab-btn active" id="tabCards" onclick="setView('cards')">Cards</button>
<button class="tab-btn" id="tabGraph" onclick="setView('graph')">Graph</button>
<button class="tab-btn" id="tabStatus" onclick="setView('status')">Status</button>
</div>
<!-- Search -->
<div class="search-box">
<input type="text" id="searchInput" placeholder="🔍 Suche..." />
<select id="filterSelect">
<option value="all">Alle</option>
<option value="pending">Pending</option>
<option value="confirmed">Confirmed</option>
<option value="errors">Errors</option>
</select>
</div>
<!-- New Engram -->
<div class="new-engram">
<textarea id="newContent" placeholder="Neues Engramm..."></textarea>
<input type="text" id="newTags" placeholder="Tags (comma sep)" />
<button onclick="createEngram()"> Speichern</button>
</div>
<!-- Cards -->
<div class="cards" id="cards"></div>
<!-- Graph -->
<div class="graph" id="graph" style="display:none;">
<canvas id="graphCanvas" width="440" height="520"></canvas>
<div class="muted small" id="graphHint">Lade Graph…</div>
</div>
<!-- Status -->
<div class="status" id="status" style="display:none;"></div>
<!-- Pagination -->
<div class="pagination" id="pagination">
<button id="btnPrev" onclick="prevPage()"></button>
<span id="pageNum">1</span>
<button id="btnNext" onclick="nextPage()"></button>
</div>
<div class="footer">
<span id="lastUpdate">--:--</span>
<button onclick="manualRefresh()" class="refresh-btn"></button>
</div>
</div>
<!-- Detail Modal -->
<div class="modal" id="detailModal">
<div class="modal-content">
<button class="close-btn" onclick="closeModal()">×</button>
<div id="modalBody"></div>
</div>
</div>
<script>
// ─── State ──────────────────────────────────────────────────────────────────
let state = {
items: [],
offset: 0,
limit: 10,
filter: 'all',
search: '',
autoRefresh: true,
view: 'cards',
lastEvent: null,
};
// ─── Fetch ──────────────────────────────────────────────────────────────────
async function api(path, opts = {}) {
const r = await fetch(path, opts);
if (!r.ok) throw new Error((await r.json()).error || r.statusText);
return r.json();
}
async function loadStats() {
const s = await api('/api/stats');
document.getElementById('statTotal').textContent = s.total;
document.getElementById('statConfirmed').textContent = s.confirmed;
document.getElementById('statPending').textContent = s.pending;
document.getElementById('statErrors').textContent = s.errors;
}
function updateStatsFromEvent(ev) {
if (!ev || !ev.stats) return;
const s = ev.stats;
document.getElementById('statTotal').textContent = s.total;
document.getElementById('statConfirmed').textContent = s.confirmed;
document.getElementById('statPending').textContent = s.pending;
document.getElementById('statErrors').textContent = s.errors;
}
function setView(view) {
state.view = view;
document.getElementById('tabCards').classList.toggle('active', view === 'cards');
document.getElementById('tabGraph').classList.toggle('active', view === 'graph');
document.getElementById('tabStatus').classList.toggle('active', view === 'status');
document.getElementById('cards').style.display = view === 'cards' ? '' : 'none';
document.getElementById('pagination').style.display = view === 'cards' ? '' : 'none';
document.getElementById('graph').style.display = view === 'graph' ? '' : 'none';
document.getElementById('status').style.display = view === 'status' ? '' : 'none';
if (view === 'graph') loadGraph();
if (view === 'status') loadStatus();
}
async function loadCards() {
let url = `/api/engrams?limit=${state.limit}&offset=${state.offset}`;
if (state.search) url += `&search=${encodeURIComponent(state.search)}`;
if (state.filter === 'confirmed') url += '&confirmed=1';
if (state.filter === 'pending') url += '&confirmed=0';
if (state.filter === 'errors') url += '&tag=error';
const data = await api(url);
state.items = data.items;
renderCards();
document.getElementById('pageNum').textContent = Math.floor(state.offset / state.limit) + 1;
document.getElementById('btnPrev').disabled = state.offset === 0;
document.getElementById('btnNext').disabled = data.items.length < state.limit;
}
async function loadStatus() {
const [cfg, db, jobs, ins, stor] = await Promise.all([
api('/api/config'),
api('/api/db_info'),
api('/api/jobs'),
api('/api/insights?limit=8'),
api('/api/storage_stats'),
]);
const pend = await api('/api/pending?limit=20&offset=0');
const el = document.getElementById('status');
const jobsHtml = (jobs.items || []).map(j => `
<div class="kv-row">
<div class="kv-key">${j.unit}</div>
<div class="kv-val">${j.error ? ('ERR: ' + j.error) : (j.active + '/' + j.sub)}</div>
</div>
`).join('');
const topTags = (ins.top_tags || []).map(t => `<span class="pill">${t.key} (${t.count})</span>`).join(' ');
const topHosts = (ins.top_hosts || []).map(t => `<span class="pill">${t.key} (${t.count})</span>`).join(' ');
const bySource = Object.entries((stor.sql && stor.sql.by_source) ? stor.sql.by_source : {})
.slice(0, 8)
.map(([k,v]) => `<span class="pill">${k}: ${v}</span>`)
.join(' ');
const pendItems = (pend.items || []);
const pendHtml = pendItems.map(p => `
<div class="kv-row" onclick="showDetail('${p.id}')">
<div class="kv-key">${(p.source||'').slice(0,12)}</div>
<div class="kv-val">
<span class="pill">${p.id.substring(0,8)}</span>
${escapeHtml((p.content||'').substring(0,120))}${(p.content||'').length>120?'…':''}
<div class="actions" style="margin-top:6px" onclick="event.stopPropagation()">
<button class="btn-ok" onclick="confirm('${p.id}', event)">✅</button>
<button class="btn-no" onclick="reject('${p.id}', event)">❌</button>
</div>
</div>
</div>
`).join('');
el.innerHTML = `
<div class="panel">
<div class="panel-title">Config</div>
<div class="kv-row"><div class="kv-key">workspace</div><div class="kv-val">${cfg.workspace}</div></div>
<div class="kv-row"><div class="kv-key">db</div><div class="kv-val">${db.db_path}</div></div>
<div class="kv-row"><div class="kv-key">db mtime</div><div class="kv-val">${new Date(db.mtime).toLocaleString()}</div></div>
</div>
<div class="panel">
<div class="panel-title">Storage</div>
<div class="kv-row"><div class="kv-key">SQL</div><div class="kv-val">${stor.sql.total_engrams} engrams (ok ${stor.sql.confirmed}, pending ${stor.sql.pending})</div></div>
<div class="kv-row"><div class="kv-key">Vector</div><div class="kv-val">chroma ${(stor.vector.chroma_size_bytes/1024/1024).toFixed(1)} MB, cache ${stor.vector.embedding_cache_files} files</div></div>
<div class="kv-row"><div class="kv-key">Obsidian</div><div class="kv-val">${stor.obsidian.configured ? 'configured' : 'not configured'}</div></div>
<div class="kv-row"><div class="kv-key">By source</div><div class="kv-val">${bySource || '-'}</div></div>
</div>
<div class="panel">
<div class="panel-title">Jobs</div>
${jobsHtml || '<div class="muted">Keine Daten</div>'}
</div>
<div class="panel">
<div class="panel-title">Insights</div>
<div class="kv-row"><div class="kv-key">pending</div><div class="kv-val">${ins.pending}</div></div>
<div class="kv-row"><div class="kv-key">top tags</div><div class="kv-val">${topTags || '-'}</div></div>
<div class="kv-row"><div class="kv-key">top hosts</div><div class="kv-val">${topHosts || '-'}</div></div>
</div>
<div class="panel">
<div class="panel-title">Pending Queue (latest)</div>
${pendHtml || '<div class="muted">Keine Pendings</div>'}
</div>
`;
}
async function loadGraph() {
const g = await api('/api/graph?limit_nodes=200');
renderGraph(g.nodes || [], g.edges || []);
}
function renderGraph(nodes, edges) {
const canvas = document.getElementById('graphCanvas');
const hint = document.getElementById('graphHint');
const ctx = canvas.getContext('2d');
// Fit canvas to container width (mobile)
const w = canvas.parentElement.clientWidth - 24;
canvas.width = Math.max(320, Math.min(520, w));
canvas.height = 520;
if (!nodes.length || !edges.length) {
hint.textContent = 'Graph: keine Kanten (noch keine Links/Tags/Hosts im Sample).';
ctx.clearRect(0,0,canvas.width,canvas.height);
return;
}
hint.textContent = `nodes=${nodes.length} edges=${edges.length}`;
const nodeById = new Map(nodes.map(n => [n.id, n]));
const sim = nodes.map(n => ({
id: n.id,
kind: n.kind,
label: n.label || n.id,
x: Math.random()*canvas.width,
y: Math.random()*canvas.height,
vx: 0, vy: 0,
}));
const simById = new Map(sim.map(n => [n.id, n]));
const links = edges
.map(e => ({a: simById.get(e.from), b: simById.get(e.to), kind: e.kind}))
.filter(l => l.a && l.b);
// Simple force layout (few iterations)
for (let iter=0; iter<180; iter++) {
// repulsion
for (let i=0; i<sim.length; i++) {
for (let j=i+1; j<sim.length; j++) {
const a = sim[i], b = sim[j];
const dx = a.x - b.x, dy = a.y - b.y;
const d2 = dx*dx + dy*dy + 0.01;
const f = 120 / d2;
a.vx += dx*f; a.vy += dy*f;
b.vx -= dx*f; b.vy -= dy*f;
}
}
// springs
for (const l of links) {
const a = l.a, b = l.b;
const dx = b.x - a.x, dy = b.y - a.y;
const dist = Math.sqrt(dx*dx + dy*dy) || 1;
const target = 60;
const k = 0.02;
const f = (dist - target) * k;
const fx = (dx/dist)*f, fy = (dy/dist)*f;
a.vx += fx; a.vy += fy;
b.vx -= fx; b.vy -= fy;
}
// integrate + bounds
for (const n of sim) {
n.vx *= 0.85; n.vy *= 0.85;
n.x += n.vx; n.y += n.vy;
n.x = Math.max(10, Math.min(canvas.width-10, n.x));
n.y = Math.max(10, Math.min(canvas.height-10, n.y));
}
}
ctx.clearRect(0,0,canvas.width,canvas.height);
// edges
ctx.globalAlpha = 0.5;
ctx.strokeStyle = '#3a3a55';
ctx.lineWidth = 1;
for (const l of links) {
ctx.beginPath();
ctx.moveTo(l.a.x, l.a.y);
ctx.lineTo(l.b.x, l.b.y);
ctx.stroke();
}
ctx.globalAlpha = 1.0;
// nodes
for (const n of sim) {
let r = 5;
let fill = '#6c8af5';
if (n.kind === 'tag') { fill = '#8a9aff'; r = 4; }
if (n.kind === 'host') { fill = '#f5b46c'; r = 4; }
if (n.kind === 'engram') { fill = '#6c8af5'; r = 5; }
ctx.beginPath();
ctx.fillStyle = fill;
ctx.arc(n.x, n.y, r, 0, Math.PI*2);
ctx.fill();
}
}
// Real-time updates via SSE
function startEvents() {
try {
const es = new EventSource('/api/events');
es.onmessage = (msg) => {
try {
state.lastEvent = JSON.parse(msg.data);
updateStatsFromEvent(state.lastEvent);
if (state.view === 'status') {
// refresh status panels without heavy re-render: just rerun loadStatus occasionally
loadStatus();
}
if (state.view === 'graph') {
// fetch graph less often (every ~15s)
const t = Date.now();
if (!state._lastGraphFetch || (t - state._lastGraphFetch) > 15000) {
state._lastGraphFetch = t;
loadGraph();
}
}
} catch (e) {}
};
es.onerror = () => {
// keep UI usable even if SSE drops
};
} catch (e) {}
}
startEvents();
function renderCards() {
const el = document.getElementById('cards');
el.innerHTML = state.items.map(item => `
<div class="card ${item.confirmed ? 'confirmed' : ''} ${item.rejections > 0 ? 'rejected' : ''}" data-id="${item.id}">
<div class="card-header">
<span class="conf-badge" style="background:hsl(${item.confidence*120},70%,40%)">${Math.round(item.confidence*100)}%</span>
<span class="tags">${item.tags.map(t => '<span class="tag">'+t+'</span>').join('')}</span>
<span class="date">${fmtDate(item.created)}</span>
</div>
<div class="card-body" onclick="showDetail('${item.id}')">
${escapeHtml(item.content.substring(0, 200))}${item.content.length>200?'...':''}
</div>
<div class="card-footer">
<input type="text" class="reason-input" placeholder="Grund (optional)" id="reason-${item.id}"/>
<div class="actions">
<button class="btn-ok" onclick="confirm('${item.id}', event)">✅</button>
<button class="btn-no" onclick="reject('${item.id}', event)">❌</button>
<button class="btn-archive" onclick="refresh('${item.id}', event)">🔄</button>
</div>
</div>
</div>
`).join('');
}
function fmtDate(iso) {
const d = new Date(iso);
return `${d.getDate().toString().padStart(2,'0')}.${(d.getMonth()+1).toString().padStart(2,'0')} ${d.getHours().toString().padStart(2,'0')}:${d.getMinutes().toString().padStart(2,'0')}`;
}
function escapeHtml(t) {
const d = document.createElement('div');
d.textContent = t;
return d.innerHTML;
}
// ─── Actions ────────────────────────────────────────────────────────────────
async function confirm(id, ev) {
ev.stopPropagation();
const reason = document.getElementById('reason-'+id).value;
await api(`/api/engrams/${id}/confirm`, {
method: 'POST',
headers: {'Content-Type': 'application/x-www-form-urlencoded'},
body: new URLSearchParams({reason})
});
await loadCards(); await loadStats();
}
async function reject(id, ev) {
ev.stopPropagation();
const reason = document.getElementById('reason-'+id).value;
await api(`/api/engrams/${id}/reject`, {
method: 'POST',
headers: {'Content-Type': 'application/x-www-form-urlencoded'},
body: new URLSearchParams({reason})
});
await loadCards(); await loadStats();
}
async function refresh(id, ev) {
ev.stopPropagation();
await api(`/api/engrams/${id}/refresh`, {method: 'POST'});
await loadCards();
}
async function createEngram() {
const content = document.getElementById('newContent').value;
const tags = document.getElementById('newTags').value;
if (!content.trim()) return;
await api('/api/engrams', {
method: 'POST',
headers: {'Content-Type': 'application/x-www-form-urlencoded'},
body: new URLSearchParams({content, tags})
});
document.getElementById('newContent').value = '';
document.getElementById('newTags').value = '';
await loadCards(); await loadStats();
}
async function showDetail(id) {
const item = await api(`/api/engrams/${id}`);
const body = document.getElementById('modalBody');
body.innerHTML = `
<h3>Engramm ${item.id.substring(0,8)}</h3>
<p><b>Confidence:</b> ${Math.round(item.confidence*100)}%</p>
<p><b>Confirmed:</b> ${item.confirmed ? '✅' : '❌'}</p>
<p><b>Tags:</b> ${item.tags.map(t => '<span class="tag">'+t+'</span>').join(' ')}</p>
<p><b>Content:</b></p>
<div class="detail-content">${escapeHtml(item.content)}</div>
<p><b>History:</b></p>
<ul class="history">
${(item.review_history || []).map(h => `<li>${fmtDate(h.at)}${h.action} (${h.note})</li>`).join('')}
</ul>
<p><b>Links:</b> ${item.links?.join(', ') || 'none'}</p>
`;
document.getElementById('detailModal').classList.add('open');
}
function closeModal() {
document.getElementById('detailModal').classList.remove('open');
}
// ─── Pagination ─────────────────────────────────────────────────────────────
function nextPage() {
state.offset += state.limit;
loadCards();
}
function prevPage() {
state.offset = Math.max(0, state.offset - state.limit);
loadCards();
}
function manualRefresh() {
loadCards(); loadStats();
}
// ─── Search ─────────────────────────────────────────────────────────────────
document.getElementById('searchInput').addEventListener('input', (e) => {
state.search = e.target.value;
state.offset = 0;
loadCards();
});
document.getElementById('filterSelect').addEventListener('change', (e) => {
state.filter = e.target.value;
state.offset = 0;
loadCards();
});
// ─── Auto Refresh ───────────────────────────────────────────────────────────
setInterval(() => {
if (!state.autoRefresh) return;
loadStats();
loadCards();
const now = new Date();
document.getElementById('lastUpdate').textContent =
`${now.getHours().toString().padStart(2,'0')}:${now.getMinutes().toString().padStart(2,'0')}`;
}, 5000);
// ─── Init ───────────────────────────────────────────────────────────────────
loadStats();
loadCards();
</script>
</body>
</html>

View File

@@ -4,16 +4,11 @@
import sys import sys
import os import os
import tempfile import tempfile
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src")) sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
try: from src.engram import Engram, Grounding, Correctness
from src.engram import Engram, Grounding, Correctness from src.store import EngramStore
from src.store import EngramStore from src.retriever import Retriever
from src.retriever import Retriever
except ImportError:
from engram import Engram, Grounding, Correctness
from store import EngramStore
from retriever import Retriever
def test_engram_creation(): def test_engram_creation():