Mnemosyne MCP Server: Async Ingestion

Level 2 (Topic) — Background worker pool and deduplication mechanism.

Concept

ingest_memory does not synchronously write to the database. Instead, it enqueues the work into a background worker pool so the caller gets an immediate response. This improves perceived latency but means ingestion failures are invisible to the caller.

Architecture

Caller → ingest_memory tool
              │
              ▼
     IngestMemoryAsync()
     ├─ generate content_hash (SHA-256)
     ├─ check in-memory cache (10-min TTL)
     ├─ check DB for existing hash
     ├─ if exists → return "already_exists"
     └─ enqueue to buffered channel (capacity: 100)
              │
              ▼ (background goroutine)
     processIngestTask()
     ├─ format composite text
     ├─ embed via Gemini API (15s timeout)
     ├─ INSERT into pgvector
     └─ log errors (never reported back)

Deduplication

Two-layer dedup prevents re-ingesting identical content:

  1. In-memory cache: map[string]time.Time — content_hash → timestamp. 10-min TTL. Checked first (fast path).
  2. DB constraint: memories.content_hash has a unique constraint. InsertMemory uses ON CONFLICT DO NOTHING.

SHA-256 is computed from the raw content string.

Known Issues

Silent failure

The caller receives {"status": "queued"} immediately. If the Gemini API call fails or the DB insert fails, the error is logged but never reported. There is no status-polling or callback mechanism.

Unbounded cache

The in-memory cache map is never explicitly evicted after the 10-minute TTL. Under sustained ingestion of unique content, the map grows without bound. Low risk at current TazLab scale but architecturally imperfect.

No graceful shutdown

The server does not handle SIGTERM/SIGINT. The background worker goroutine is not cleanly drained. In-flight embeddings may be lost during shutdown.

Code Paths

FileFunctionRole
internal/logic/logic.goIngestMemoryAsync()Producer: dedup check + enqueue
internal/logic/logic.goprocessIngestTask()Consumer: embed + store
internal/logic/logic.gocache mapIn-memory dedup cache

See Also