Give Your Local AI Shared Memory with a Librarian Agent

Local models can't follow complex memory specs

Frontier models like Claude Opus can handle tool instructions with half-baked specs, but regular local models struggle to follow memory system specifications precisely while also doing their actual job. This dilutes their intelligence across too many simultaneous tasks.

Why RAG isn't the answer for personal memory

RAG uses semantic vector matching for fast retrieval across huge static datasets, but personal memory is different—it requires careful linked retrieval, precise refinement using the AI's intelligence, and constant evolution. RAG trades away the LLM's reasoning power for speed, which doesn't fit how memory should work.

The amnesia problem with MCP tools

When a client model connects via MCP, it only sees tool names (query, add, update) but has no idea what knowledge actually exists in the memory system. Without knowing useful information is stored, the model never thinks to query it and instead answers from its own context.

Separate the memory burden from the AI

Instead of asking the local model to manage memory, create a dedicated librarian agent that handles all memory organization, indexing, and retrieval. The main AI just asks the librarian for information, the same way a person asks a librarian for books without knowing how the library is organized.

Three MCP tools for memory interaction

Query (read-only): any AI system looks up information. Update memory (read and edit): AI refines outdated memories. Add memory (read, edit, write): AI creates new knowledge entries. The librarian handles all complexity, keeping the client AI's context clean.

Three MCP tools handle all memory operations through the librarian

Deterministic rules prevent drift

The system uses fixed, mechanical rules (not AI decisions) for file organization, indexing, and linking. This ensures the memory structure stays consistent and never drifts away from the OKF spec, even if the underlying model changes.

Seed memory solves the discovery problem

At session startup, generate a small live summary of what's in the knowledge base and push it through MCP instructions and tool descriptions. Now the model opens already knowing 'there's a segment about billing, people, infrastructure' and starts querying instead of guessing.

Two rules prevent memory fragmentation

First, enrich before you create: search for the concept a fact belongs to and patch it in rather than spawning a new file. Second, link both ways: when creating genuinely new concepts, go back and update related ones bidirectionally so nothing stays isolated.

Memory enrichment prevents orphaned, disconnected facts

Model choice matters for agentic decisions

Gemma 12B handled the mechanics but struggled with agentic decisions about memory organization. Qwen 3.6 35B succeeded at deciding what to enrich, what to link, and how to structure knowledge without hand-holding.

Qwen 35B succeeded where Gemma 12B struggled with agentic memory decisions

Lint tool detects orphans and broken links

A completely deterministic step walks all links between concepts and flags orphans (concepts nothing links to) and broken links (pointing nowhere). Auto-generated index files must be excluded as link sources, otherwise orphan detection becomes useless.

Maintain tool auto-repairs the graph

When lint finds orphans, an internal agent automatically wires them back into related concepts bidirectionally. This keeps the memory graph healthy without manual intervention.

Mechanical contradiction resolution

When facts contradict (e.g., moving from Hyderabad to Bangalore), the rule is mechanical: the old fact must be gone everywhere. Rewrite the entire concept if needed so it cleanly reflects only the current truth.

Mechanical contradiction resolution keeps facts current

All local AI agents share one memory layer

Because memory goes through the MCP layer, every AI tool in your stack accesses the same knowledge base. Whichever tool you open already knows your projects, contacts, and what you've been working on across all other tools.

Fully local, no cloud APIs in the chain

The librarian connects to the same llama.cpp server as everything else, auto-discovers the loaded model, works behind llama swap, and never triggers a model swap for memory queries. The entire loop—harness, memory, model—runs on your own box.

Personal AI means owning what it knows

Running a model locally is only half of personal AI. The other half is owning the memory—stored as plain markdown on your disk that you can read yourself. This transforms a generic chatbot into your personal AI that knows you across different tools.

Cache layer for recent memories

A hot tier of most recent memories that the agent keeps close, so it never has to dig through the whole library for facts from the last few minutes. Recent context stays instantly accessible.

Dreaming feature for background consolidation

When idle at certain times of day, the memory reorganizes itself, cleans things up, and links concepts the way a human brain consolidates memories during sleep. The system gets better at remembering while you're not using it.