I built a local RAG system months ago so Claude Code could search past sessions. What it actually searches, most of the time, is a pile of raw transcripts. A session summary gets written, embedded, and dropped into pgvector as one more blob of text. It works for “did I do something like this before,” but it’s not memory. It’s an archive.
The difference matters once the archive gets big. A raw log from three months ago about a debugging session has the right words in it somewhere, but they’re buried in back-and-forth, dead ends, and the paragraph where I was thinking out loud. Semantic search still finds it, sort of. It just returns something noisier than it should.
So I built a second stage: something that takes the raw logs and turns them into a knowledge base worth the name.
Source, compiler, executable
The mental model I landed on is a compiler pipeline, and it’s blunt enough to actually guide the code:
| |
daily/ is append-only. One markdown file per day, sessions stacked underneath each other as they happen. Nobody edits it by hand and nothing deletes from it. That’s the source code: messy, chronological, complete.
knowledge/ is what a nightly job produces from it: short, structured articles with frontmatter, organized into concepts/ for atomic facts and connections/ for the cross-cutting stuff, the kind of insight that only shows up once you’ve seen the same problem twice. That’s the executable. It’s not just an extraction of the source. It’s a build artifact, meant to be queried, not read start to finish.
Treating raw and compiled memory as two different layers, instead of one, turned out to be the whole point.
graph LR
A["Claude Code session"] -->|"session-end / pre-compact hook"| B["flush.py
Gemini Flash summarize"]
B -->|"append"| C["daily/YYYY-MM-DD.md"]
C -->|"nightly cron"| D["compile.py
Gemini Flash extract"]
D -->|"3-7 articles"| E["knowledge/concepts/
knowledge/connections/"]
C -->|"save raw"| F[("pgvector RAG
diary entry")]
E -->|"save distilled"| F
E -->|"mirror"| G["Obsidian vault"]
style D fill:#f3e5f5,stroke:#6a1b9a
style F fill:#e3f2fd,stroke:#1565c0
style G fill:#fff3e0,stroke:#e65100
Capturing at the edges
The tricky part isn’t the nightly job. It’s getting the raw material in the first place, at the moments a session actually ends or its context gets thrown away.
Two hooks handle that. hooks/session-end.py fires when a session closes normally. hooks/pre-compact.py fires right before Claude Code compacts the context window, the one moment conversation history is genuinely about to be lost rather than just sitting unread. Both hand off to scripts/flush.py, which sends the recent conversation to Gemini Flash for summarization and appends the result to today’s daily/YYYY-MM-DD.md.
Using the cheap, fast model here is deliberate. This runs on every session end and every compaction, sometimes several times a day. It doesn’t need to be clever, just produce a decent paragraph of “here’s what happened” without cost or a delay I’d notice while closing a session.
The daily file itself follows a fixed shape, so the next stage has something predictable to parse:
| |
Append-only, one file per day. If I have three sessions in a day, they stack up under the same date. Nothing here is meant to be pretty. It’s meant to be complete.
Deciding what’s worth keeping
scripts/compile.py runs nightly via cron. It reads whatever daily logs are new or changed since the last run, and for each one, asks Gemini Flash to extract knowledge articles: not a summary of the summary, but 3 to 7 distinct, structured articles, saved into knowledge/concepts/ or knowledge/connections/.
The 3-7 range is the part that actually does the work. A day can contain one real insight or six. Forcing a range instead of “extract everything” or “extract one thing” makes the model prioritize. It has to decide that a passing comment about a config flag isn’t article-worthy but a decision to switch a retry strategy is. Without that constraint, the lazy failure mode is either dumping the whole day back out restructured, which defeats the purpose, or grabbing one vague theme and calling it done.
Each article gets frontmatter and a fixed section layout:
| |
That’s a constructed example, not a real one, but it matches the shape every real article takes: title, slug, tags, the source daily file, an update date, a short bulleted summary, a couple of paragraphs of detail, links to related articles, and a citation back to the specific line in the daily log it came from. The schema requires at least two [[wikilinks]] to other concepts per article, which forces the model to check what already exists instead of writing every article as if it’s the first one. That single rule does more than anything else to stop the knowledge base from turning into a pile of disconnected notes that all restate the same idea with a different title.
concepts/ holds the atomic stuff, one fact or one pattern per article. connections/ is for the ideas that only make sense once you’ve connected two separate sessions, the kind of thing you notice on the fifth time you hit a similar problem, not the first.
Where it ends up living, twice
Every compiled article gets saved to the pgvector RAG store, the same one behind the rag_save tool I built for direct document ingestion. That’s what makes it queryable. Ask Claude Code something during a session and it can retrieve the distilled article, not the raw transcript it came from.
But it also gets mirrored to Obsidian as plain markdown, wikilinks and all. That felt a little absurd to build. I wrote a whole post about deleting my Obsidian vault because keeping two systems in sync was exactly the drift I didn’t want to maintain. Here I am, writing files back into a vault on purpose.
The difference is what each copy is for. The database is for retrieval: a query comes in, an embedding gets compared, the best match comes back. Good for “what have I learned about X,” bad for “let me wander through what I know about CI in general.” Obsidian is for browsing: click through wikilinks, notice a cluster of concepts forming around a topic without knowing what to search for first. One answers a question you already know how to phrase. The other is for the question you haven’t formed yet, and I open it rarely enough that this is a fair trade for not maintaining it by hand.
The raw daily log also gets saved to RAG directly, as a diary entry, separate from the compiled articles. So both versions survive: the distilled article for querying, and the messy original in case the compiler missed something or got it wrong.
Backfilling the gap
The hooks only started capturing sessions going forward. There were weeks of daily logs from before this existed, sitting in whatever ad hoc note format I was using at the time, with nothing compiled from them.
scripts/backfill.py exists for exactly that: a one-off run over historical logs, same extraction path as the nightly job, just pointed at the backlog instead of “since yesterday.” I ran it once, checked the output looked sane, and haven’t needed it since. It’s the kind of script that’s only worth having if you expect to need it exactly once, which describes most backfill scripts.
What I’d do differently
The extraction quality depends entirely on what Gemini Flash decides is worth 3 to 7 articles on a given day, and it’s not always right. Some nights it splits one idea into three articles that all say almost the same thing with different titles. I haven’t added deduplication for that yet, just an occasional manual cleanup pass through knowledge/concepts/ when the index file starts looking repetitive.
There’s also no feedback loop. The compiler never learns that an article it wrote turned out to be wrong or unused. It just keeps producing more articles, and the good ones and the mediocre ones sit next to each other with equal-looking frontmatter. A usage counter, bump it whenever rag_search actually returns an article, would probably tell me more about what’s worth keeping than rereading the index ever will.
Cron running it nightly also means I never see the extraction happen. I get the result the next morning, which is fine for how I use it, but it makes debugging a bad extraction annoying: by the time I notice an article is off, the daily log it came from might already be a week old and I’ve forgotten the context myself. --file and --dry-run flags on compile.py help, but only if I remember to check.