Emily is my personal assistant. She keeps a semantic memory of what comes up in our conversations, manages tasks, syncs my calendar, runs scheduled jobs, agents and workflows, and transcribes and speaks voice messages. One of her features is a second brain, a searchable document archive: PDFs, scanned contracts, spreadsheets, screenshots and notes go into an index, and from then on Emily answers from those documents and names the file each answer came from. This article is about the second brain.
The whole system runs on one base Mac Mini M4 with 16 GB of unified RAM, shared between CPU and GPU. The agent is Claude Code, started on the machine by the Claude Agent SDK. Tools, hooks, the database, the original files, the index and the local models all live there. The only traffic that leaves is the inference call to Anthropic's API, which carries the conversation plus whatever the tools added to it. Running inference locally is a choice I made for cost and privacy; with cloud models the RAM problem wouldn't exist.
RAM is the constraint behind the whole project. 16GB have to hold PostgreSQL, the API process, the inference runtime with two resident models and a cross-encoder loaded on demand, all at the same time. And the numbers the tools report turned out to be unreliable: ollama ps put the chat model at 6.4 GB, while footprint measured 16, of which 7.6 was prompt cache. Every decision described here came out of that gap and was made by measuring. The same habit turned up five defects that no functional test would have caught, because the system answered, cited documents and looked like it worked.
At a glance
- Hardware: Mac Mini M4, 16 GB unified RAM. Three local models plus macOS OCR; Claude Code runs on the machine and only Claude's inference goes through Anthropic's API.
- A 143,000 character document enters the retrieval index in 31 s (extraction, chunking, embedding) and can be retrieved from that moment.
- Analysis (summary, description, categories) runs afterwards in the background and takes 487 s. Both are paid once, at upload.
- A search takes about 6 s with warm models; the cross-encoder accounts for 5.8 of them.
- On the reference baseline (84 documents, 40 questions) the expected document is in the top three for 33 of 40 questions, and for 16 of 20 when question and document are in different languages.
- Measuring turned up five defects, and four techniques borrowed from LlamaIndex and RAGFlow were dropped after measurement.
The article describes Emily's retrieval (RAG) system, which is one of many possible designs: how it is built, why, and what each step costs in RAM and time on this machine. It assumes you know language models and relational databases; retrieval-specific concepts are defined where they first appear. There are no install instructions and no public benchmark comparisons.
1. A system sized to its hardware
The constraints below exist because I chose to run everything except the conversation locally. With embeddings, reranking and document analysis behind an API, the RAM problem would go away, and I decided against it for two reasons. The first is cost. Ingestion calls a model for every chunk and for every group it summarises, and changing a parameter means re-indexing or re-analysing the corpus; locally those calls cost time, through an API they would cost tokens every time. The second is privacy. The full documents, the index and every ingestion call stay on the machine, and only what a search returns reaches Anthropic's API: passages, descriptions, summaries. That privacy is partial, and section 9 describes where the line is.
What follows is not the best possible implementation. It's the best compromise I found for a base M4 with 16 GB, and several choices would flip on a bigger machine. Three constraints decided nearly all of them.
The first is RAM. Besides the processes above, the 16 GB also hold the web dashboard and the operating system. Every gigabyte a model gets is taken from something else, and once memory pressure pushes the system into swap, latency goes up by an order of magnitude. So no model stays resident out of convenience. Each has a residency window set from how often it's actually used.
The second is language. I write in Italian, and the corpus is split roughly half Italian, half English, often inside the same document. Multilingual support is therefore a hard requirement for every model, and it shrinks the field a lot. Among the models that fit the RAM budget, few handle Italian well, and fewer still can match an Italian query against English text. The chat model I use is not the most RAM-efficient one I evaluated. It's the most efficient one that meets this requirement and the vision requirement.
The third is compute. The GPU on a base M4 is small, and on this hardware a call's latency is dominated by prefill, reading the input tokens, more than by generation. The timings in this article (5.8 s to rerank 20 candidates, about 44 s per summary call) belong to this chip. Where a decision would reverse with more RAM or more compute, I say so in an "On a different machine" box.
Figure 1. The 16 GB budget.
2. Emily's architecture
The second brain is one of Emily's modules. The others in service are the semantic memory, task management, Google Calendar sync, a scheduler for recurring jobs, an inventory of configurable agents with a workflow engine, and the voice pipeline (transcription in, speech out). An email inbox and development agents are in progress.
The main interface is Discord. A gateway receives the message and forwards it to a FastAPI service, which queues it in PostgreSQL. A worker per channel claims the message with SELECT … FOR UPDATE SKIP LOCKED and, through the Claude Agent SDK, starts a Claude Code session on the machine with Emily's persona and tools. The reply travels back the same way. Mission Control, a Next.js dashboard, shows the queue, documents, memories, scheduled jobs and configuration.
Any tool with an external effect, creating a calendar event for example, goes through an approval manager and waits until I approve it.
Figure 2. Emily's architecture.
3. Anatomy of a turn
One point needs clearing up before getting into the second brain, because it's the one people most often get wrong: nothing routes the question. There's no classifier and no rule deciding whether a message needs the documents. The message reaches Claude, and Claude decides whether to call a tool, and which.
A turn has three phases.
When a session opens, Claude Code's UserPromptSubmit hook injects the semantic memory entries marked as core: facts that should shape every answer and that the model would have no reason to search for. It's a SELECT with no query and no model, and its output stays in the session like a message.
During the turn, Claude can answer directly or call one of three tools. search_memory runs a hybrid search over the memories extracted from past conversations. search_knowledge searches the second brain and returns the relevant passages with the list of documents they came from. get_document opens one specific document when I name it.
At the end, the Stop hook takes my last message and, if it's at least 20 characters long, passes it to the local chat model, which pulls out any facts worth keeping. A fact is saved only if it's new: one the memory already holds is dropped, and one that updates an existing memory replaces it.
The asymmetry between Emily's semantic memory and the corpus is deliberate. Core memories are injected unasked because I wrote them, so they're trusted. Documents enter the context only through an explicit call, because they're external content and therefore a vector for indirect prompt injection. Keeping them behind a tool makes that surface countable and easy to watch (section 9).
Figure 3. A turn in Emily.
Phase
Model
Where it runs
Conversation and tool choice
Claude
Claude Code on the Mac Mini; inference through Anthropic's API
Query embedding
qwen3-embedding:0.6b
Ollama
Candidate reranking
Qwen3-Reranker-0.6B
API process, GPU (
mps
)
Memory extraction at the end of the turn
qwen3-vl:8b-instruct
Ollama
Document extraction, description and summary
qwen3-vl:8b-instruct
and macOS OCR
outside the turn, in the ingestion queue
The local chat model never takes part in the answer. It works before, on indexing, and after, on memory extraction.
4. The RAM budget
The models
Local inference is served by Ollama, with two exceptions that run inside the API process.
Role
Model
On disk
Resident
Context
Vision, descriptions, summaries, memory extraction
qwen3-vl:8b-instruct
6.1 GB
6.4 GB
8,192 tokens
Embedding
qwen3-embedding:0.6b
639 MB
2.4 GB
4,096 tokens
Reranking, outside Ollama
Qwen3-Reranker-0.6B
1.1 GB
1.07 GB, 1.87 GB peak
n/a
Transcription, outside Ollama
faster-whisper small, int8 compute
~484 MB on disk (FP16 weights)
0.57 GB
n/a
On top of these there's Apple's Vision framework, used only for OCR. It takes no model RAM, and as section 5 shows, it saves more latency than any choice of model.
Requirements before benchmarks
A candidate for the chat model has to pass five requirements before its scores mean anything. They come from the calls the system actually makes, and no public benchmark covers all of them.
- Vision. When OCR returns no text (charts, photos, diagrams), the only option is a model that can read the image.
- Structured output against a JSON schema. Four callers pass a schema, and JSON wrapped in a markdown block makes the parser fall back to a sentinel value with no error and no log line.
- A non-thinking mode that works. Those calls want a verdict, and document analysis caps generation at 300 tokens, a budget that 800 tokens of reasoning use up before any answer appears.
- 8,192 tokens of context within budget, because the KV cache eats the same RAM as everything else.
- Italian and English of comparable quality. Descriptions and summaries have to be written in the document's language, and facts extracted from Italian conversations are stored in English.
Candidates that pass then face a single test of 19 real decisions the system has to make. The model in use gets one wrong.
Two traps cost me more time than the test did. The first was the build. A community MLX conversion of one candidate ignored Ollama's format parameter and returned JSON inside a markdown block, and it ignored num_ctx as well, taking about 12 GB. The official GGUF build of the same model behaved correctly with no code change. The second was the edition. The plain tag produced about 800 tokens of hidden reasoning, roughly 40 s for a trivial extraction against 3.4 s for the -instruct tag, and think: false didn't turn it off. The two tags are separate models with separate weights; the layer digests in the registry manifest differ. So a candidate has to be checked on both counts.
What context costs depends on attention
Attention
Growth
at 8,192 tokens
at 32,768 tokens
Sliding window
nearly flat
~7 GB
~8 GB
Global (model in use)
linear
6.6 GB
10 GB
The model I use has global attention, which makes it the one least able to afford a long context. I picked it for vision and language and accepted that cost. The hierarchical summary in section 5 exists to work around it.
Figure 4. RAM versus context length.
On a different machine. With 32 GB of RAM the model table would look different: a chat model with more parameters in place of qwen3-vl:8b-instruct, larger embedder and cross-encoder variants, and a context window well beyond 8,192 tokens. Model size and context window are both set by the RAM budget.
Reported RAM and real RAM
ollama ps reported 8.8 GB for the two resident models. Measured with footprint, the two processes took 17.2 GB of RAM, with 18.8 GB of a 20.5 GB swap file in use and 0.46 GB free. The chat model broke down like this:
Region
Size
Contents
MALLOC_LARGE
8.2 GB
weights
untagged
VM_ALLOCATE
7.6 GB
the inference engine's prompt cache, default cap 8 GiB
of which KV cache
~1.1 GB
0.14 MiB per token at 8,192 context
None of these regions is clean memory in the macOS virtual memory sense, so under pressure the system can't discard them. It can only compress them or push them to swap. The prompt cache wasn't waste, though. The server log recorded 2,035 lookups and 2,035 hits with an average reuse of 200 tokens, because ingestion always sends the same system prefix and the engine reuses the longest common prefix. So I capped it instead of turning it off: a 1 GB cap keeps the prefix warm and gives about 6.5 GB back to the machine.
Figure 5. Reported RAM and real RAM.
Keeping models loaded forever also turned out to be a promise nobody kept. The embedder got evicted anyway whenever the chat model reloaded under pressure, and then stayed out for half an hour. The current windows are 30 minutes for the chat model and 15 for the embedder, sized from ten hours of logs: one embedding call every 9.6 minutes and one search every 14, both in bursts. Reloading is cheap:
Cold reload
Warm call
Chat model, 8,192 context
4.26 s
0.26 s
Embedder
0.67 s
0.06 s
The cross-encoder follows the same logic: it loads lazily on the first search and is released after 30 minutes idle. Keeping it resident would remove the load after a pause, at the price of 1.07 GB held permanently on top of the chat model's 6.4 GB. That is exactly the configuration that sends the machine into swap.
On a different machine. With 32 or 64 GB the cross-encoder would stay resident, the prompt cache would go back to its default cap, and a model with sliding-window attention and a longer context would make most of the hierarchical summary unnecessary.
5. Ingestion: from file to index
Ingestion turns a file into rows you can query: text extraction, normalisation, splitting into chunks, a 1,024-dimension embedding per chunk, and a write to PostgreSQL. A single worker with concurrency one drains the queue, because on this machine two concurrent model calls mean swap.
Figure 6. The write path.
Extraction: OCR first, vision only on escalation
A PDF with a text layer is read directly. A page with fewer than 16 extractable characters counts as scanned. It's rendered at 2x zoom and passed to Apple Vision, set to the accurate recognition level with language correction, for Italian and English. The page goes up to the vision model only if OCR returns nothing (the content is graphical) or returns a flattened table. A flattened table shows up as runs of three or more spaces on at least three lines and on at least half of the non-empty lines. The decision is per page, so a scanned page bound into a text PDF takes the OCR route on its own.
The two components do different jobs. Apple Vision reads the text that's there, in about 0.5 s per page. qwen3-vl interprets the image: it transcribes the text and describes the structure and content of charts and diagrams, in 14 to 29 s per image. On the same input:
Route
Time
Input tokens
Quality
Image straight to the vision model
18.6 s
1,078 visual
excellent
OCR, then text to the model
13.6 s, 0.5 s of it OCR
178 text
equivalent
The result is just as good, but the OCR route takes 13.6 s instead of 18.6 s, 27% less. The difference is in the tokens. An image passed to the model becomes 1,078 visual tokens, while the text OCR pulls out of it takes 178, a sixth. On this chip, a call's duration depends mostly on how many tokens the model has to read in (the prefill), so a sixth of the input tokens means a shorter call, and the saving repeats for every scanned page in the document.
Figure 7. OCR before vision.
Embedded images
No text extractor reads embedded images. markitdown leaves an empty placeholder in a .docx, and a .pdf leaves nothing at all. Diagrams and screenshots pasted into documents were invisible to retrieval. Now every embedded image is pulled out in document order (.docx, .pptx, .xlsx, .epub, .odt, .rtf, .webarchive, .pdf, and .doc through olefile, since no maintained library extracts its images). It's skipped if its shorter side is under 64 pixels, identified by SHA-256 so a recurring image is read only once, and put through the same OCR → vision escalation. The vision model can be called at most 50 times per document, embedded images and scanned pages counted together. The cap doesn't drop images, because OCR reads all of them. Past the fiftieth escalation, an image where OCR finds no text keeps the OCR result and is marked as unread. The cap exists because the worker serves one queue item at a time, and a file full of text-free images would block it; the value can be changed from Mission Control. The result goes into the text inside a block that says where it came from (read by OCR, described by a model, not read), so generated text never gets mixed up with the original.
On the heaviest document in the corpus, 7.7 MB with 67 images, Apple Vision reads them all in 7.4 s. Only three go up to the vision model, and the whole document takes about a minute.
Tables, dedup, chunking, language
Office formats are converted to markdown before splitting, so tables stay tables. For legacy Word files I compared three conversion routes, and only the one that goes through HTML keeps the table structure. The text is then normalised and deduplicated twice: on the SHA-256 of the bytes, which catches an identical file, and on the SHA-256 of the normalised text, which catches the same document re-exported with cosmetic changes.
Chunking follows pages: at most 512 tokens, 50 of overlap, never across a page boundary, so every chunk keeps its page for citation. The splitter walks down a ladder of separators (blank line, line, sentence, semicolon, comma, word) and cuts hard only where the text offers no seam. The 512-token cap has a consequence for retrieval, which section 7.5 measures.
Language is detected per chunk by py3langid, with no model involved (0.08 ms per call), and it decides which stemmer the chunk is indexed under for full-text search. Per-chunk granularity is needed because the documents really are mixed. An Italian report that quotes English tickets contains whole chunks in English.
Indexing and analysis: two separate phases
The most important structural decision in the pipeline was forced by timing. Each document produces two things of a different kind: chunks and embeddings, which make it retrievable, and a description, categories and a summary, which present it when a search offers it up. Both are produced once, at upload. What a search costs is covered in section 6.
143,000-character document
Indexing
Analysis
Produces
chunks, embeddings
summary, description, categories
Model calls
87 embeddings (27.8 s) plus 3.5 s of extraction
11 chat model calls, ~44 s each
Total time
31 s
487 s
Blocks the document from being available
yes
no
So the two phases are separate queue items, and the queue never puts a document waiting to be indexed behind one waiting to be analysed. A document is retrievable as soon as its chunks are written, before it has a description. Each phase has its own status, its own retry and a fingerprint of the configuration it ran with (chunking parameters, embedding model and full-text mapping for the first; model, context and generation caps for the second). When a parameter changes, rows with a different fingerprint are marked stale and a search says so, instead of quietly serving a corpus indexed two different ways.
Figure 8. The two phases over time.
Hierarchical summary
Every document gets a summary, even a short document, and the description and categories are written from the summary, not from the raw text. This is where the constraint from section 4 bites. A context of 8,192 tokens holds about 16,000 characters, 11% of a 143,000-character document. Reading only what fit in the window meant describing the first few pages, and a document that opened with a title page got classified by its title page.
So the summary is a multi-level map-reduce. The text is split into groups of 14,784 characters, that is (8,192 − 800 tokens reserved for the answer) × 2 characters per token. Two characters per token is the lowest ratio you see, on dense text in non-Latin scripts, and it guarantees that no group overflows the window in any language. Each group is summarised, then the summaries are summarised. The prompt sets the length, 200 words per group and 400 for the merge, with caps of 400 and 800 tokens as a safety margin. Without an explicit length the model wrote right up to the cap, and a three-group document needed six calls over three levels instead of four.
Four constraints protect the output. Text is cut at the last complete sentence, and a full stop between two digits doesn't count as one, otherwise "12.5 Gbps" would become "12.". A summary with a missing group isn't saved, because it would pass for a summary of the whole document. The number of pieces has to shrink at every level, and if it doesn't, the summaries are forced into half as many groups. The document's language is decided by the detector on the original text and never by the summary, because an 8B model tends to write in English and the stemmer depends on that language.
Document
Characters
Calls
Time
Technical document
34,023
4
137 s
Architecture document
142,895
11
487 s
Figure 9. Hierarchical summary.
On a different machine. With a 32,000-token context a 34,000-character document would fit in one call, and the hierarchical summary would only be needed for very long documents.
A vector index on the SSD
For a corpus this size, the obvious choice would be an HNSW index held in RAM. I use StreamingDiskANN through pgvectorscale instead, a graph index that lives on disk with a bounded working set, because the RAM is reserved for the models. The accuracy cost is measured: on this corpus the index returns 19 of the 20 exact nearest neighbours on average.
6. Retrieval: from query to passages
The corpus never lands in the context automatically. When Claude calls search_knowledge, the query goes through this pipeline:
- embed the query with the query-side instruction Qwen3-Embedding expects (the model card puts the loss without it at 1 to 5%);
- two retrieval arms, 20 candidates each: vector search and full-text search, the latter using the stemmer of each chunk's language;
- merge them with Reciprocal Rank Fusion, which combines the two rankings by position, into a pool of 20 candidates;
- rerank with the cross-encoder, which scores each query-chunk pair along with the document's name and description;
- apply an absolute relevance threshold;
- take the top five in the cross-encoder's order, then group by document, with added notes ahead of the original text;
- widen each selected chunk by one chunk on each side, cut from the original text;
- return one block per document (name, description, notes, passages) inside the untrusted-content wrapper.
Figure 10. The read path.
Why two stages
The two-stage design comes down to cost. The first stage uses a bi-encoder: query and chunks are encoded separately, chunk vectors are computed once at indexing time, and the comparison is a cosine distance served by an ANN index in a few milliseconds. The catch is that query and document never meet before they're compared. The second stage uses a cross-encoder, which reads query and chunk together in one forward pass and produces a relevance judgement. It's much more accurate, but has to run again for every pair, about 0.3 s each on this GPU. That's why it only sees the 20 candidates the first stage picked, and why it alone accounts for over 90% of a search's latency.
Figure 11. Bi-encoder and cross-encoder.
Step
Latency
Query embedding
120-390 ms
Hybrid SQL: vector, full-text, fusion
47-58 ms
Chunk hydration
10-15 ms
Rerank of 20 candidates, warm model
5.8 s
(median)
Expansion of the selected documents
9-22 ms
The steps other than the rerank were measured on 2 September 2026, the rerank on 15 September; added up, they stay under half a second. The cold load of Qwen3-Reranker-0.6B and how its 5.8 s break down haven't been measured yet.
On a different machine. With a stronger GPU, reranking 20 pairs would drop under a second, and the candidate pool, held at 20 today mostly because of latency, could grow.
7. Five defects found by measuring
The pipeline in section 6 is the one running today. A few days ago a different one was running, and nothing on the outside suggested anything was wrong. It answered, cited relevant documents, and passed every functional check. The five defects below only showed up when I stopped looking at the answers and started measuring each stage of the pipeline on its own.
The instrument is simple. There are 40 questions, each labelled with the document that should answer it; 20 of them are written in a different language from their document. Next to them are the 79 real queries Claude had sent to the search tool over the previous weeks. For every question the script records where the expected document ends up: among the first-stage candidates, in the cross-encoder's ranking, past the relevance threshold, in the final results. Two metrics sum it up: hit@3, the share of questions with the expected document in the top three, and MRR, the mean of the reciprocal of its position. The per-stage breakdown is what tells you where a question gets lost, though, and it's what led to each of the fixes.
The measurements come from two consecutive test corpora: 82 documents and 1,810 chunks on 16 September 2026, and 84 documents and 1,829 chunks on 18 September, after two documents were added that don't appear among the questions that moved. Every before/after comparison was taken on the same day on the same corpus, and the corpus is stated.
7.1 A multilingual reranker that is not cross-lingual
The first defect hit cross-lingual questions. An Italian question about a document written in English brought the right document back low in the list, or not at all, even when the vector arm had put it among the candidates. The embedding was doing its job; the problem was further down.
The cross-encoder at the time, bge-reranker-v2-m3, is described as "multilingual" on its model card, and it is. It scores pairs well in many languages, as long as query and text are in the same one. It isn't cross-lingual. The benchmarks it publishes (BEIR in English, CMTEB in Chinese, MIRACL, which is multilingual but monolingual within each pair) never measure a query in one language against text in another, and the authors confirmed in the model's discussion page that no cross-lingual training data was used.
The first response was a workaround: translate the query into every indexed language and have the cross-encoder score each chunk against the version in its own language. It worked, but it added a chat model call to every search, 1.44 s warm and 11.02 s if the model had been unloaded. It also needed a quality check on the translation, because a translation that collapsed to a single word had to be caught, and a degraded path for when that check failed.
The proper fix was to change the model while staying within what the RAM budget allows. I evaluated two cross-encoders of about the same size as the one in use. jina-reranker-v2-base-multilingual crossed the language boundary but lost the ability to discriminate (a plain "grazie", Italian for "thanks", scored 0.33 against 0.49 for a real answer). It also has a non-commercial licence (CC-BY-NC-4.0) and wouldn't load with the transformers version in use. Qwen3-Reranker-0.6B really does compare across languages: an Italian query against the English document that answers it scores 0.9840, the same query in English 0.9954, and "grazie" drops to 0.18. It takes 1.07 GB resident, the same as the previous model, and costs about a second more on 20 pairs, which is less than the translation it removes.
The swap was more than changing a constant. Qwen3-Reranker-0.6B is a causal language model rather than a classifier: you get the score by formatting the pair with the model card's template and reading the probability of the token "yes" against "no" at the last position. The translation was removed along with all its code, and the degraded path went with it.
Figure 12. Same question, two languages.
This episode left me with a rule I now apply to any model meant for a mixed-language corpus: measure the cross-lingual property on a real pair, because the model card usually only claims the multilingual one.
7.2 A full-text arm that returned nothing
The second defect turned up by accident, while I was re-running the measurement script after an unrelated change. The script records the two first-stage arms separately, and one of its summary lines read questions_with_empty_lexical_arm: 40. On forty questions out of forty, full-text search returned not a single row. The "hybrid" search was in practice a vector search with an empty second arm.
My first guess was a bug in the script, and I ruled it out quickly. The script rebuilds the RRF fusion from the two arms it recorded and checks that it matches what the production function returns. If production had a non-empty full-text arm, the two lists would have diverged, and the check passed on all forty questions. The emptiness was real. Next I ruled out a corrupt index (no chunk produced an empty tsvector) and a side effect of removing the translation. That left the parser.
websearch_to_tsquery, the PostgreSQL function that turns the query text into a search expression, joins unquoted words with AND. A 9- to 14-word question therefore requires every one of its lexemes to appear in the same 512-token chunk, and no chunk has them all. Taken one at a time, the terms of the longest question each appeared in hundreds of chunks. The real queries followed the same curve by length: four of five one-word queries got results, none of the eleven with eight words or more did, 24 of 79 overall.
The answer was a disjunction, but OR alone wasn't enough. A first variant with the terms joined by OR and PostgreSQL's standard ranking (ts_rank_cd) filled the arm for every question and put the expected document in its top 20 for 21 of 40, yet recovered none of the lost questions: the slots went to chunks with the most common words in the corpus. Before picking a ranking I read how others do it. RAGFlow combines OR with a minimum-match threshold and a per-term weight read from a frequency file its repository doesn't ship. LlamaIndex uses OR ranked by ts_rank. Haystack keeps a conjunction. The variant I adopted takes the half of RAGFlow's approach that needs no weights: it ranks each chunk by how many distinct query lexemes it contains. Because that count means the same thing under every language configuration, it also allows the per-language versions to be merged into a single ranking. Before, they were interleaved by position, and each got half of the 20 slots whatever language the query was in.
Measured on 16 September, 82 documents:
Full-text arm
Expected document in its top 20
Real queries with results, of 79
Conjunction (before)
0/40
24
Disjunction,
ts_rank_cd
21/40
78
Disjunction, ranked by lexemes matched, versions merged
27/40
78
The end-to-end result taught me something. On the 40 questions hit@3 stayed at 34/40 before and after, but not because nothing changed: one question recovered and one lost cancelled out. The lost one had its document in twelfth place in the vector arm. With the full-text arm empty, the pool of 20 candidates came entirely from the vector arm and twelfth place was inside it. With the full-text arm working, 9 of the 20 slots went to chunks only it had found, and twelfth place fell out.
The fix went in for what the 40 questions can't see: 78 of 79 real queries now get lexical evidence, against 24 before. Those are the short queries that name a product, a file or an identifier, which is exactly what a lexical arm is for.
7.3 A threshold that depended on the corpus
Without a relevance threshold, search always returned five chunks whatever you asked it, even a greeting. The first threshold, added on 3 September on a three-document corpus and marked provisional in the commit itself, was relative to the top score: max(0.05, 0.30 × best). The idea was that a chunk should be worth at least 30% of the best one in the same search.
The flaw showed up when a completely unrelated document was ingested. Its arrival raised one search's top score from 0.175 to 0.231, and the threshold with it, from 0.0526 to 0.0693. A correct answer whose score of 0.0526 hadn't moved fell under the line and disappeared from the results. With a relative threshold, what a query returns depends on what the rest of the corpus contains, and a growing corpus shifts the results even for questions that have nothing to do with the new material.
Before changing the formula I checked how reference systems handle this, across fifteen frameworks, vector databases and reranking APIs (LangChain, LlamaIndex, Haystack, RAGFlow, Weaviate, Vespa, Elasticsearch, Qdrant, Cohere and others). None of them uses a threshold relative to the best score. Common practice is to return the top N, with an absolute threshold available but off by default. An absolute threshold makes sense here because the cross-encoder scores each pair in isolation and returns a judgement in [0, 1] that doesn't depend on the other candidates. The search threshold is now 0.005. Inside get_document the relative threshold stays, with its own value: there, all the scores belong to chunks of the same document, and corpus growth can't move them.
Measured on 16 September, 82 documents, before the fix in 7.4:
Threshold
hit@3
Real queries with no results, of 79
relative,
max(0.05, 0.30 × best)
33/40
13
absolute, 0.005
34/40
5
hit@3 holds at 34/40 for every absolute threshold between 0 and 0.05, so it was the relative part that cost a question. Even so, 0.005 doesn't cleanly separate noise from weak answers, and no value could. One correct query retrieved the right document in first place at 0.0009, while a genuinely off-topic query reached 0.0044. The two distributions overlap, and that's a property of the cross-encoder. At 0.005 the threshold removes all four noise queries in the sample and one of the four weak correct answers. A search can therefore return fewer than five results, or none, and when that happens it says so.
Figure 13. The overlap zone.
7.4 One document taking every slot
The fourth defect came out of what was meant to be a routine measurement. The numbers gathered over two weeks had been taken on different corpora and pipelines, so I re-ran them on the current corpus, which had grown to 84 documents in the meantime. Candidate recall was unchanged, 34 of 40 questions with the expected document among the candidates, but hit@3 had dropped from 34 to 32. The first stage was finding the same documents. Something later was losing them.
Both lost questions fell at the top-five selection, and neither list contained the two new documents, so the corpus wasn't the cause. One of the two asked how a sensor system classified the devices it detected. The cross-encoder had ranked the candidates sensibly, 0.9946, 0.9937, 0.9917, 0.9897, spread over three documents, and the expected document was the one at 0.9917. The final answer, though, held five chunks from a single document, the last at 0.8774, and the chunks at 0.9937 and 0.9917 from the other two documents were gone.
The cause was one line: rank_knowledge returned apply_addendum_priority(ranked)[:top_k]. The priority function exists to put notes added to a document ahead of its original text, and it does that by gathering all of a document's chunks at the position of its best chunk. Applied to the whole ranking before the cut to five, it gave the document holding the top chunk every slot its candidates could fill, whatever their scores. The cross-encoder decided the order, and one step later that decision was undone.
The fix swaps the two operations. First take the five best chunks by the cross-encoder, then group. Applied to a selection that's already been made, grouping reorders what the scores picked and can no longer add or remove anything.
Measured on 18 September, 84 documents:
Before
After
hit@3
32/40
33/40
MRR
0.775
0.783
Distinct documents per search
2.15
2.73
Searches returning a single document
10/40
5/40
Searches returning a single document were halved. The other lost question stayed lost. Its expected document is seventh at 0.9805, under six chunks from four other documents, and no ordering of the two operations brings it into the top five. That one is the cross-encoder's call, and section 11 lists it as an open problem.
7.5 Passages cut off at the chunk boundary
The last defect didn't show up in the metrics, and why it didn't is as interesting as the defect. A chunk ends wherever the 512-token cut falls, and the sentence it falls in carries on in the next chunk, which the search doesn't return. The corpus has chunks ending with lines like "according to the following operating procedure:" while the procedure sits in the next chunk. A model handed the promise of a list without the list has no way of knowing something is missing, and may answer as though the passage were complete.
hit@3, candidate recall and MRR measure which documents and chunks retrieval finds. A truncated chunk was found correctly, so none of the three metrics could see the problem or judge a fix for it. I measured the two things the decision actually rested on instead: how often it happens and what fixing it costs. On 18 September, on the 84-document corpus, 53 of the 194 chunks selected by the 40 questions ended cut off.
The mechanism to fix it already existed. get_document widened each match by one chunk on either side, cutting from the document's original text; the search simply never called it. The fix wires that mechanism into search as well. Each selected chunk is extended to its neighbours, ranges that touch are merged, and the text is cut from the original. Unlike gluing chunks together, that doesn't repeat the overlap and doesn't carry the markers the chunker inserted.
Chunks added per side
Characters per search
Truncations whose continuation reaches Claude
0
8,526
0 of 53
1
18,560
53 of 53
2
26,988
53 of 53
One chunk per side recovers every continuation and doubles the characters returned. Two cost another 45% and recover nothing more. Because even a widened passage ends on a chunk boundary, the response includes a note, written by the system and placed outside the untrusted-content wrapper, that says the passages are excerpts and marks the jumps between them.
Figure 15. A chunk and its neighbourhood.
The reference baseline
After the five fixes I re-ran the full measurement on 18 September 2026, on 84 documents and 1,829 chunks, with the current pipeline: Qwen3-Reranker-0.6B, untranslated query, disjunctive full-text, absolute threshold 0.005, pool of 20, five results. Two runs of the script on the same corpus reproduce every figure.
Metric
Value
hit@3
33/40
hit@3, question and document in different languages
16/20
MRR
0.783
Candidate recall (expected document among the 20 candidates)
34/40
Distinct documents per search
2.73
Real queries with no results, of 79
4
This table is the only reference for the current state. The comparisons in 7.2 and 7.3 are on the 82-document corpus and come before the fix in 7.4. They measure the effect of each individual fix, and their 34/40 can't be compared with the 33/40 above.
Figure 16. Where the seven missed questions go.
8. Opening a document
When the answer isn't in the passages, Claude can open a document with get_document and must say which mode it wants. summary returns the summary written during analysis, for questions about what a document is. answer runs the same pipeline as a search, restricted to that document: up to 60 candidates from the vector and full-text arms, rerank, threshold and widening. The mode is an enum in the tool's schema, which is what Anthropic's guide to writing tools for agents recommends.
A document is never returned whole, however long it is. Until 17 September, a document under 40,000 characters did come back in full: on the 82-document corpus, 64 of them went through no relevance selection at all, and their text reached Anthropic's API without anyone judging it relevant.
The 60-candidate limit is a concession to the hardware. Reranking every chunk of the largest document in the corpus, 223 chunks, takes 73 s with a warm model. With 60 candidates it takes 19.7. What that loses, measured on 17 September against the full ranking:
Candidates
Documents up to 24 chunks: same answer
Larger documents: share of the full ranking's chunks kept, mean
20
32/32
0.74
60
32/32
0.92
120
32/32
0.99
The chunk holding the answer, labelled by hand, is there in all eight large cases at every level. Eight questions over four documents is still a small sample.
The model answers from what it already has
The most stubborn behaviour I ran into has nothing to do with Emily's code, and it's the one that taught me the most about measuring an agent.
It appeared right after get_document got its two modes. With a document's summary already in the conversation, I asked whether the document mentioned a certain topic, and Claude said no without calling any tool. The answer happened to be right, but only by luck. A summary compresses a document, and a topic missing from the summary can still be in the text. An explicit search inside the document, run straight afterwards, confirmed the topic wasn't there, but the model hadn't checked. Anthropic's documentation describes this as the default: the model calls a tool when the request matches what the tool does and the answer isn't already in context. As far as the model was concerned, the summary already answered the question.
The first attempt was the obvious one, a sentence in the tool descriptions explaining that a summary doesn't cover everything in a document. Tried live, it changed nothing. The second added an instruction at the top of the search context block and a system-written note after every summary returned. Measured on a harness that replayed recorded sessions, it raised re-opens from 0 to 6 sessions out of 8 on the question "does it also mention X?". The result was statistically solid, and the change shipped.
Live, straight after the merge, the same question after a summary produced no call. The harness had measured the model with a bare system prompt, without the workspace settings, and with tools loaded differently from the way Emily loads them. It had measured a context that doesn't exist in production. So I rewrote the measurement to start from real sessions. Each run forks a recorded conversation at the point where the summary or passages are already in context, and sends the next question the way the worker would, with the persona, the workspace settings and the same model. Measured on 18 September, 8 runs per cell, on claude-sonnet-4-6, with the instruction and the note in place:
Part of the document already in context
Question
Document re-opened
The summary
Does it also mention carbonara?
1/8
The summary
Which AWS regions does it use?
3/8
Nine passages from an earlier open
Does it also mention carbonara?
0/8
The passages from a search
Does it also mention carbonara?
0/8
In the same harness, an explicit rule in the workspace CLAUDE.md took all four rows to 8 out of 8, and the rule was adopted. That result didn't hold up either. Live, the carbonara question after a summary produced no calls. Going back through the transcript, I found the harness attached the modified CLAUDE.md right after the question, while a real session loads it on the first turn, dozens of messages earlier. The rule stays in place, but its effect is unmeasured, and the table above is the only figure I consider valid.
Context is a second budget
Every character a search returns uses up Claude's context and is the part of the corpus that leaves the machine. With widening, a search returns about 18,500 characters; three whole documents cost 30,309 on a real turn. Passages answer most questions on the first call and leave opening a document for the cases where the answer runs past their edges.
9. The corpus as untrusted content
Every document is external content, and so a vector for indirect prompt injection. Three rules apply without exception.
- All extracted content that reaches a model is wrapped in an <external_content> tag, with the rule for reading it written on the line above. The rule travels with the content instead of living only in the system prompt, because a spawned subagent never receives the system prompt but does receive the content.
- Every tool with an external effect goes through an approval manager, and every surface that shows an approval authenticates the person before passing it on. The API authenticates the client; it knows nothing about the human behind it.
- Claude can only open documents the conversation has already shown it. The set of allowed IDs is kept by the code, and an ID that was never shown gets refused regardless of what the text suggests.
Only what the two tools return goes to Anthropic's API: passages, descriptions, summaries. The original files, the index and every ingestion call stay on the machine.
10. Techniques from LlamaIndex and RAGFlow dropped after measurement
After the fixes in section 7, six questions were still lost at the first stage: documents that neither the vector arm nor the full-text arm brought into the 20 candidates. Going through them one by one, the question's terms were in the expected document, but spread across different chunks. No single chunk "looked like" the question, even though the document as a whole answered it. Three of the six were Italian questions on English documents that shared only a product name and one stem with their document.
That's a limit of chunk-level retrieval, and the maintained open-source systems have techniques designed for exactly this: enrich the chunk with vocabulary it doesn't contain, or index the document in a form other than its text. Before writing any mechanism of my own I read how LlamaIndex and RAGFlow implement these in their source, and measured the four most promising techniques on the 82-document, 1,810-chunk corpus, with the pipeline as it was before the fix in 7.4.
The two enrichment techniques, keywords and questions generated per chunk, need one chat model call per chunk. Over the whole corpus that's 12.6 hours of generation, and it made no sense to spend that before knowing whether it helped. So I tried them on a pilot: the 45 chunks of the five documents lost at the first stage, with everything else left as it was. The setup favours the technique by construction, because the enriched chunks compete against chunks nobody enriched, so the numbers should be read as an upper bound.
The pilot also turned up something on the side. Keywords generated in the chunk's own language repeated vocabulary the chunk already had, and moved nothing. The only variant that made sense for a mixed corpus was generating them in the other language too. Asked for Italian keywords on 45 English chunks, though, the model returned English ones in 40 cases, because most of what it extracts is product and company names. It took a second translation pass.
None of the four techniques went into production. Negative results with their numbers are the least published part of this kind of work, and the most useful to anyone about to try the same thing.
Technique
Source
Measured outcome
Table indexed as a generated summary plus column schema
LlamaIndex,
base_element.py
On the target query the summary scores 0.2065, against 0.3960 for the original chunk and 0.4902 for the twentieth candidate, so it moves the answer further away. The model writes an abstraction ("enterprise-grade IT tools") that drops exactly the detail being searched for
Table rows rewritten as
column: value
RAGFlow,
rag/app/table.py
Fully rewritten, the answering chunk goes from 0.3960 to 0.3963, against 0.4901 for the twentieth candidate. None of the 40 questions moves
Keywords generated per chunk, weighted in the full-text arm
RAGFlow,
task_executor.py
No effect in the chunk's own language; in both languages, one more question out of 40. Generation: 12.6 hours over 1,810 chunks
Questions generated per chunk, included in the embedding
LlamaIndex,
metadata_extractors.py
Candidate recall from 34/40 to 36/40, hit@3 unchanged (one question gained, one lost). Same generation cost
The table case deserves a few more lines. The column: value rewrite came from a question about a service vendor, lost at the first stage. The chunk that answers it is a twenty-row table, and the vector arm put it 119th out of 1,810. Rewriting every row moves it to 115th. Even isolating the one relevant row, in RAGFlow's format, only raises the cosine to 0.4398, which would place it 44th. The gain exists only at the granularity of a single row, a unit the index doesn't contain, and indexing one row per chunk would take the corpus from 1,810 to 6,513 chunks. The table summary failed the other way round: the model wrote an abstract, accurate description that for that very reason lacked the term the question was looking for.
On this hardware, any technique that needs a model call per chunk costs hours of ingestion, and 12.6 hours for an effect inside the noise of 40 questions is hard to justify. One technique shared by Haystack and RAGFlow did make it in: passing the document's name and description to the cross-encoder along with the chunk.
Figure 17. Cost and benefit.
11. What the measurements cover, and what they don't
The instrument
Forty questions give a resolution of one question in 40, so a change of one is inside the noise. The questions were generated by a model from the documents that answer them, and they share wording with the right passage in a way a question typed by a person wouldn't. The corpus is made of test documents; the real one will hold hundreds, split roughly half Italian and half English.
Open problems
- Six of 40 questions lose the expected document at the first stage: neither the vector nor the full-text arm puts it among the 20 candidates. Three are Italian queries on English documents that share only a product name and one stem with the document.
- One question loses its document at the top-five selection: the cross-encoder puts it seventh at 0.9805, under six chunks from four other documents.
- The model tends to answer from part of a document instead of re-opening it (section 8).
- The new cross-encoder's cold load and the breakdown of its 5.8 s haven't been measured.
What would change on different hardware
The architecture would stay: two ingestion phases, OCR before vision, two-stage hybrid retrieval, an absolute threshold, widened passages. The models and parameters would change: a chat model with more parameters in place of qwen3-vl:8b-instruct, with a long enough context to summarise a document in one call; larger embedder and cross-encoder variants; a resident cross-encoder; a larger candidate pool. Each of these is set today by the 16 GB and the base chip's GPU.
Most of these parameters aren't hardcoded. The second brain exposes 18 configurable values, 14 of them in Mission Control: pool size, number of results, relevance threshold, passage widening, chunk size and overlap, the cap on vision model calls, the tool prompts. The default lives in the code and goes through code review, and a row in the database overrides it while the row exists. When a change makes the index stale, a new chunk size for example, the panel says so and says what needs re-indexing. The models are the exception: changing them means changing the Ollama configuration, and every measurement in this article would have to be taken again.
Appendix A. Stack
PostgreSQL 16 (timescale/timescaledb-ha) with pgvector and pgvectorscale. Ollama for qwen3-vl:8b-instruct and qwen3-embedding:0.6b. Qwen3-Reranker-0.6B through sentence-transformers, on the Apple GPU in fp16. Apple Vision for OCR. Python 3.11. The agent is Claude Code, started on the Mac Mini by the Claude Agent SDK; Claude's inference goes through Anthropic's API, and every other model runs locally.
Package
Role
PyMuPDF
PDF text layer, scanned page detection, PNG rendering, embedded images per page
markitdown[docx,xlsx,xls,pptx]
Office formats to markdown, with tables preserved
python-pptx
image order per slide
defusedxml
parsing document XML without entity expansion, which could turn a 1 KB file into gigabytes of RAM in the worker
olefile
opening
.doc
files (Word 97-2003) to extract their images
charset-normalizer
encoding detection for text files
py3langid
language per document and per chunk, no model: 221 ms to load, 0.08 ms per call
sentence-transformers
the cross-encoder, loaded lazily in the API process
pyobjc-framework-Vision
,
pyobjc-framework-Quartz
OCR with Apple Vision, macOS only, kept out of the shared library so it stays OS-independent
claude-agent-sdk
starts Claude Code sessions and exposes
search_knowledge
and
get_document
as MCP tools
asyncpg
,
httpx
,
structlog
database, Ollama calls, logging
Appendix B. Implementation details
Chunking and tables. On a markdown table almost every cut lands at the end of a row, so a table whose rows are longer than the overlap window (200 characters) produces consecutive chunks with no overlap at all. The corpus has rows up to 1,966 characters long, so the 50-token overlap is a ceiling and can drop to zero.
Cross-encoder. It runs off the event loop through asyncio.to_thread, and it's a lazy singleton behind a lock, so two concurrent first calls load it once. A background sweep releases it after 30 minutes idle. If it fails to load, search reports itself unavailable and raises a critical alert instead of falling back to an unscored ordering. With a reranker that returns positions rather than judgements, the relevance threshold is switched off.
Realignment. Chunks are hydrated by a set of IDs, so the database returns them in arbitrary order, and the fusion order is reapplied before the rerank.
Configuration fingerprints. The indexing phase records chunk size and overlap, the embedding model, a hash of the full-text mapping, the language detector's confidence threshold and the indexed languages. The analysis phase records the model, context, characters-per-token ratio, generation caps, minimum description length and a hash of the categories. Each has a rules version that is bumped by hand.
Appendix C. How the measurements were taken
Every measurement was taken on the 16 GB Mac Mini M4. Retrieval measurements ran with Emily's services stopped and Ollama up, read-only against the database. The reference baseline in section 7 is from 18 September 2026, on 84 documents and 1,829 chunks, with 40 questions (20 cross-lingual) and 79 real queries. The comparisons in 7.2, 7.3 and section 10 are from 16 September, on 82 documents and 1,810 chunks. The Qwen3-Reranker-0.6B figures in 7.1 are from 15 September, on 14 documents. RAM was measured with footprint, not ollama ps. Figures taken from model cards or third-party benchmarks are cited as such, and any number with an old date should be measured again before it's reused.
