https://github.com/randclem/sitt-rag/

This is a companion project to Something in the trees. The development of a Retrieval-Augmented Generation system for LLMs to provide information related to cryptids and cryptid lore.

Building a RAG isn't just about stuffing documents into a vector store and hoping the LLM figures it out. There are a lot of small decisions along the way, and most of them have a real effect on the quality of what comes back out.

In this article I'll walk through how the project came together, starting with using the /wayfinder skill to turn a vague idea into an actual spec and a set of tracker items. From there we'll get into the design itself: the component layout, why I landed on Voyage AI for embeddings, and what the research turned up on Wikipedia's licensing and attribution requirements. Then we'll look at the MCP server that sits in front of the data, the chunking strategy and why chunking matters more than you'd expect, and how retrieval, updates, and error handling were handled. I'll close out with the evals and a list of the things I would do differently if this were headed for production.

Wayfinder

Using /wayfinder skill from Matt Pocock's set of AI assistant skills https://github.com/mattpocock/skills. This skill takes you on a journey to flesh out ideas that are very early in their design and do not have a firm path yet. Similar to Matt's /grill-me skill, the AI assistant acts as an active planning tool. Through a series of design questions about the idea, you develop a more thought out version of the idea, which results in several to-do items that the skill will publish to your task tracker of choice, defaulting to GitHub Issues.

Some of these tracker items were a set of research tasks to better understand some licensing and cost questions. These research tasks were Agent-ready and able to run as sub-agents within the project. Others created tracker items to perform /grill-me sessions based on architecture that still required deeper questions to design. This included topics like:

• Designing the MCP tool surface
• Designing the chunking strategy
• Designing the update script

We will discuss the outcome of several of these designs in this article.

If you are using GitHub issues, wayfinder will also add relevant tags to the tracker items and assignment to the issues. The tags included:

• ready-for-agent
• ready-for-human
• wayfinder:research
• wayfinder:grilling
• and others as it sees necessary

RAG Design

Using /wayfinder a specification is created once the initial set of design questions are settled. This spec is then stored in GitHub as an Issue.

https://github.com/randclem/sitt-rag/issues/1

The spec is designed to be a main map that lists our design decisions and becomes a launching pad for almost all of the resulting tasks in the project. Some basic
examples of the design decisions on this project.

• Language: Python
• Vector Store: ChromaDB, local client on disk
• Embeddings model: voyage-4 (not free but at a low enough usage, essentially free)
• Data source: Wikipedia's List of cryptids article, chunk the data by section with a one paragraph overlap
• Content Updates: Use a manual script invocation.

RAG Component Design

The following component map of the repository shows the design of the project.

Two independent entry points share the same storage and embedding code:
update.py runs offline to rebuild the corpus, while server.py runs
long-lived and only ever reads from it. Dashed arrows are configuration, not
data flow.

Module reference

Ingest

• update.py — Orchestrates a full diff-and-confirm ingest run: added /
  changed / removed / unchanged / failed buckets, then a [y/N] commit.
• wikipedia.py — Fetches the taxonomy and article HTML, with
  retry/backoff on transient (429/5xx) errors only.
• chunking.py — Splits article sections into token-budgeted chunks with
  one paragraph of overlap, via tiktoken.

Serving

• server.py — MCP server exposing four tools, all returning
  {"error": {...}} instead of raising.

Shared

• embeddings.py — Thin Voyage AI client wrapper — embed_documents for
  ingest, embed_query for search.
• store.py — ChromaDB access: chunks (embedded, searched) and
  articles (full text) collections.
• config.py — Env vars, data dir, chunk budget, and the Voyage model
  name — read once at import.

Stack

python 3.11+ · mcp · chromadb · voyageai · tiktoken · requests + bs4

data/ · chroma.sqlite3 + hnsw index

https://github.com/randclem/sitt-rag/blob/main/components.md

Embeddings

Voyage AI using voyage-4. Why did we choose this? This was one of the research tasks wayfinder handed off to a sub-agent, and it came back recommending voyage-4 as Voyage's balanced, general-purpose default. voyage-4 is part of a newer family (voyage-4, voyage-4-lite, voyage-4-large, voyage-context-4) that supersedes voyage-3.5/voyage-3-large, and the plain voyage-4 model won out over the lite/large variants on fit rather than price — at this project's scale, cost was basically a non-issue either way.

A few of the numbers that made the decision easy:

• Price: $0.06/1M tokens, but effectively free — Voyage's 200M-token shared free pool covers this project's small corpus (roughly 100K–500K tokens) hundreds of times over.
• Dimensions: 1024, Matryoshka-truncatable down to 256/512/2048 if we ever need to trade accuracy for storage.
• Context limit: 32,000 tokens per input, which is what ended up bounding chunk size in the chunking-strategy design.
• Rate limits: 2000 RPM / 8M TPM — far more than a Wikipedia-sized corpus and a low-traffic MCP server will ever push.

Content Licensing and Attribution

An important part of working with data from an external source is determining if you are legally allowed to use that data in your product. It's only fair to the content creators and ultimately will keep you out of legal hot water to first check if the content is usable.
One of the research subtasks that wayfinder handed off to a sub-agent was determining the licensing and attribution requirements for Wikipedia content. Wikipedia text is published under CC BY-SA 4.0, and the research came back with two findings that shaped how the store and server were built:

• Attribution is required. CC BY-SA 4.0's "Share" definition is broad enough to cover the excerpt text returned from an MCP tool call, but the bar is low — a link back to the source article satisfies it, no per-article author list needed.
• ShareAlike likely doesn't apply, as long as excerpts stay verbatim. Unmodified chunking, embedding, and JSON-serializing don't count as producing "Adapted Material" under the license. That would change the moment the pipeline has an LLM rewrite or summarize the text before storing it — something to watch if that ever gets added.

The recommendation was to attach a small source metadata block — title, canonical URL, license string — to every excerpt at ingest time, and to keep excerpts verbatim rather than paraphrased. That's what update.py does: every fetched article gets stamped with a Source(title, url, license=CC_BY_SA_LICENSE) before it's chunked and embedded, and that metadata rides along as ChromaDB chunk metadata. store.py reads it back out on every query, and both search_cryptid_lore and get_cryptid return a source: {title, url, license} block alongside the text — so any agent or LLM consuming the tool response has the attribution right there, without having to go dig for it.

Model Context Protocol (MCP) Server

The RAG was only one part of the system to be built. The other part was to provide an API surface for an LLM or Agent harness to interact with the RAG data. That was provided by the MCP (Model Context Protocol) server. The MCP server exposed four tools for an agent to work with.

• search_cryptid_lore
• get_cryptid
• list_categories
• list_cryptids

These four provided generally enough capability for any agent harness to access the data and answer queries about cryptids. The list_cryptids and list_categories tools are useful for the agent, not only to check what cryptids are in the RAG itself, but also to know when it must retrieve data from other sources.

Chunking Strategy

The documents were retrieved from Wikipedia through their REST API. Each document is chunked into sections based on section headers (structurally) such as H2, H3, etc. Each chunk has an overlap of about 1 paragraph to allow the LLM to better reference and join data when necessary.

Why is chunking strategy important?

There are several choices one can make on document chunking. Fixed-length, structural, or semantic chunking were considered. In this project we used structural chunking as a mid-way choice between the three and I'll explain the differences.

• Fixed-length chunking A very simple strategy that chunks documents on fixed length boundaries, in other words, a set number of words. While this is simple to implement, the drawbacks are quickly realized. Fixed-length chunking can cut a phrase up mid-sentence or mid-paragraph causing confusion for the LLM due to missing data. LLM's don't handle this well and can either simply fail in their reply, or worse, hallucinate (make things up) to complete the request.

• Structural chunking is a little more complex. It uses language structure as boundaries for chunks. In other words, sentence markers, paragraphs, headers, and chapters become markers to chunk the data into its sections. Structured documents like DOCX and HTML that use headings, sections, and paragraphs are great to use for this strategy. This ensures that relevant data for a topic is maintained in the same chunk and results in less errors by the LLM.

• Semantic chunking makes use of a second LLM that is capable of reasoning about the semantic meaning in the documents. This method can produce a much more semantically defined set of chunks that minimize additional errata from paragraphs or sections that are not relevant to the current topic. However, this comes at the cost of the computation power of a second LLM to perform inference on the documents. Despite this cost, this can be a useful strategy to use when documents are unstructured, or, where there is lots of irrelevant data mixed up in the topics.

What is the relevance of overlap?

Overlapping sections is a technique that helps in all three strategies to "stitch" together disparate chunks. Take for instance, a structural chunk that was cut at a paragraph boundary, and, the next paragraph has relevant semantic information for that topic. Without overlapping data, the two paragraphs would end up in separated chunks without any way to link the information together. When you overlap data in the chunks, the LLM can reason that the chunks were adjacent to one another and would fetch the relevant data. This enriches the LLM's understanding of the topic resulting in a more complete context that in turn results in better generation outcomes.

In this project we implemented a single paragraph overlap on oversized sections. Most sections however were kept together by merging consecutive paragraphs up to a 500 token limit.

Retrieval, Updates, and Error Handling

Many of design decisions centered around how to retrieve the data efficiently, store it, and handle errors and failures gracefully. Wikipedia is a moving target, so the corpus can't be a one-and-done build. Articles get edited, renamed, merged, and occasionally removed entirely, and the store has to be able to catch up without a full rebuild every time. The other half of the problem is that we're talking to someone else's API over someone else's network, and both of those will fail on you eventually. The goal here was to make an update run boring and repeatable, even when things go sideways.

Change detection

Changes are detected using SHA-256 hashes to determine easily if an article has been updated. This is a low computational requirement on modern hardware and was easy way to determine articles that require updating. Updates are then upserted. The nice part about hashing is that it saves us from the expensive step — if the hash matches what we already have, we skip fetching, chunking, and embedding that article entirely. On a corpus where most articles don't change week to week, that means a typical update run only touches a handful of documents. update.py sorts everything it finds into added, changed, removed, unchanged, and failed buckets and prints that summary before it commits anything, so you can see exactly what the run intends to do.

Transient network failures

Deletes have checks integrated to prevent inadvertent data loss. Retries for transient network failures provide resiliency in case of a shaky network connection. The retry logic is deliberately narrow — it backs off and retries on 429s and 5xx responses only, since those are the ones that tend to clear up on their own. A 404 or a malformed response doesn't get retried, because hammering the API isn't going to change the answer, and quietly retrying a real error just hides the problem. Articles that fail land in the failed bucket and are left alone rather than being treated as removed, which keeps a bad network day from wiping out good data.

Deletion handling

There are a couple of ways we could have handled the data source removing articles upon updates. A hard delete simply removes the data since it no longer exists. A soft delete retains some data but at the risk of maintaining stale data over time.

Due to the simplicity of this project I decided to perform a hard delete for documents that are no longer found in the source material on an update. However, in a production system, I may not have made the same choice. The risk with a hard delete is that it trusts the source completely. If the taxonomy fetch comes back short for some reason that has nothing to do with the content actually being gone, a hard delete will happily throw away everything that went missing. That's exactly why the delete path has checks on it and why the run asks for confirmation before it commits — the human gets one last look at the removed list before anything disappears.

Soft-delete would be a better approach in production to retain some historical information so data is not immediately lost. To aid in marking where data is stale, we would use an active/inactive marker to be able to filter stale data and add dates to show how out of date the content is. However on top of the active/inactive filters, this decision also has some additional work involved to enable robustness, such as needing an additional long-term clean-up design and figuring out how to find and merge the latest data (assuming it exists at all).

Dry-run testing

A dry-run option was added to test and stage updates. It walks the whole diff and reports what it would add, change, and remove, then stops short of writing anything to the store or spending any tokens on embeddings. This turned out to be the fastest way to sanity check a change to the chunking or fetch code, since you can see the shape of the result without touching the data you already have. It also makes the update script safe to just run and look at, which is the behavior you want when you come back to a project after a few weeks away.

Evals

Code is only as good as its tests. I think I heard that somewhere before. Because the project is small, evals were added as a standalone script — python -m sitt_rag.eval — mainly for checking the quality of the RAG data from Wikipedia. In a larger project, I would place the evals in a pipeline for automated evaluation, alongside the DevOps code quality, QA, and static analysis tests.

The evals in this project were developed as one of the few Human-in-the-Loop tasks. Almost all the other tasks on the project were Agent ready. The Agent prepared the mechanical half of the set — 62 generated "Tell me about X" queries, one per ingested cryptid, which establish a floor: can the store find a thing when you ask for it by name. The human task was the other half, ten hand-written thematic queries phrased the way a person actually asks. "Lake monsters of Scotland" never mentions the Loch Ness Monster, and knowing that Ogopogo, Manipogo, and Igopogo are the ones you want back from "monsters said to live in Canadian lakes" is in-depth domain knowledge an agent isn't going to supply on its own.

Scoring: Recall@5

Each query is checked to see if the appropriate cryptid's lore came back in the top-5 results. A query passes if any of its expected cryptids shows up in those five — not all of them. That's a deliberate choice for the thematic queries, which have several equally correct answers competing for the same handful of slots; demanding all of them would fail a result that was actually good. The eval calls the real search_cryptid_lore MCP tool rather than reaching into the store directly, so what it scores is what a client would actually receive.

PASS  [mechanical] Tell me about Mothman  -> Mothman
FAIL  [thematic] Scandinavian lake creature folklore
        expected any of: Selma, Storsjö monster
        top 5:
          1. Loch Ness Monster (0.71)
          ...

Recall@5: 72/72 — 100.0%

Passes stay on one line. Failures print the actual top five with their scores, which is the part you care about — it tells you whether you had a near miss or whether the corpus simply can't answer the question. The score is purely informational: there's no threshold and it never touches the exit code.

What Recall@5 doesn't measure

It's worth being clear about what a number like 72/72 is and isn't telling you.

• It's recall, not precision. Nothing scores the other four results that weren't the answer.
• Rank doesn't count. A hit sitting at position 5 scores exactly the same as one at position 1.
• It measures retrieval, not generation. Whether an LLM handed those chunks produces a faithful, grounded answer is a separate question this script doesn't ask.
• It can't fail a build. With no threshold, the eval is something you run and read. That's the right call here and the wrong one for the automated pipeline I mentioned above.

Production Improvements

As stated in several sections already, there are improvements I would make if this project were to operate in a production capacity.

1. Semantic chunking - using an LLM in the chunking step to provide semantically designed chunks. Again not necessary for this project, but could be an interesting add-on.
2. MCP on HTTP - adding streamable_http and transport_security to the MCP module to be able to host this over a network. Placing this behind a load-balancer and optimize it for full hosting.
3. MCP authorization - The MCP server can be set as a resource server in an OAUTH2 bearer token authorization setup. Additional components would need to be added like the authorization server or hosting provider (ex. Auth0) and authentication methods, token registration and validation.
4. Online database for hosting. The local ChromaDB option works fine for development and testing. In production we'd want a more robust solution that scales. Hosting your own on a cloud provider is possible. ChromaDB also has Chroma Cloud as a fully hosted option.
5. Additional data sources - Wikipedia was chosen because of its generally open licensing, easy access, and general renown. Additional data sources, given the correct licensing and attribution, could be used to enrich or add to the dataset.

Conclusion

And that's about it. This was a really fun project to develop filled with strange creatures we all love. I hope you enjoyed this. Until next time...