CDN_CAPTAIN
A retrieval-first Discord bot for a gaming community whose most important feature is knowing when to say nothing.
Every community Discord has the same nine questions asked forever. The obvious fix is to attach a language model to the chat and let it answer — which produces something worse than an unanswered question, because a bot that is confidently wrong erodes the trust that makes it useful at all. The design goal was therefore inverted: answering had to earn its way past several checks, and silence became the default.
What it runs on, and why
- Python
- Roughly 1,700 lines, of which the interesting parts are deterministic rather than probabilistic.
- SQLite
- Fact store and state. A single-file database is the right size for this problem and needs no server.
- Playwright
- Weekly crawl of the community site, capped at 60 pages and four concurrent.
- Docker
- Runs on my own TrueNAS host rather than a VPS — the box was already doing enough to absorb it.
How it fits together
Read this diagram as text
- Discord → CDN_Captain (message) — crosses to the public internet
- CDN_Captain → Retrieval gate
- Retrieval gate → Fact store
- Retrieval gate → Model API — crosses to the public internet
- Fact store → Citation + grounding
- Model API → Citation + grounding
- Citation + grounding → Answer
- Citation + grounding → Failure log
- 01
A message arrives and hits a scoring function written in plain Python before it reaches anything expensive. Keywords are extracted, stop words dropped, a small hand-tuned synonym map applied, and the result scored against the fact store. No match, no answer — and no API call.
- 02
That gate is the most valuable code in the project and it contains no machine learning at all. It also means cost scales with the number of questions the bot can actually answer rather than with how busy the channel is.
- 03
Separately, once a week Playwright crawls the community site — capped at 60 pages, four at a time — and hashes each page. Only pages whose content actually changed are sent for fact extraction, so the expensive step runs on the delta rather than the whole site.
- 04
Extraction is deliberately fine-grained. The prompt refuses summaries and produces individual facts, because small facts retrieve better than paragraphs and, more importantly, can be cited precisely.
- 05
When facts do match, the bot makes exactly one answer call with the retrieved facts in context. Then the part that actually makes it work: the citations it returns are verified in code against the facts that were really retrieved, and a second independent call acts as a grounding verifier. Asking a model to cite its sources is a prompt; checking those citations is an assertion.
- 06
Anything failing either check is suppressed and written to a ranked failure log rather than posted. Admins can also mark answers good or bad directly, which feeds the same file. That log is the most useful artifact the system produces.
Decisions, trade-offs, failures and measurements. Everything below is collapsed by default because most readers do not need it — and expanded, because if you are still here you probably do.
DECISION_LOGWhy, and what it cost
Why, and what it cost
Why gate retrieval locally instead of just calling the model?
Most messages in a community channel are not questions the bot should answer. Passing them all to a model costs money on every message and, worse, invites an answer to things there is no source for. A local scorer decides whether the system even *can* answer before it decides what to say.
A hand-tuned keyword and synonym map is less flexible than embeddings and will miss phrasings a semantic search would catch. That is the accepted cost: a missed question is recoverable, a confident fabrication is not.
Why verify citations in code rather than prompting for them?
A prompt asking for citations is a request. Nothing enforces it, and the failure mode is silent — a fabricated citation looks exactly like a real one. Checking the returned citations against the actually-retrieved facts converts a request into a guarantee, and the second grounding call catches claims that cite real sources but do not follow from them.
Two calls per answer instead of one, so verified answers cost roughly double. Given the retrieval gate means the bot answers rarely, doubling the cost of a rare event was an easy trade.
Why self-host it rather than deploy to a platform?
The infrastructure already existed, was already monitored, and had capacity to spare. Running it there costs nothing incremental and keeps the fact store and the failure log on storage I control.
It inherits every weakness of that host — no UPS, manual power sequencing, a single residential uplink. If the house loses power the bot goes with it, which is acceptable for a community helper and would not be for anything a client depended on commercially.
INCIDENT_LOGWhat has actually gone wrong
What has actually gone wrong
Symptom
Someone asked the bot whether vehicles drop from airdrops. It said yes, and then elaborated — confidently, in detail, and entirely incorrectly.
Impact
A community support bot gave a wrong answer to a real question, in public, in the authoritative voice it uses for correct ones. No outage; the damage is to trust, which is the only thing this bot actually has.
Investigation
- 01Traced the answer back through the pipeline to find which retrieved fact had produced it. There was not one.
- 02Confirmed the model had answered from its own priors rather than from anything the retrieval layer supplied — the failure was not bad source data, it was an answer generated without sources at all.
- 03Recognised the deeper problem: asking a model to cite its sources is a prompt, and prompts are requests rather than guarantees. Nothing in the system was checking that a citation corresponded to a real retrieved fact.
Root cause
The bot was trusted to be honest about its own grounding. It was permitted to answer where retrieval had returned nothing useful, and its citations were accepted as given rather than checked against the facts actually retrieved.
Fix
Citations are now verified in code against the retrieved set rather than trusted, and a second independent call acts as a grounding verifier before anything is sent. Answers that fail either check are suppressed rather than posted.
Prevention
The incident became a permanent golden question in the test suite. Every subsequent wrong answer joins it, so the regression suite is assembled entirely from the bot’s own past mistakes. Suppressed answers are written to a ranked failure log, which is the most useful artifact the system produces.
Lesson
The valuable engineering here was not the model call. It was deciding what the system does when it does not know — and then enforcing that in deterministic code rather than asking the model nicely. A bot that is confidently wrong is worse than one that stays quiet, so silence became the default and answering became the thing that has to earn its way past two checks.
TRUST_BOUNDARIESWhat is defended, and what is not
What is defended, and what is not
Model exposure
Untrusted user text only reaches a model after passing the local retrieval gate, and answers are checked against retrieved facts before being posted.
GapThis is a grounding control, not a prompt-injection defence. A crafted message that scores well against the fact store still reaches the model.
Output
Nothing is posted that fails citation verification or the grounding check. The failure path is silence plus a log entry, not a best-effort answer.
Hosting
Runs in a container on infrastructure I operate; no inbound ports, outbound connections only.
MEASUREMENTSNumbers, and where they came from
Numbers, and where they came from
Configured limits on the Playwright crawler.
Per-page content hashing between crawls — unchanged pages skip fact extraction entirely.
Test suite assembled from the bot’s own past wrong answers.
Approximate line count. The majority is deterministic logic rather than model interaction.
POST_MORTEMWhat I would do differently
What I would do differently
- ▹
The retrieval gate should have existed before the first model call, not after the first embarrassing answer. Building the cheap deterministic layer first would have prevented the incident that eventually forced it.
- ▹
Fine-grained fact extraction was the right instinct and I under-committed to it early. Paragraph-level chunks retrieved worse and could not be cited precisely, which made verification harder than it needed to be.
- ▹
The failure log turned out to be more valuable than the answer path. If I started again I would build the suppression log and the feedback loop on day one and treat the answering as the feature that has to justify itself against it.