Portfolio
Project

Drag: a RAG assistant that learns from what it can't answer

An information-security assistant I designed and shipped on my own at Botpress, and the self-healing loop that turns every “I don't know” into work for a human expert, then into the next person's answer.

Software engineering internship at Botpress, Montréal · March – August 2026

TL;DR: Drag answers Botpress employees’ security and compliance questions in Slack, and fills in the security questionnaires that big customers send before signing. I owned it alone, from design to production: v1 shipped in under three weeks. Over 16 weeks in production, 74 % of questions were fully answered. The part I’m proudest of is what happens with the other 26 %.

1. The problem

The internal one. The answers exist, but they’re buried in policy documents nobody reads. So someone asks an expert on Slack, waits, and three weeks later someone else asks the same question. That costs twice: the search, and pulling an expert onto a question that was already settled.

The commercial one, and the real trigger for the project. Before a large customer signs, they send a security questionnaire: a spreadsheet of dozens to hundreds of questions (“Do you encrypt data at rest?”, “What’s your incident response SLA?”), worded differently every time, and filled in by hand.

Two constraints shaped everything after: freshness (policies are revised regularly and synced every night) and traceability (an auditor must be able to reach the clause behind every answer). That second one settled the approach: a fine-tuned model can’t cite its source, and on a compliance corpus a correct but untraceable answer is useless. So: retrieval-augmented generation (RAG), with every answer citing the policy it came from.

When the policies don’t cover something, Drag says so plainly instead of improvising from general knowledge: “I don’t see that covered in our indexed policies.” That reply is where the interesting part starts.

2. The self-healing loop

After I demoed v1, our CTO said something that changed the project:

An “I don’t know” shouldn’t be a dead end. It should be a work item for a human.

It wasn’t in the original spec. It became the most distinctive feature of the bot: every question Drag can’t fully answer goes to the experts, and their answer goes back into Drag’s knowledge, so the next person who asks gets an answer. The knowledge base fills its own gaps, and nobody has to maintain a FAQ.

  1. 1Employee

    Asks a question

    In a Slack DM, or as one row of a CSV security questionnaire.

  2. 2Drag

    Answers and classifies

    Searches three sources, answers, and labels the result yes / partial / no.

  3. 3Drag

    Isolates the gap

    Only the uncovered part, rewritten as one self-contained sentence under 25 words.

  4. 4Drag

    Dedupes, then posts

    New question → new thread in the security experts’ Slack channel. Already asked → “asked again” bump linking the original thread.

  5. 5Expert

    Replies SAVE: …

    Plain replies stay discussion. Only an explicit SAVE is stored; latest save wins, DELETE removes.

  6. 6Drag

    Stores it, instantly searchable

    Upserted into a table read synchronously by the next search, with no indexing delay.

Step 6 feeds step 2: the next person who asks gets an answer, and nobody had to maintain a FAQ.

The risk was simple: a channel full of redundant, vague bot messages gets muted, and then the loop is dead. Four mechanisms keep it precise and reliable.

Mechanism 1: isolate the gap

If someone asks “What’s our SOC 2 status, pen-test cadence, and incident response plan?” and the policies cover the first two, forwarding the whole message would waste the expert’s time. So when an answer is partial or missing, the model must also write out only the uncovered part, as one short self-contained sentence.

“What’s our SOC 2 status, pen-test cadence, and IR plan?” → “What is the company’s incident response plan?”

Why so strict? Because that sentence has two readers, and neither has the conversation: the expert reading it in Slack, and the deduplication search in mechanism 2. Both break on “and what about that?”.

Mechanism 2: deduplicate, and prefer missing a duplicate over merging wrongly

Before posting, Drag checks whether the question is already waiting for an answer, and asks the model to pick a match among the unresolved ones. Two choices make that safe:

Mechanism 3: a protocol experts can’t trigger by accident

The first version saved any thread reply as the answer. In practice, experts discuss before they agree, and that discussion was being stored as the canonical answer. So saving became opt-in: SAVE: followed by the answer stores it (latest save wins), DELETE removes it, and anything else is just discussion. That last rule, printed on every post, is what keeps the channel usable as a normal channel.

Expert answers then become a third source next to the policies, but a less trusted one: good enough to answer a colleague on Slack, never cited in a questionnaire an auditor will read.

Mechanism 4: never lose an answer silently

A loop that spans an LLM, a database and Slack can break quietly in a lot of places. If an expert’s answer lands on a thread Drag lost track of, it recovers the question from its own original post. If even that fails, it tells the expert nothing was saved: they asked for a save, so they can’t be left thinking it happened. The general rule I followed: anything a person can observe gets an explicit message, and anything that degrades the data gets a log with enough context to repair it by hand.

3. Does it work?

Over 16 weeks in production:

Classification Share
Fully answered 74.1 %
Partially answered (gap sent to experts) 18.4 %
Legitimate but not covered 7.5 %

The 7.5 % “not covered” isn’t a failure rate. It measures how well the corpus fits the real need, and those questions are exactly what feeds the loop.

The time saved relies on a baseline (~15 min per question by hand) that is the requester’s estimate, not a measurement, and some of the volume is probably induced demand: questions nobody would have asked without the bot.

Quality is checked by a regression suite built for a system that never answers the same way twice. Assertions are graded from the OpenTelemetry traces the runtime already emits, so the tests check what the bot did (whether it searched, whether it forwarded the gap) and not just its text, with a judge model scoring the rest. The key design choice: negative cases count as much as positive ones. An assistant tuned only against hallucination ends up refusing to answer, which is just a different failure, so the strictest test spends half its turns making sure Drag does answer what the policies cover. The same suite picked the model, by measurement rather than by feel.

4. Takeaway

Most of the loop’s refinements came from real usage rather than the spec. That’s a point for shipping early, but also a reminder that users paid for the cases I hadn’t planned for.

What I take away for any LLM product: let the model judge, let the code decide, and make every failure loud for the person who can act on it.

Drag is an internal Botpress tool, so there’s no public repo or demo.

Stack: TypeScript · Botpress ADK · Claude Sonnet 4.6 · Slack · RAG · LLM-as-judge evals over OpenTelemetry traces.