diff --git a/docs/plans/2026-08-05-ia-improvements-design.md b/docs/plans/2026-08-05-ia-improvements-design.md new file mode 100644 index 0000000..6b167e3 --- /dev/null +++ b/docs/plans/2026-08-05-ia-improvements-design.md @@ -0,0 +1,172 @@ +# Design — grounding and feedback for the `/ia` module + +**Date:** 2026-08-05 +**Status:** approved + +## Why + +`/ia` shipped earlier today and was used five times by real players. Every +substantive answer contained an error: + +| asked | answered | reality | +| --- | --- | --- | +| how to get a camel | spawns in desert villages "and badlands" | badlands is wrong; never said a saddle is involved | +| crossing the nether roof | "boat on soul sand, water on top, push the boat" | invented; not a technique | +| best mob farm spot | "Y 0–64, mobs spawnam mais *concentrated there*" | dubious, and English leaked into the Portuguese | + +The model states Minecraft mechanics confidently and incorrectly. No amount of +prompt tuning fixes that, because the model does not know what it does not +know. It needs sources. + +Two further defects were found while measuring: + +- **Empty answers.** M2.7 emits hidden reasoning that counts against + `max_tokens`. At the deployed value of 300 it returned empty content twice in + testing. Players see "Não consegui resposta agora" and assume the feature is + broken. +- **Foreign-token leakage.** Replies contained `搭档` (Chinese) and + `contiennent` (French) mid-sentence. + +## What was measured + +Everything below is measured against the live MiniMax key, not assumed. + +### Retrieval source + +`pt.minecraft.wiki` answers in Portuguese, needs no API key, and responds in +~0.3s. Its camel article contains the exact fact the model missed. It returns +**HTTP 403 to a default user agent** — an identifying `User-Agent` is required. + +### How to choose the search term + +MediaWiki cannot parse conversational Portuguese, so the raw question is +useless as a query. Four methods, same questions: + +| method | usable terms | +| --- | --- | +| raw question | 0 / 6 | +| local stopword strip | 1 / 6 | +| free-text extraction call | 2 / 7 | +| **tool call with forced `tool_choice`** | **5 / 5** | + +Free-text extraction fails for the same reason answers came back empty: hidden +reasoning eats a small `max_tokens` budget. A tool call returns a *structured* +argument, which survives. Optional tool choice is not enough — with +`tool_choice: auto` the model skipped the search on the hardest question, the +one it had already answered wrong twice. Forcing the call fixes that. + +### How much of the article to send + +`exintro` returns only the lead paragraph. Grounded on intros alone, both +models honestly answered "não tenho certeza" to questions they had previously +answered correctly — truthful but useless. Sending the **full article text** +(~7k chars) produced correct answers for camels, mob farms and creeper drops. + +### Which model + +M2.7 and M3, both grounded on full articles: + +| | tokens | outcome | +| --- | --- | --- | +| **M2.7** | 6647 | correct on camel, mob farm, creeper | +| M3 | 7749 | refused the mob farm question M2.7 answered well | + +M2.7 is chosen. M3 costs ~17% more and refuses more. Ungrounded, M3 also +invented "poeira de guncotton" as a creeper drop. + +### The gap wiki text cannot close + +`explaintext` strips tables, and brewing and crafting recipes live in tables. +The fire-resistance article is only 1086 characters for this reason, and the +recipe question failed even fully grounded. Recipes therefore come from +`Bukkit.recipeIterator()` — the running server's own data, authoritative for +this exact version, free and instant. + +## Constraints + +- **No restart now.** The jar is built and deployed dormant; it goes live at + the next restart. +- **Cost is not a constraint.** The MiniMax Plus plan allows 4.5–12M tokens per + 5 hours. At ~2000 tokens per grounded question that is ~1800 questions per + window, which a five-player server will never approach. `limite-diario` + stays, but as chat-spam protection, not spend control. +- **Latency is the real budget.** Answers already take 4–6s. Every added + context token makes chat feel slower. + +## Architecture + +``` +/ia + │ + ├─ permission · module · cooldown · daily cap (existing) + │ + ├─ context assembly + │ contexto.yml always server facts, hand written + │ correcoes.yml on keyword match operator corrections + │ recipe lookup recipe questions Bukkit.recipeIterator() + │ wiki article PRECISO only forced tool call → pt.minecraft.wiki + │ last 3 exchanges same player, 10 min window + │ + ├─ MiniMax-M2.7, max_tokens 1200 + │ + └─ sanitise → chat, with reaction buttons +``` + +Both HTTP calls run on the async thread that already exists; only delivery hops +back to the main thread. + +**Caching.** Wiki articles cached in memory by title, 200 entries, 6h TTL. +Five people asking about creepers costs one fetch. Lost on restart, which is +fine. + +**Profiles.** `ia.perfil: ECONOMICO | PRECISO`, switched live with +`/ia perfil `, no restart. ECONOMICO skips the wiki round trip. Its +purpose is latency, not cost. + +## Safety + +The boundary is structural, not prompt-level, and it does not weaken by adding +a tool: + +- **Tools may only read.** Never write, never execute, never touch game state. + `buscar_wiki` performs one HTTPS GET against a hardcoded host with a + URL-encoded term. The model chooses an *argument*, never an operation. +- The reply goes to `sendMessage` and nowhere else. It never reaches the + command dispatcher. +- `sanitise()` strips `§` codes, markdown, emoji and leading slashes. + +A player asking the model to run `ls` or `/give` gets a sentence back. There is +no function that could do otherwise. + +## Error handling + +| failure | behaviour | +| --- | --- | +| empty content | retry once at a higher token ceiling, then apologise | +| reply contains CJK or other foreign script | discard, retry once | +| wiki 403 / timeout / no hit | answer without the article, and say the wiki was not consulted | +| MiniMax non-zero `base_resp` | log and apologise; HTTP 200 does not mean success | +| tool call absent despite forcing | fall back to answering ungrounded | + +## Features + +- **Privacy per question.** `/ia` public, `/iap` visible only to the asker. + Operators can force the module private in config. +- **Reactions** on answers, reusing the existing `Reactions` class so Bedrock + gets typed equivalents. +- **Correction loop.** The asker marks an answer wrong; an operator writes the + right answer with `/ia corrigir`; it is stored in `correcoes.yml` and + injected on similar future questions. No fine-tuning, no extra API cost. +- **New permissions.** `canalhandia.ia.privado`, `canalhandia.ia.corrigir`, + `canalhandia.ia.perfil`. + +## Testing + +- Unit: `sanitise()` against markdown, emoji, `§`, leading slashes, CJK. +- Unit: term extraction from a stubbed tool response; recipe lookup for a known + item. +- Integration, against the live key: the five questions in this document, with + the camel, mob farm and creeper answers checked for the specific facts they + previously got wrong. +- Live: after the next restart, confirm `/canalhandia status` reports the + profile and that a real question is grounded.