docs: design for grounding and feedback in the /ia module
Records what was measured rather than assumed: pt.minecraft.wiki as the source, forced tool_choice for term selection (5/5 against 2/7 for free-text extraction), full article text over exintro, and M2.7 over M3. Also records two defects found while measuring — empty replies caused by hidden reasoning eating max_tokens at the deployed value of 300, and foreign-token leakage into Portuguese answers.
This commit is contained in:
marcos
@@ -0,0 +1,172 @@
|
||||
# Design — grounding and feedback for the `/ia` module
|
||||
|
||||
**Date:** 2026-08-05
|
||||
**Status:** approved
|
||||
|
||||
## Why
|
||||
|
||||
`/ia` shipped earlier today and was used five times by real players. Every
|
||||
substantive answer contained an error:
|
||||
|
||||
| asked | answered | reality |
|
||||
| --- | --- | --- |
|
||||
| how to get a camel | spawns in desert villages "and badlands" | badlands is wrong; never said a saddle is involved |
|
||||
| crossing the nether roof | "boat on soul sand, water on top, push the boat" | invented; not a technique |
|
||||
| best mob farm spot | "Y 0–64, mobs spawnam mais *concentrated there*" | dubious, and English leaked into the Portuguese |
|
||||
|
||||
The model states Minecraft mechanics confidently and incorrectly. No amount of
|
||||
prompt tuning fixes that, because the model does not know what it does not
|
||||
know. It needs sources.
|
||||
|
||||
Two further defects were found while measuring:
|
||||
|
||||
- **Empty answers.** M2.7 emits hidden reasoning that counts against
|
||||
`max_tokens`. At the deployed value of 300 it returned empty content twice in
|
||||
testing. Players see "Não consegui resposta agora" and assume the feature is
|
||||
broken.
|
||||
- **Foreign-token leakage.** Replies contained `搭档` (Chinese) and
|
||||
`contiennent` (French) mid-sentence.
|
||||
|
||||
## What was measured
|
||||
|
||||
Everything below is measured against the live MiniMax key, not assumed.
|
||||
|
||||
### Retrieval source
|
||||
|
||||
`pt.minecraft.wiki` answers in Portuguese, needs no API key, and responds in
|
||||
~0.3s. Its camel article contains the exact fact the model missed. It returns
|
||||
**HTTP 403 to a default user agent** — an identifying `User-Agent` is required.
|
||||
|
||||
### How to choose the search term
|
||||
|
||||
MediaWiki cannot parse conversational Portuguese, so the raw question is
|
||||
useless as a query. Four methods, same questions:
|
||||
|
||||
| method | usable terms |
|
||||
| --- | --- |
|
||||
| raw question | 0 / 6 |
|
||||
| local stopword strip | 1 / 6 |
|
||||
| free-text extraction call | 2 / 7 |
|
||||
| **tool call with forced `tool_choice`** | **5 / 5** |
|
||||
|
||||
Free-text extraction fails for the same reason answers came back empty: hidden
|
||||
reasoning eats a small `max_tokens` budget. A tool call returns a *structured*
|
||||
argument, which survives. Optional tool choice is not enough — with
|
||||
`tool_choice: auto` the model skipped the search on the hardest question, the
|
||||
one it had already answered wrong twice. Forcing the call fixes that.
|
||||
|
||||
### How much of the article to send
|
||||
|
||||
`exintro` returns only the lead paragraph. Grounded on intros alone, both
|
||||
models honestly answered "não tenho certeza" to questions they had previously
|
||||
answered correctly — truthful but useless. Sending the **full article text**
|
||||
(~7k chars) produced correct answers for camels, mob farms and creeper drops.
|
||||
|
||||
### Which model
|
||||
|
||||
M2.7 and M3, both grounded on full articles:
|
||||
|
||||
| | tokens | outcome |
|
||||
| --- | --- | --- |
|
||||
| **M2.7** | 6647 | correct on camel, mob farm, creeper |
|
||||
| M3 | 7749 | refused the mob farm question M2.7 answered well |
|
||||
|
||||
M2.7 is chosen. M3 costs ~17% more and refuses more. Ungrounded, M3 also
|
||||
invented "poeira de guncotton" as a creeper drop.
|
||||
|
||||
### The gap wiki text cannot close
|
||||
|
||||
`explaintext` strips tables, and brewing and crafting recipes live in tables.
|
||||
The fire-resistance article is only 1086 characters for this reason, and the
|
||||
recipe question failed even fully grounded. Recipes therefore come from
|
||||
`Bukkit.recipeIterator()` — the running server's own data, authoritative for
|
||||
this exact version, free and instant.
|
||||
|
||||
## Constraints
|
||||
|
||||
- **No restart now.** The jar is built and deployed dormant; it goes live at
|
||||
the next restart.
|
||||
- **Cost is not a constraint.** The MiniMax Plus plan allows 4.5–12M tokens per
|
||||
5 hours. At ~2000 tokens per grounded question that is ~1800 questions per
|
||||
window, which a five-player server will never approach. `limite-diario`
|
||||
stays, but as chat-spam protection, not spend control.
|
||||
- **Latency is the real budget.** Answers already take 4–6s. Every added
|
||||
context token makes chat feel slower.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
/ia <pergunta>
|
||||
│
|
||||
├─ permission · module · cooldown · daily cap (existing)
|
||||
│
|
||||
├─ context assembly
|
||||
│ contexto.yml always server facts, hand written
|
||||
│ correcoes.yml on keyword match operator corrections
|
||||
│ recipe lookup recipe questions Bukkit.recipeIterator()
|
||||
│ wiki article PRECISO only forced tool call → pt.minecraft.wiki
|
||||
│ last 3 exchanges same player, 10 min window
|
||||
│
|
||||
├─ MiniMax-M2.7, max_tokens 1200
|
||||
│
|
||||
└─ sanitise → chat, with reaction buttons
|
||||
```
|
||||
|
||||
Both HTTP calls run on the async thread that already exists; only delivery hops
|
||||
back to the main thread.
|
||||
|
||||
**Caching.** Wiki articles cached in memory by title, 200 entries, 6h TTL.
|
||||
Five people asking about creepers costs one fetch. Lost on restart, which is
|
||||
fine.
|
||||
|
||||
**Profiles.** `ia.perfil: ECONOMICO | PRECISO`, switched live with
|
||||
`/ia perfil <nome>`, no restart. ECONOMICO skips the wiki round trip. Its
|
||||
purpose is latency, not cost.
|
||||
|
||||
## Safety
|
||||
|
||||
The boundary is structural, not prompt-level, and it does not weaken by adding
|
||||
a tool:
|
||||
|
||||
- **Tools may only read.** Never write, never execute, never touch game state.
|
||||
`buscar_wiki` performs one HTTPS GET against a hardcoded host with a
|
||||
URL-encoded term. The model chooses an *argument*, never an operation.
|
||||
- The reply goes to `sendMessage` and nowhere else. It never reaches the
|
||||
command dispatcher.
|
||||
- `sanitise()` strips `§` codes, markdown, emoji and leading slashes.
|
||||
|
||||
A player asking the model to run `ls` or `/give` gets a sentence back. There is
|
||||
no function that could do otherwise.
|
||||
|
||||
## Error handling
|
||||
|
||||
| failure | behaviour |
|
||||
| --- | --- |
|
||||
| empty content | retry once at a higher token ceiling, then apologise |
|
||||
| reply contains CJK or other foreign script | discard, retry once |
|
||||
| wiki 403 / timeout / no hit | answer without the article, and say the wiki was not consulted |
|
||||
| MiniMax non-zero `base_resp` | log and apologise; HTTP 200 does not mean success |
|
||||
| tool call absent despite forcing | fall back to answering ungrounded |
|
||||
|
||||
## Features
|
||||
|
||||
- **Privacy per question.** `/ia` public, `/iap` visible only to the asker.
|
||||
Operators can force the module private in config.
|
||||
- **Reactions** on answers, reusing the existing `Reactions` class so Bedrock
|
||||
gets typed equivalents.
|
||||
- **Correction loop.** The asker marks an answer wrong; an operator writes the
|
||||
right answer with `/ia corrigir`; it is stored in `correcoes.yml` and
|
||||
injected on similar future questions. No fine-tuning, no extra API cost.
|
||||
- **New permissions.** `canalhandia.ia.privado`, `canalhandia.ia.corrigir`,
|
||||
`canalhandia.ia.perfil`.
|
||||
|
||||
## Testing
|
||||
|
||||
- Unit: `sanitise()` against markdown, emoji, `§`, leading slashes, CJK.
|
||||
- Unit: term extraction from a stubbed tool response; recipe lookup for a known
|
||||
item.
|
||||
- Integration, against the live key: the five questions in this document, with
|
||||
the camel, mob farm and creeper answers checked for the specific facts they
|
||||
previously got wrong.
|
||||
- Live: after the next restart, confirm `/canalhandia status` reports the
|
||||
profile and that a real question is grounded.
|
||||
Reference in New Issue
Block a user