3.4 KiB
TASK htmlmd-tokenizer: new crate notescraft-htmlmd with a forgiving HTML tokenizer (Clean paste)
Runs through OpenCode (specs/agents/opencode-runbook.md is attached automatically). Task 1 of specs/plugins-parity/clean-paste/tasks.md.
OWNS
crates/htmlmd/Cargo.toml,crates/htmlmd/src/lib.rs,crates/htmlmd/src/tokens.rs,crates/htmlmd/src/entities.rs,crates/htmlmd/tests/tokens.rs- registration (this task only): root
Cargo.toml(addnotescraft-htmlmd = { path = "crates/htmlmd" }in[workspace.dependencies], next tonotescraft-tasks),xtask/src/layers.rs(add("htmlmd", Class::Layer(0)),in the L0 block ofTABLE)
FILES TO OPEN
specs/plugins-parity/clean-paste/spec.mdD1, D4crates/tasks/Cargo.toml,crates/tasks/src/lib.rs,crates/tasks/src/error.rs(manifest, deny header, error style to copy)xtask/src/layers.rslines 36-60
GOAL / API
Write crates/htmlmd/src/tokens.rs in your SECOND tool call (stub + one failing test).
pub enum Token { Text(String), Open { name: String, attrs: Vec<(String, String)>, self_closing: bool }, Close(String), Comment(String) }
pub fn tokenize(html: &str, max_tokens: usize) -> Result<Vec<Token>>;
pub fn decode_entities(text: &str) -> String; // in entities.rs
Tag and attribute names lowercased. Attribute values quoted ("/') or bare; duplicate attribute: first wins; value decoded. <script>/<style> content is returned as a single raw Text token (not tokenized, not decoded). <!-- --> and <![CDATA[ and <!DOCTYPE and <?xml are consumed (comments returned as Comment, the rest dropped). A < that does not start a tag is text. Unterminated tag/comment/quote at end of input: treat the rest as text, no error. Void elements (br hr img input meta link) get self_closing: true. Entities: the 30 most common named ones (amp lt gt quot apos nbsp ndash mdash hellip lsquo rsquo ldquo rdquo copy reg trade bull middot laquo raquo euro pound yen cent deg times divide+), decimal &#NNN; and hex &#xHHH;; invalid code points (surrogates, > 0x10FFFF, 0) and unknown names stay verbatim. Limits: max_tokens exceeded -> Err(Error::TooLarge); input over 2 MiB (MAX_INPUT_BYTES const) -> Err(TooLarge). Error is one thiserror enum Error, pub type Result<T>. Crate header #![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)], [lints] workspace = true, rustdoc on public items, //! per module; dependency thiserror only. No [i], no [a..b]: walk char_indices/get(a..b); no recursion.
TESTS
<p class="a b">Hi & bye</p>; unquoted and single-quoted attributes; duplicate attribute; uppercase tags; void tags; script/style raw content with < inside; comment; <!--[if gte mso 9]>...<![endif]--> returns Comment; unterminated tag/comment/quote; a < b; entities (named, decimal, hex, invalid, unknown); emoji; 3 MiB input -> TooLarge; token cap; 5 000 LCG-seeded random strings of <>/="' a&;!- never panic.
VERIFY
export PATH="$HOME/.cargo/bin:$PATH"
cargo test -p notescraft-htmlmd
cargo clippy -p notescraft-htmlmd --all-targets -- -D warnings
rustfmt --edition 2024 crates/htmlmd/src/*.rs crates/htmlmd/tests/tokens.rs
cargo xtask layers
DO NOT
Add a dependency but thiserror; build a DOM or the Markdown converter (task 2); edit other crates; cargo fmt --all.