agent-orchestrator-benchmark/calculators/builder-adversary/run-04/machine-docs/REVIEW-lex.md

REVIEW-lex.md — Adversary verdicts

Status

All D1–D4 PASS. Phase complete.

Verdicts

lex/D1: PASS @2026-06-15T01:20Z

Cold-ran:

• tokenize("42") → [Token('NUMBER', 42), Token('EOF', None)] — value is int ✓
• tokenize("3.14") → NUMBER(3.14) as float ✓
• tokenize(".5") → NUMBER(0.5) as float ✓
• tokenize("10.") → NUMBER(10.0) as float ✓
• EOF is always the final token with value=None across all inputs ✓

lex/D2: PASS @2026-06-15T01:20Z

Cold-ran:

• tokenize("+-*/()") → PLUS MINUS STAR SLASH LPAREN RPAREN EOF ✓
• tokenize("1+2*3") → NUMBER PLUS NUMBER STAR NUMBER EOF ✓
• tokenize("3.5*(1-2)") → [('NUMBER',3.5),('STAR','*'),('LPAREN','('),('NUMBER',1),('MINUS','-'),('NUMBER',2),('RPAREN',')'),('EOF',None)] — matches expected exactly ✓

lex/D3: PASS @2026-06-15T01:20Z

Cold-ran:

• tokenize(" 12 + 3 ") → NUMBER(12) PLUS NUMBER(3) EOF (spaces skipped) ✓
• tokenize("1\t+\t2") → tabs skipped ✓
• tokenize("1 @ 2") → raises LexError: unexpected character '@' at position 2 ✓
• tokenize("$5") → raises LexError with '$' in message ✓
• tokenize("1 x 2") → raises LexError with 'x' in message ✓
• Error message includes the offending char and its position ✓

lex/D4: PASS @2026-06-15T01:20Z

Cold-ran python -m unittest -q from repo root:

Ran 16 tests in 0.000s
OK

16 tests, 0 failures. Required coverage confirmed present:

• " 12 + 3 " tested in test_spaces_between ✓
• "3.5*(1-2)" tested in test_complex_expr ✓
• "1 @ 2" raises LexError tested in test_at_raises and test_error_position_in_message ✓

Adversary findings

AF-1: ValueError on malformed number literals (not a DoD blocker)

Malformed number sequences raise bare ValueError instead of LexError:

tokenize("..")   → ValueError: could not convert string to float: '..'
tokenize("1.2.3") → ValueError: could not convert string to float: '1.2.3'
tokenize(".")    → ValueError: could not convert string to float: '.'

Root cause: lexer.py:32 calls float(raw) without a try/except. The number scanner at lines 29–31 greedily consumes any sequence of digits and dots without validating it's a valid float first.

This does NOT block D1–D4 — these inputs are outside the explicit DoD scope (D3 only requires LexError for @, $, and letters). Filing as a quality note for the Builder to address optionally, or for a future phase.