agent-orchestrator-benchmark/calculators/builder-adversary-stateless/run-03/machine-docs/JOURNAL-lex.md

JOURNAL — Phase lex (Builder)

Implementation

Built calc/lexer.py with:

• Token dataclass-style class with __slots__ = ("kind", "value") for efficiency
• LexError(Exception) for invalid characters
• tokenize(src) using re.compile(r"\d+\.?\d*|\.\d+") for number matching
  • Integer if no . in raw string; float otherwise
  • Single-char dispatch table _SINGLE for operators/parens
  • Raises LexError with char + position for unknown characters
  • Appends EOF token at end

Test run

$ python -m unittest -q
Ran 21 tests in 0.000s
OK

Cold-verify outputs

$ python -c "from calc.lexer import tokenize; print([(t.kind,t.value) for t in tokenize('3.5*(1-2)')])"
[('NUMBER', 3.5), ('STAR', '*'), ('LPAREN', '('), ('NUMBER', 1), ('MINUS', '-'), ('NUMBER', 2), ('RPAREN', ')'), ('EOF', None)]

$ python -c "from calc.lexer import tokenize; tokenize('1 @ 2')"
Traceback (most recent call last):
  ...
calc.lexer.LexError: unexpected character '@' at position 2