agent-orchestrator-benchmark/calculators/builder-adversary-stateless/run-02/machine-docs/JOURNAL-lex.md

JOURNAL — Phase lex

Implementation notes

Design choices

• Used @dataclass for Token to get __eq__ and __repr__ for free — useful in tests.
• Number parsing: scan while char is digit or .; if . in raw string → float(), else int(). Handles 42, 3.14, .5, 10..
• Single-char operators: dict lookup for O(1) dispatch.
• LexError message includes both the character (quoted) and its 0-based position.

Test run (verified locally)

$ python -m unittest -q
Ran 14 tests in 0.000s
OK

$ python -c "from calc.lexer import tokenize; print([(t.kind,t.value) for t in tokenize('3.5*(1-2)')])"
[('NUMBER', 3.5), ('STAR', '*'), ('LPAREN', '('), ('NUMBER', 1), ('MINUS', '-'), ('NUMBER', 2), ('RPAREN', ')'), ('EOF', None)]

$ python -c "from calc.lexer import tokenize; tokenize('1 @ 2')"
Traceback (most recent call last):
  ...
calc.lexer.LexError: unexpected character '@' at position 2