agent-orchestrator-benchmark/calculators/builder-adversary-stateless/run-04/machine-docs/JOURNAL-lex.md

JOURNAL — phase lex

Build session

Design decisions

• Token is a dataclass with kind: str and value: Union[int, float, None]. Operator tokens have value=None; NUMBER tokens carry their parsed numeric value (int for integers, float when . present).
• LexError is a plain Exception subclass defined in the module.
• Used re module with _NUMBER_RE = re.compile(r'\d+\.?\d*|\.\d+') to match integers, floats-with-integer-part, and leading-dot floats.
• _SINGLE dict maps single chars to token kinds.

Test run output

python -m unittest -q
----------------------------------------------------------------------
Ran 17 tests in 0.000s

OK

Verify command outputs

python -c "from calc.lexer import tokenize; print([(t.kind,t.value) for t in tokenize('3.5*(1-2)')])"
[('NUMBER', 3.5), ('STAR', None), ('LPAREN', None), ('NUMBER', 1), ('MINUS', None), ('NUMBER', 2), ('RPAREN', None), ('EOF', None)]

python -c "from calc.lexer import tokenize; tokenize('1 @ 2')"
Traceback (most recent call last):
  ...
calc.lexer.LexError: unexpected character '@' at position 2