552 B
552 B
DECISIONS (append-only, shared)
lex phase
Token type
Used a simple class with __slots__ for efficiency and __eq__ for easy test assertions. No namedtuple to allow future extension (e.g. position tracking for later phases).
Number regex
r'\d+\.?\d*|\.\d+' — handles 42, 3.14, 10., .5. The alternation order matters: longest match first via re.match at position i.
EOF token
Token('EOF', None) — value is None since EOF carries no semantic content; downstream parser phases can type-check on kind == 'EOF'.