We present a deterministic finite-state machine (FSM) tokenizer coupled with a recursive-descent parser implemented in TypeScript. The tokenizer operates on a single pass over the source, emitting a stream of tokens that satisfy a strict lexical invariant: every token corresponds to a maximal prefix of the input that matches a terminal of the grammar. The parser consumes this stream to construct an abstract syntax tree (AST) while maintaining a height-balanced binary search tree for identifier resolution, guaranteeing that lookup, insertion, and rebalancing remain O(log N). Formal invariants are proven for both the FSM and the balanced tree, and static analysis confirms the absence of runtime errors on the target platform.
Manuscript Attribution: This manuscript is an empirical research preprint synthesized via agentic autonomous systems engineering pipelines adhering to academic two-column preprint conventions under the direction of Mateus Yonathan.
Hardware & Algorithmic Provenance: Algorithmic complexity bounds, execution profiles, and memory models cite verified static AST analysis and compiler toolchains evaluated by autonomous agent feedback loops.
Limitation of Liability & Scholarly Purpose: Disseminated solely for scholarly inquiry, peer discussion, and reproducible empirical comparison. It does not constitute formal peer-reviewed accreditation. Any reliance upon the mathematical proofs or complexity bounds herein is strictly at the reader's own legal and technical risk.
@article{yonathan2026_2026_10_15_finite_state_machine_tokenizer_and_,
title={Height-Balanced State-Machine Tokenizer and Lexical Parser in TypeScript: Invarian...},
author={Yonathan, Mateus},
journal={Polyglot Systems Lab Research Preprints},
year={2026},
publisher={Polyglot Systems Lab},
url={https://myonathanlinkedin.github.io/papers/2026-10-15_finite_state_machine_tokenizer_and_/},
howpublished={\url{https://raw.githubusercontent.com/myonathanlinkedin/mini-papers/main/papers/2026-10-15_finite_state_machine_tokenizer_and_/2026-10-15_finite_state_machine_tokenizer_and_.pdf}}
}