Blog

Writing a Lexer: From Characters to Tokens with a Bookstore Filter Language in C#

A lexer is a state machine that splits characters into tokens and knows the line and column of each one. We write a single-pass C# lexer over ReadOnlySpan<char> for the Bookstore filter language: maximal munch, keyword matching that survives the Turkish culture, number parsing with InvariantCulture, error recovery that respects surrogate pairs, and clear error messages. Real output shows where a regex lexer breaks and why Roslyn keeps whitespace as trivia.