Scan left to right, trying each rule at the current position and taking
the first that claims it. Nothing overlaps, and every position advances, so
the result is always a valid non-overlapping token list.
When no rule claims the position, the scanner still skips the longest
lexeme any rule recognised there. An identifier that turned out not to
be a keyword, a word that is not a property name. Advancing a single
character instead would re-run every rule against the same lexeme at every
offset inside it, which is quadratic: a 20k-character identifier pasted
into a code block would freeze the editor. Skipping the whole lexeme is
also the more correct reading, constant is one word, and const inside
it was never a keyword.
Which span counts as the lexeme matters. A keywords rule is matched
against a word set, so its span is exactly one word. A followedBy rule's
span may reach past the word into punctuation that is a token in its own
right, Rust's macro rule recognises x! in x!=y, and skipping that far
swallows the ! the operator rule was about to claim. So a word-set rule's
span wins whenever one is available, and a conditional rule's span is used
only when no rule recognised a bare word here at all (HTML and CSS have no
word-set rule, and their attribute rules are what keeps a long word linear
there).
Scan left to right, trying each rule at the current position and taking the first that claims it. Nothing overlaps, and every position advances, so the result is always a valid non-overlapping token list.
When no rule claims the position, the scanner still skips the longest lexeme any rule recognised there. An identifier that turned out not to be a keyword, a word that is not a property name. Advancing a single character instead would re-run every rule against the same lexeme at every offset inside it, which is quadratic: a 20k-character identifier pasted into a code block would freeze the editor. Skipping the whole lexeme is also the more correct reading,
constantis one word, andconstinside it was never a keyword.Which span counts as the lexeme matters. A
keywordsrule is matched against a word set, so its span is exactly one word. AfollowedByrule's span may reach past the word into punctuation that is a token in its own right, Rust's macro rule recognisesx!inx!=y, and skipping that far swallows the!the operator rule was about to claim. So a word-set rule's span wins whenever one is available, and a conditional rule's span is used only when no rule recognised a bare word here at all (HTML and CSS have no word-set rule, and their attribute rules are what keeps a long word linear there).