The REALbasic Tokenizer class allows developers to easily split text into tokens according to a set of regular expressions you specify.
Tokenizer can be used for a variety of purposes - for parsing data files, or even as part of the first stage for a programming language parser. Its functionality is similar to the LEXX library from the Unix world.
The Tokenizer test application demonstrates using the Tokenizer class to parse a simple BASIC program into its constituent tokens, which are displayed in a list beneath the text entry field.
The example program contains an intentional error - the word "nteger" should be "integer" - which serves to demonstrate how the Tokenizer handles unexpected tokens.
The application is set to re-tokenize the text whenever it is changed, so try editing the program and observe how the token list is regenerated.
Requirements:
· REALbasic
What`s New in This Release: [ read full changelog ]
· Fixed issue with the use of \\ to match the \ character immediately before a ( or [ bracket.
