diff --git a/anubis_dev/library/lexical_analysis/fast_lexer_4.anubis b/anubis_dev/library/lexical_analysis/fast_lexer_4.anubis index 3be4e2e..034188f 100644 --- a/anubis_dev/library/lexical_analysis/fast_lexer_4.anubis +++ b/anubis_dev/library/lexical_analysis/fast_lexer_4.anubis @@ -18,6 +18,10 @@ now have different values for this parameter for multiple lexers plugged on the same lexing stream. + - The reconstruction of precompiled lexers is automated. Each one is reconstructed + only if something is changed in the description of the lexer. This is performed + by using an sha1 hash of the description, which is stored in the first line of the + generated file. This tool is similar to the Unix tool LEX/FLEX (with some differences, but it is more or less equivalent). If you want to use this tool, you @@ -407,7 +411,8 @@ public type LexerAction($Token,$Aux): The third alternative in 'LexerAction($Token,$Aux)' is a variant of the second one. Instead of extracting the token from the buffer, the function provides tools for extracting a part of the token. The argument 'length' is the total length of the token. The function 'extract' enables to extract the part of the token - located between positions 's' (included) and 'e' (not included). + located between positions 's' (included) and 'e' (not included), relative to the token. For example, + extract(0,length) gives the whole token. public type LexerItem($Token,$Aux): @@ -3040,6 +3045,7 @@ define One ) = if l is precompiled_fast_lexer(fba,sba) then print(f,"\n This file was automatically generated by 'fast_lexer_4.anubis'.\n"); + print(f," Don't modify it in any manner.\n"); print(f,"\n The (deterministic) automaton has "+to_decimal(length(actions_ranks))+" states.\n"); //dump(f,dfa); print(f,"\npublic define (List(Int),PrecompiledFastLexer)\n"); @@ -3075,7 +3081,7 @@ define List(Int) reconstructing the lexer when the description did not change). The signature is changed if any of the regular expressions is changed or if their order is changed or if the sort of action is changed (this last point ensures that the list of - action ranks remains correct in the generated file). + action sorts/ranks remains correct in the generated file). define List(ByteArray) extract_regexprs ( -- libgit2 0.21.4