Anubis / Anubis

Commit 09f8720dc229462069495bb182b637bddc99dba3

Authored by Alain Prouté
1 parent bcb6ccd9

Added more comments in fast_lexer_4.

Showing 1 changed file with 8 additions and 2 deletions   Show diff stats
Inline Side-by-side
anubis_dev/library/lexical_analysis/fast_lexer_4.anubis
  Show/Hide comments   View file @09f8720
... ... @@ -18,6 +18,10 @@
18 18 now have different values for this parameter for multiple lexers plugged on
19 19 the same lexing stream.
20 20  
  21 + - The reconstruction of precompiled lexers is automated. Each one is reconstructed
  22 + only if something is changed in the description of the lexer. This is performed
  23 + by using an sha1 hash of the description, which is stored in the first line of the
  24 + generated file.
21 25  
22 26 This tool is similar to the Unix tool LEX/FLEX (with some differences,
23 27 but it is more or less equivalent). If you want to use this tool, you
... ... @@ -407,7 +411,8 @@ public type LexerAction($Token,$Aux):
407 411 The third alternative in 'LexerAction($Token,$Aux)' is a variant of the second one. Instead of extracting
408 412 the token from the buffer, the function provides tools for extracting a part of the token. The argument
409 413 'length' is the total length of the token. The function 'extract' enables to extract the part of the token
410   - located between positions 's' (included) and 'e' (not included).
  414 + located between positions 's' (included) and 'e' (not included), relative to the token. For example,
  415 + extract(0,length) gives the whole token.
411 416  
412 417  
413 418 public type LexerItem($Token,$Aux):
... ... @@ -3040,6 +3045,7 @@ define One
3040 3045 ) =
3041 3046 if l is precompiled_fast_lexer(fba,sba) then
3042 3047 print(f,"\n This file was automatically generated by 'fast_lexer_4.anubis'.\n");
  3048 + print(f," Don't modify it in any manner.\n");
3043 3049 print(f,"\n The (deterministic) automaton has "+to_decimal(length(actions_ranks))+" states.\n");
3044 3050 //dump(f,dfa);
3045 3051 print(f,"\npublic define (List(Int),PrecompiledFastLexer)\n");
... ... @@ -3075,7 +3081,7 @@ define List(Int)
3075 3081 reconstructing the lexer when the description did not change). The signature is
3076 3082 changed if any of the regular expressions is changed or if their order is changed
3077 3083 or if the sort of action is changed (this last point ensures that the list of
3078   - action ranks remains correct in the generated file).
  3084 + action sorts/ranks remains correct in the generated file).
3079 3085 define List(ByteArray)
3080 3086 extract_regexprs
3081 3087 (
... ...