Anubis / Anubis

                           The Anubis Project
                           
     A tool for an easy automatic loading/reloading of secondary modules.
        
        
   -------------------------------------------------------------------------
   *** (1) Overview. 
   *** (2) Carrying on. 
   *** (3) Automatic loading of servers. 
   *** (4) Organizing the loading of modules. 
   *** (5) Having a module loaded only temporarily.
   *** (6) Getting informations about loaded modules.  
   -------------------------------------------------------------------------


   *** (1) Overview. 
   
   This is a metaprogram generating an Anubis source file handling automatic
   loading/reloading of your secondary module (.adm files). Each module file is 
   loaded a first time and reloaded (within a parametrizable number of milliseconds) 
   each time the module file is updated (i.e. each time the adm file gets a more recent
   last modification time). 
   
   This means that while your program is running, you can recompile parts of your 
   program which will be automatically linked to the running program without stopping
   it. Everything will work fine because of the mechanisms implemented here (and if you
   follow some simple rules explained below). In particular, if the previous version 
   of a secondary module is currently in use, it will continue to execute conccurrently with
   the new one, until it has finished its job. 
   
   Note: the file names can be the names of symbolic links pointing to the actual 
   modules. 
   
   This metaprogram also manages families of modules. Such a family of modules is 
   identified by a file mask. All the modules in a family must have the same type. 
   
   Each adm loader has a name. In the explanations below we assume that the name of 
   the adm loader is 'bubu'. This name is used for constructing other names, namely
   type names and function names. 
   
   An adm file cannot contain non deterministic data such as network connections or
   dynamic variables. Nevertheless, when a module is loaded by 'load_adm' (defined
   in 'predefined.anubis'), the main program in the adm is executed. For example, 
   if the secondary adm is defined as: 
   
   global define MyAdm
      myadm 
        =
      <term>.

   <term> is executed while loading, and this execution can create dynamic variables
   and/or network connections or any other non deterministic item. We shall discuss
   this question in more details below in the section 'Automatic loading of servers', 
   because loading servers and properly starting and shutdowning automatically reloaded
   servers is somewhat delicate. 

   
          
   *** (2) Carrying on. 
          
   In order to use this metaprogram, you have to provide a list of metadata of type: 
   
public type SecondaryAdm:     // such a datum describes an adm (or a family of adms)
   adm
     (
       Int            delay,         // number of milliseconds that the loader waits in the loader loop
       String         directory,     // the path of the directory where to find the module(s)
       String         name,          // name (may be a mask) of the module(s) to be (re)loaded
       String         type           // type of the module(s) (in Anubis syntax)
     ). 
          
   If 'directory' is an absolute path, the modules(s) will be searched for in this path
   only. Otherwise, 'directory' is first assumed to be relative to the current working
   directory, and if not successful, it is assumed to be relative to the directory 
   'my_anubis/modules/'. 
          
   Create a file of your own, say 'my_modules.load', looking like this:
   
   
 
 --- my_modules.load ----------------------------------------------
    
 read tools/adm_autoreload.anubis
    
 global define One 
    make_my_adm_loader
      (
        List(String) args     // not used
      ) =
    make_adm_loader(      // this function is defined below
      "bubu",                                        // name of adm loader
      "generated/my_modules.load.anubis",            // path of the file to be generated
      ["something.anubis",...],                      // files that must be 'read' by the generated program
      [
        adm(1000, "...",  "*_gaga.adm",  "Gaga"),       // this is a family of modules of type 'Gaga'
        adm(2000, "...",  "toto.adm",    "Toto"),       // this is a single module of type 'Toto'
        ...
      ]).
    
 execute anbexec make_my_adm_loader            // generate the file 
 read generated/my_modules.load.anubis         // check that everything is ok
 
 --- end of my_modules.load ----------------------------------------
 
 
    
   'make_adm_loader' is declared as follows:
   
public define One
   make_adm_loader
     (
       String              loader_name, 
       String              target_file_path, 
       List(String)        to_be_read, 
       List(SecondaryAdm)  modules
     ). 
   
   Compiling 'my_modules.load' produce the file 'generated/my_modules.load.anubis' 
   that you can 'read' in other source files of your project. The public part in 
   this file contains: 
   
 public define List(Gaga)    __gaga_adm      (AdmLoader_bubu a). 
 public define Maybe(Toto)   toto_adm        (AdmLoader_bubu a). 
 ...
 
   These functions are non deterministic since they return the current 
   version of the content of secondary modules, which may change during
   execution of your main program. Hence, they must be called each time 
   you need these contents. 
   
   The above commands take an argument of type 'AdmLoader_bubu' that you can 
   obtain by 'starting the adm loader'. The starter function is declared like
   this in the generated file:  
   
 public define AdmLoader_bubu
    start_adm_loader_bubu
      (
        Var(Bool)                 shutdown_loader,
        String -> One             warn,
        Gaga -> One               start___gaga_adm, 
        Gaga -> One               stop___gaga_adm, 
        Toto -> One               start_toto_adm,
        Toto -> One               stop_toto_adm,
        ...
      ).
 
   The command 'start_adm_loader_bubu' creates and starts the adm loader named 
   'bubu'. The reason why the adm loader must have a name is that you can launch
   several adm loaders (which also have distinct types).  
   
   In general, you call the above function only once. The variable 'shutdown_loader'
   must contain 'false'. The adm loader (which is running in the backgound)
   is shutdown as soon as the value of this variable becomes 'true'. 
   
   The function 'warn' is called each time a loading problem occurs. Its argument
   is the error message. This function can print on the standard output, or to 
   a log file, or send an email to the administrator, ...
   
   The other arguments are callbacks which are used as follows by the loader. When
   a module is loaded the function 'stop_...' is applied to the previous version
   (if any) of the module. This allows for example, in the case
   the module contains a server, to shutdown the previous 
   version if there was one. The function 'start...' can start the new server. 
   The functions 'stop_...' are also used when the adm loader 
   itself is shutdown, i.e. when the variable 'shutdown_loader' receives the value 
   'true'. In case of a family of modules, the 'stop_...' function is applied to all
   the modules of the familly which are currently loaded. 
   
   Typically, these callbacks are used for starting and stopping servers, and in this
   case, the tool for retrieving the value of the module is likely not to be used. 
   
   
   
   *** (3) Automatic loading of servers. 
   
   As explained above, if a secondary adm contains a server, it is important to be able
   to start and shutdown such a server. This is the job of the 'start...' and 'stop...'
   functions. 
   
   Since version 1.13, the functions 'start_server' and 'start_ssl_server' defined in
   'predefined.anubis' have a new argument: 
   
      Var(Bool)    shutdown_required
      
   which can be used to shutdown the server. 
      
   In order to be able to properly start and shutdown servers, in particular when
   they are reloaded, and even if you have a family of servers (of the same type), 
   proceed as follows. In order to be better understood, we consider a real example. 
   
   Have a look at 'library/mail/smtp_server.anubis'. You will find the tool: 
   
   public define One
      start_smtp_server
        (
          SMTP_ServerDescription description, 
          Var(Bool)              shutdown
        ). 

   which allows to start a SMTP (outgoing mail) server from a formal description of 
   the server and a shutdown variable. In order to make a secondary adm for such a server, 
   write this:
   
   type LoadableSmtpServer: 
      smtp_serv(
          SMTP_ServerDescription description, 
          Var(Bool)              shutdown
               ). 
         
   and define your secondary module like this:       
               
   global define LoadableSmtpServer
      smtp_server_datum
        =
      with shutdown = var((Bool)false), 
      smtp_serv(my_server_description,
                shutdown). 
                
   When the secondary module is loaded, we get a datum of type LoadableSmtpServer. Then we
   can define the 'start' and 'stop' functions like this: 
   
   define One 
      start_smtp_srv
        (
          LoadableSmtpServer  s
        ) =
      if s is smtp_serv(desc,shutdown) then 
      start_smtp_server(desc,shutdown). 
      
   define One
      stop_smtp_srv
        (
          LoadableSmtpServer  s
        ) =
      if s is smtp_serv(desc,shutdown) then 
      shutdown <- true. 
      
   Notice that a new boolean variable is created at each loading so that each
   server has its own variable and executing any of the above two functions
   has an action only on this particular server.    
   
   If defined like this, these functions will work properly, even if you run
   a family of SMTP servers. In particular, garbage-collection will eliminate
   all stopped servers. For example, you may have three SMTP servers on ports
   8025, 8026 and 8027. The adm files should be named 'smtp_serv_8025.adm', 
   'smtp_serv_8026.adm' and 'smtp_serv_8027.adm', and the mask should be 
   'smtp_serv_*.adm'.  


   *** (4) Organizing the loading of modules. 
   
   Another important issue is the organization of adm loaders in a 'adm modularized'
   project. Here by 'adm modularized' we mean that the project is split into a main
   adm file and (many) secondary adm files. Doing like this has the advantage that 
   modifying the source of a module does not require to recompile the whole project
   (not even to stop the running programs). 
   
   However, the pitfall is that if two modules, say 'mod1' and 'mod2' are both using
   a third one, say 'mod_tools', they should not dynamically load 'mod_tools' independantly, 
   otherwise you will have two instances of 'mod_load' running in your program, which is 
   of course a waste of memory. 
   
   In this case, the module 'mod_tools' should better be loaded by the module 'mod0' whose
   job could be to load 'mod1' and 'mod2' and to transmit to them the content of 'mod_tools'. 
   This is like in some restaurants : you can either come with your own food or buy it in 
   the restaurant. Anyway, if your project is quite big, deciding who is loading who can be 
   tricky. 
   
   So, it appears that if 'mod1' is using 'mod_tools', you have two methods: 
   
     (1) mod1 loads mod_tools (and doesn't worry about other modules)
     (2) mod0 (or the primary module) loads mod_tools and transmits its
         content to mod1 after mod1 is loaded. 
         
   Case (1) is convenient if 'mod_tools' is a 'specialized' module (used only by
   mod1), and case (2) is convenient if 'mod_tools' is a 'general purpose' module. 
   
   Notice that a general purpose module is not the same thing as a source file. Indeed, 
   consider the case of 'tools/basis.anubis' which contains general purpose tools, and 
   which is quite big. It is not organized like a toolbox. It looks much more like
   a shop, where you can just peek what you need, no more. If your program has a 
   'read tools/basis.anubis' it does not contain all the stuff in 'basis.anubis'  
   because the compiler generates code only for what is actually used by your program. 
   Actually, this is even a little more complicated because the compiler generates
   a piece of code for each required instance of a schema. For example, in basis.anubis
   we have the schema:  
   
   public define Bool
      $T x /= $T y
         =
      ...
      
   If you use this in your program for comparing strings and for comparing integers, the 
   compiler will include two instances of this schema into your adm, one for comparing
   strings and one for comparing integers (they don't have the same code). 
   
   On the contrary, a secondary module contains only actual data (not schemas), and if 
   you wanted to include the above schema in a library, you would have to limit yourself
   to a sequences of instances for particular values of $T. 
   
   So, secondary modules and files like basis.anubis are clearly for distinct purposes. 
   Don't try to transform every file in your project into a secondary module. This will
   just not work. 
   
   Now, consider that we have an actual, fully justified, secondary module. If it is 
   a specialized module, just load it from the module using it. On the contrary, if it 
   is a general purpose module, you may want to load it when your program starts and to
   transmit its content to some modules to be loaded next. 
   
   However, there is a problem, because if you have two general purpose modules, it may be
   the case that one of them requires the content of the other one. You cannot load them
   in the same loader, because the loader loads the modules asynchronously. So, you must have 
   at least two loaders, one for loading a first bunch of modules, and another one for a
   second bunch of modules needing the contents of the modules of the first bunch. Of course, 
   it may be necessary to have more than two loaders. In order to put this method at work, 
   you probably need to draw a graph of dependancies of your secondary modules in order to 
   decide how many loaders you need. 
   
   Now, we come to the question of transmitting data from a module to another one. Assume that
   mod2 needs to have access to the content of mod1. After mod1 is loaded, its content is 
   available. Say it is of type T. This is the right moment for starting the second loader
   with a start... function containing the datum of type T within its microcontext. It's 
   clear that if you design the actual type of mod2 cleverly, you can transmit the content
   of mod1 to mod2 when this start function is executed (by the second loader). 
   
   All this needs a lot of organization, and we plan to propose a program for automating
   (at least partially) it in the future. 
   
   
   
   *** (5) Having a module loaded only temporarily.
   
   It may be the case that you have a quite big module which is seldom used. In this case, 
   you don't want to have it permanently into memory. You can proceed as follows. 
   
   Create a dedicated adm loader for this module, and don't start it when your program
   starts. On the contrary, start it only when needed. When the module should no more be used, 
   put the value 'true' in the 'shutdown_loader' variable of this adm loader. The loaded module
   will be collected together with the adm loader itself. Of course, you can restart this adm loader
   at any time.  
   
   
   
   *** (6) Getting informations about loaded modules.  
   
   During execution of your adm loader you can get informations about its current state. 
   Such a state is a list of data of type 'SecondaryAdmInfo' defined in the generated file. 
   In order to get this informations you can use the function 'get_info' of 
   type: 
   
        AdmLoader_bubu -> List(SecondaryAdmInfo)
        
   There is one item in the list for each currently loaded module, containing the directory, 
   name, mask, file name, loading time, ... 
    
  
   --- That's all for the public part ! --------------------------------
   
  
   

read tools/basis.anubis


   A tool for a prettier presentation of the generated text. 

define String
   pad_to
     (
       Int n, 
       String s
     ) =
   with l = length(s), 
   if n < l
   then s
   else s + constant_string(n-l,' '). 


   Constructing a symbol from a file mask. 

define List(Word8)
   strip_name
     (
       List(Word8) name
     ) =
   if name is 
   {
     [ ] then [ ], 
     [h . t] then 
       if member(h,"*.") 
       then ['_' . strip_name(t)] 
       else [h . strip_name(t)]
   }.

define String
   strip_name
     (
       String name
     ) =
   implode(strip_name(explode(name))). 


   Checking if a secondary module has a mask as its name:
   
define Bool   
   is_mask
     (
       SecondaryAdm a
     ) =
   member('*',name(a)). 


   Constructing the output type for the 'utilities'. 

define String
   out_type
     (
       SecondaryAdm a
     ) =
   (if is_mask(a) then "List" else "Maybe")+"("+type(a)+")". 


   Declaring and defining the 'utilities'. 

define SecondaryAdm -> String 
   utility_decl
     (
       String       loader_name
     ) =
   (SecondaryAdm a) |-> 
   "public define "+pad_to(35,out_type(a))+" "+
      pad_to(20,strip_name(name(a)))+"(AdmLoader_"+loader_name+" a).". 

define SecondaryAdm -> String 
   utility_def
     (
       String       loader_name
     ) =
   (SecondaryAdm a) |-> 
   with n = strip_name(name(a)),
   if is_mask(a)
   then "public define "+out_type(a)+" "+n+"(AdmLoader_"+loader_name+" a) =\n"+
        "   map(content,*instances(info_"+n+"(a))).\n" 
   else "public define "+out_type(a)+" "+n+"(AdmLoader_"+loader_name+" a) =\n"+
        "   if map(content,*instances(info_"+n+"(a))) is\n"+
        "   {\n"+
        "     [ ] then failure,\n"+
        "     [h . _] then success(h)\n"+
        "   }.\n\n".   


   Formatting a component of type 'AdmLoader_...':

define String
   make_component
     (
       SecondaryAdm a
     ) =
   "     AdmLoaderInfo("+type(a)+")  info_"+strip_name(name(a)). 

   
define String
   make_vars
     (
       List(SecondaryAdm) l
     ) =
   if l is 
   {
     [ ] then "", 
     [h . t] then if h is adm(delay,dir,name,type) then 
       with n = strip_name(name), 
       "   with "+n+"_v = var((List(AdmLoaderInstance("+type+")))[]),\n"
       + make_vars(t)
   }.          

define String
   make_info_items
     (
       List(SecondaryAdm) l
     ) =
   if l is 
   {
     [ ] then "", 
     [h . t] then if h is adm(delay,dir,name,type) then 
       with n = strip_name(name), 
       "   with info_"+n+" = adm_info(\""+dir+"\",\""+name+"\",\n"+
       "                              start_"+n+",\n"+
       "                              stop_"+n+",\n"+
       "                              "+n+"_v),\n"+
       make_info_items(t)
   }.

define String
   make_delegates
     (
       List(SecondaryAdm) l
     ) =
   if l is 
   {
     [ ] then "", 
     [h . t] then if h is adm(delay,dir,mask,type) then 
       with n = strip_name(mask), 
       "   delegate adm_reload_loop(adm_info(\""+directory(h)+"\",\n"+
       "                                     \""+mask+"\",\n"+
       "                                     start_"+n+",\n"+
       "                                     stop_"+n+",\n"+
       "                                     "+n+"_v),\n"+
       "                            shutdown_loader,\n"+
       "                            warn,\n"+
       "                            "+to_decimal(delay)+"),\n"+
       make_delegates(t)
   }.     



define String
   make_start_stop_decl
     (
       SecondaryAdm a
     ) =
   "     "+pad_to(50,type(a)+" -> One")+"  start_"+strip_name(name(a))+",\n"+  
   "     "+pad_to(50,type(a)+" -> One")+"  stop_"+strip_name(name(a)). 
   
define String
   make_to_info
     (
       SecondaryAdm  a
     ) =   
   if a is adm(delay,dir,mask,type) then 
   "   to_info(\""+abs_to_decimal(delay)+"\",\""+
                   dir+"\",\""+
                   mask+"\",\""+
                   type+"\")(info_"+strip_name(mask)+"(a))". 
   

public define One
   make_adm_loader
     (
       String                 loader_name, 
       String                 target_path,
       List(String)           to_be_read, 
       List(SecondaryAdm)     l
     ) =
   if file(target_path,new) is 
   {
     failure then print("Cannot create file "+target_path+"\n")
     success(t) then with target = (WStream)weaken(t), 
       print(target,
       
       "\n\n   This file was automatically generated.\n"+
       "   See 'library/tools/adm_autoreload.anubis' for explanations.\n\n"+
       
       "read tools/basis.anubis\n"+
       "read tools/ISO-8601.anubis\n"+
       "read tools/adm_autoreload_tools.anubis\n"+
       concat(map((String n) |-> "read "+n ,to_be_read),"\n")+"\n\n"+

       "public type AdmLoader_"+loader_name+":...   (an opaque type)\n\n"+
       "   'start_adm_loader_"+loader_name+"' creates and starts your adm loader.\n"+
       "   The dynamic variable 'shutdown_loader' must contain 'false'.\n"+
       "   The adm loader will shutdown as soon as this variable receives\n"+
       "   the value 'true'.\n\n"+
       
       "public define AdmLoader_"+loader_name+"    start_adm_loader_"+loader_name+"\n"+
       "   (\n"+
       "     Var(Bool)                                           shutdown_loader,\n"+
       "     String -> One                                       warn,\n"+
       concat(map(make_start_stop_decl,l),",\n")+
       "\n   ).\n\n"+ 
       
       
       
       "   Below are the utilities available in your adm loader.\n"+
       "   They always give you the latest version of your modules.\n"+
       "   Hence, you must not keep their return values, but call them each\n"+
       "   time you need their value.\n\n"+
       concat(map(utility_decl(loader_name),l),"\n")+
       "\n\n"); 
       
       
       print(target,
       "   Getting informations on currently loaded adms:\n\n"+
       "public type SecondaryAdmInfo:\n"+
       "   adm(\n"+
       "     String       directory,\n"+
       "     String       file_mask,\n"+
       "     String       file_name,\n"+
       "     String       loading_time,\n"+
       "     String       type,\n"+
       "     String       delay           // (milliseconds)\n"+
       "      ).\n\n"+
       "public define List(SecondaryAdmInfo)     get_info    (AdmLoader_"+loader_name+" a).\n\n"
       );
       
       print(target,
       "\n   --- That's all for the public part ! -------------------------\n\n"+
 
       //tools+

       "\n\npublic type AdmLoader_"+loader_name+":\n"+
       "   loader(\n"+
       concat(map(make_component,l),",\n")+
       "\n        ).\n\n"+
       
       //make_reload_loops(l)+
       
       "public define AdmLoader_"+loader_name+"    start_adm_loader_"+loader_name+"\n"+
       "   (\n"+
       "     Var(Bool)                                           shutdown_loader,\n"+
       "     String -> One                                       warn,\n"+
       concat(map(make_start_stop_decl,l),",\n")+
       "\n   ) =\n"+ 
       make_vars(l)+
       make_info_items(l)+
       make_delegates(l)+
       "   loader(\n"+concat(map((SecondaryAdm a) |-> 
           "      info_"+strip_name(name(a)),
                               l),",\n")+").\n\n"+
      
       concat(map(utility_def(loader_name),l),"\n")); 
       
       print(target,
       "define AdmLoaderInfo($T) -> List(SecondaryAdmInfo) to_info(\n"+
       "   String delay, String dir, String mask, String type) =\n"+
       "   (AdmLoaderInfo($T) a) |-> map((AdmLoaderInstance($T) i) |->\n"+
       "        if i is instance(fn,lt,_) then\n"+
       "        adm(dir,mask,fn,_Int_to_ISO_8601_date(to_Int(lt)),type,delay),\n"+
       "     *instances(a))."+
       "\n\n"); 
       
       print(target,
       "public define List(SecondaryAdmInfo)     get_info     (AdmLoader_"+loader_name+" a) =\n"+
       concat(map(make_to_info,l),"+\n")+
       ".\n\n"
       );

       print(target,
       "\n   --- end ---\n\n"
            )
   }.