[Dev-luatex] Snapshot 20060901

[Taco Hoekwater]

Hi all,

Yesterday's snapshot is in need of some documentation, so here it is.
All changes are related to lua this time, and most are quite unstable
/ experimental. But first the things that are stabel/fixed bugs:

* A VF loading bug that turned up in some of Hans' fonts
   has been fixed

* A small series of bounds checking fixes to \ocplist has been
   added to prevent the system from crashing due to array indexes
   running out of bounds.

* The Lua file searching paths are now fixed. The search path for lua
   script files now contains the following items (tried in order)
   1. the local directory:
        ./?.lua
     (for document-specific files)
   2. the items from the expansion of kpathsea's $TEXMFSCRIPTS variable,
     but only the parts containing 'lua' as a subpath:
        $TEXMFSCRIPTS<lua>/?.lua
        $TEXMFSCRIPTS<lua>/?/init.lua
      (for format-specific files)
   3. the $SELFAUTOPARENT sibling directory named 'lib'.
        $SELFAUTOPARENT/lib/lua/5.1/?.lua
        $SELFAUTOPARENT/lib/lua/5.1/?/init.lua
      (for files that are not related to tex)

   The search path for dynamic libraries has only
   1. the local directory:
        ./?.so
     (for document-specific files)
   2. and the $SELFAUTOPARENT sibling directory named 'lib'.
        $SELFAUTOPARENT/lib/lua/5.1/?.so

   (of course the extension is .dll on windows, but .dlls do not
    work at the moment so it will not do you much good)

* There are two functions available within a new table called
   texio:
	texio.write (luastring)
	texio.write_nl (luastring)
   both write the luastring to the same location(s) TeX writes
   its stuff. So if \batchmode is on, it writes only to the
   log, inside a \write, it prints to the current write file,
   etc.

   A read|write interface to TeX's "file selector" will
   follow shortly.

* At startup, luatex searches for a script named
	startup.lua
   in the path list I explained above. If such a file exists, it is
   loaded.

   This happens right before the first input file needs to be opened
   (that is after format loading, but before any \everyjob tokens).

   From within the script, you can check the value of

	tex.formatname

   that is the 'format identification' as used by TEX. When the variable
   is equal to nil, luatex in in 'initex' mode, otherwise it will be
   something like: " (format=plain 2006.9.1)"



Now for the experimental portion: callbacks. Here is what I have done
so far:

* The main reason for wanting startup.lua is file (input) re-encoding.
   For this purpose, it is now possible to set up a callback for
   luatex to execute.

   If you attach a Lua function to

	texio.input_line

   then from the next input line onwards, luatex will run that
   function whenever it needs a new input line from a text file.

   Your function will receive a file handle as argument, and
   should return either a string or nil (with nil signalling that
   the end of file has occurred).

   The trivial case is simply this:

     function reader (f)
         return f:read()
     end
     texio.input_line = reader


   Warning: The implementation is not totally finished yet. For the
   moment the file handle ("f" in the example) is a normal lua file,
   with a simple but important restriction: you cannot alter its value.
   You cannot f:close() it, or assign it a different value. luatex
   will eventually close the file itself.

   The restriction is a side-effect of a synchronisation problem with
   the lua garbage collector. Because of this, it also was necessary
   to turn off the automatic file closing code for normal lua io
   files (In other words: you have to close yourself all the files
   you opened yourself, and you should not close any files you did
   not open yourself).

   In the near future, "f" will become a special 'texio' file object
   and the needed functionality from the normal io library will be
   reimplemented. Along with that change, there will also be a callback
   to open (i.e. find) files, and a simple interface to the compiled-in
   kpathsea to use within that callback.



The experimental bit is not in the i/o properties of the "reader"
function: the "file handle in, line out" is very unlikely to change.
It is the interfacing from luatex to the callbacks that is the
experimental thing.

At this moment, I have used a rather stupid system based on a
name lookup: luatex searches for the "input_line" key in the
"texio" table in the lua interpreter defined by the value of
the register \luacallback (initially 0). If that key exists and
its value is a function, then it is used, otherwise luatex sticks
to the normal built-in input_ln() function.

This is definately not the optimal solution, but it happens to
be one of the simplest approaches possible (which is why i did
it: it gives something tanglible to think about).


A different (and much nicer) way would be explicit registration
of callbacks, say:

    success = callback.register("input_line",reader)

Some of the advantages of that approach: it would be a bit faster
to do the actual callback; no need for the strange \luacallback
register; and conflicts between various 'reader' functions
would be easier to manage/spot.

For future callbacks that can have multiple values (like the
ocp replacements) this syntax can easily extended to something
along these lines:

      local flist = callback.get("ocplist")
       -- mutate flist here --
      callback.register("ocplist",flist)


And the assignment could even adhere to TeX's grouping rules?


But there are quite a few other ways of doing this. Any thoughts
on this rather fundamental part of luatex (and especially syntax
proposals) are very, very welcome.

Cheers, Taco

-------------------

Downloading and installation details:

If you go to

     https://foundry.supelec.fr/frs/?group_id=10

you will see that there are two released files:

* luatex-snapshot-20060901.tar.gz
     This is the source tree.

* luatex-snapshot-20060901-win32.zip
     A cross-compiled (mingw) windows binary. This is a web2c
     based binary, so it needs a texmf.cnf file (It will NOT
     work if you have only miktex installed).

     This executable cannot run dynamically loaded lua dlls. Perhaps
     that can be fixed but I do not know how.