[Dev-luatex] Collected notes

[Hans Hagen]

Jonathan Sauer wrote:
> Hello,
> 
>>> -	When setting a \toks register in Lua using tex.toks['foo'] =
>>> 'bar' et.al., what catcodetable is used? From looking inside
>>> luatex.web, I would guess that, since str_toks is used, all 
>>> characters get catcode 12, except for spaces, which get catcode 10.
>> Correct. It is one of the first things I wrote, and it definately
>> needs updating. I now believe the toks array should accept and return
>> token list tables (instead of lua strings), and there should be helper
>> functions to_string() and to_tokenlist() for going back and forth.
>> That needs a bit of (not-yet-done) programming, but it is not hard at
>> all.
> 
> I'm wondering how this would affect performance. How costly is the
> conversion from a string to a table?

costly;

input -> texscanner -> internal tex structure

with callback or toks or ...:

input -> texscanner -> callback that gets token being table -> returns 
table (or not) -> internal tex structure

so, in the tokentable case there are two conversions (+ allocation and 
garbage collection for the token table) which is way slower

> And what about tex.print? Since it creates/writes tokens as well, will
> it be changed to accept token list tables, too? (at any rate, I think
> both should accept the same parameter type)

no, that will be string based; pushing strings into texs scanner is 
currently pretty efficient and te need to convert them to tokens first 
would make the interface clumsy; so tex.tprint (token print) would be a 
natural candidate then (one argument, a table of tokens)

in practice tex.sprint is used more often than pushing token tables into 
tex

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------