[Dev-luatex] Lua states and \dump

[Hans Hagen]

Jonathan Sauer wrote:

> Well, I experimented a bit and am now a lot wiser. Also, I have
> confirmation on what I thought all along: My computer's internal
> harddisk is much too slow:
> 
> How to measure the performance of virtual memory:
> 
> \directlua0{lua.bytecode[100000000] = function() end}

works ok here (but i have a 4 gig machine); technically it should only 
allocate one lua state; on the other hand, it may be that a non sparse 
array is used for the register housekeeping

> On a more serious matter, though, there is a buffer overflow in
> llualib.c, function set_bytecode: If `sizeof(bytecode)*(k+1)' is
> larger than UINT_MAX, a numeric overflow occurs and not enough memory
> is allocated in line 162. When initializing the newly allocated
> bytecode registers afterwards, random memory is overwritten.
> 
> Example (without exploit, of course):
> 
> Register number = 1000000000:

such overruns happen also when you do wild things with lua code, tex 
does not manage that memory

another overflow can be in the piping data to tex (tex.print) .. if you 
collect 2 gig data there you may also run into problems

> (I killed the process in the latter experiment, as it still consumed
> too much memory [about 1.9GB]. But: Even though the memory
> requirements have been higher than in the first example, in this case
> the allocation did not fail, as the overflow created a smaller
> allocation, `only' 1935228944 bytes)
> 
> 
> Register number = 3000000000:

test \directlua0{lua.bytecode[1024*1024*1024*1024] = function() end}

test \directlua0{for i=1,100000000 do lua.bytecode[i] = function() end end }

seems to work ok here but

test \directlua0{lua.bytecode[3000000000] = function() end}

test \directlua0{for i=1,3000000000 do lua.bytecode[i] = function() end 
end }

report a problem with a negavtive value, so it may be that there is a 
problem there (not sure if taco tests the max value)

! LuaTeX error negative values not allowed.
l.7 ...{lua.bytecode[3000000000] = function() end}

> IMO it would be sensible to limit the number of bytecode registers to
> UINT_MAX/sizeof(bytecode) or -- to be platform-independent in the
> light of 64 bit processors[2] -- (2^32-1)/sizeof(bytecode).

such a limitation is not that meaningful; one can have 1 milion bytecode 
functions savely but 10 using large datastructures and bombing; there is 
no control over the lua end of the game; also, when using much data, in 
practice the garbage collectors will bring down your system (so slow 
that one will abort the job); luatex kind of assumes modern memory 
management and machines with memort in the gig range

> [3] I think, this is the result of the sig-handler LuaTeX installs
> which displays an error message. But it seems that this message has
> been overwritten as well.

error messages and cathing errors with proper messages is on the agenda 
for next year but segfaults and crashes indeed need to be cached

(i can at least imagine a practical limit of 64K bytecode registers)

> Of course, all of this is void, if the current bytecode register
> implementation is only temporary.

not that i know -)

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------