[Dev-luatex] .BEILMOPS or how I stopped worrying and love Open Source

[David Kastrup]

"Jonathan Sauer" <Jonathan.Sauer at silverstroke.com> writes:

> Hello,
>
>> >> when using lpeg an dmatching, keep in mind that \unexpanded has the
>
>> >> side effect of introducing spaces
>> >
>> > Oh. I used \unexpanded, because sometime someone on this mailing list 
>> > noted that it was faster than \detokenize.
>> 
>> Should make no difference, I guess.  \detokenize should 
>> introduce the same spaces, using the same algorithm.
>
> Well, currently it makes a difference, three actually:
>
> 1.	\unexpanded introduces "IMPOSSIBLE."

Hm.

> 2.	\unexpanded introduces spaces after control sequences.

Why wouldn't \detokenize do the same?  Wait, it does:

dak at lola:~$ etex
This is pdfTeXk, Version 3.141592-1.40.3 (Web2C 7.5.6)
 %&-line parsing enabled.
**\message{\detokenize{\a b}}
entering extended mode
\a b
*\message{\unexpanded{\a b}}
\a b
*\message{\detokenize{\:x}}
\:x
*\message{\unexpanded{\:x}}
\:x
*\end
No pages of output.
Transcript written on texput.log.

> 3.	\unexpanded ignores \par's, while \detokenize does not. So empty
> 	lines in \directlua are no problem with \unexpanded, but result
> 	in a (Lua) parse error when \detokenize is used.
>
> At least in the context of \directlua.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I guess that this may be the real reason.

> Also, in terms of speed: \unexpanded only marks all tokens as
> unexpandable, while \detokenize creates new character tokens. So the
> latter should be somewhat slower.

No question about that.

>> >> btw, if you change \unexpanded by \detokenize you get the desired 
>> >> result
>> >
>> > I know.
>> 
>> Oops.  Why would that be?
>
> Because of a bug? :-)

I should think so.  Not sure whether this behavioral difference is
intended, but it certainly feels wrong to me.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum