[Aleph] OpenType-to-TeX and other musings...

[gabor.bella at enst-bretagne.fr]

> Consider the last phrase, "and will remain so for some years yet". My
> point: with a concerted effort, TeX could conceivable beat both MS and
> Adobe to have the best support on every OS for their own format,
> OpenTypeFonts! The otp mechanism could provide the key. Maybe it needs
> to
> be gutted and rebuilt, but that is the key, I think.

Hi,

Before attacking the problem, consider the following points:

1) The majority of the extra "knowledge" in OpenType fonts is stored in its
GSUB (glyph substitution) and GPOS (glyph positioning) tables. For scripts
such as Arabic, Hebrew, or the Indic scripts, as you know, both of them
are essential. However, Omega's/Aleph's OTPs work on the token level and
are basically designed to do character replacements. (TeX/Omega/Aleph do
not distinguish between characters and glyphs, more on this later.) OTPs
therefore are not directly able to do glyph positioning at all, due to the
early stage in which they intervene (they have no access to nodes) but also
simply because they were rather designed to do substitutions. Of course,
one could imagine OTPs inserting \raise and \kern primitives but we all
know how these primitives interrupt text flow, breaking hyphenation
algorithms or searching in the output PDF. That would be a \emph{bad}
solution.

2) OpenType's chaining contextual rules are slightly incompatible with
OTPs. The problem is that chaining contextual lookups rely on
"backtracking" and "looking ahead" in the glyph string. While the latter
has an equivalent in OTPs (pushback), the former does NOT. It can be
envisaged to encode backtracking information into states---theoretically,
on an abstract level, OTPs are equivalent in power to OpenType
engines---but this solution is not straightforward and not practical.

3) OpenType fonts, as Taco has already mentioned, have a declarative
nature: they tell you what to do but not how to do it. Example: they will
be able to give you the initial, medial, final, or isolated form of your
Arabic "lam" but you will have to do the contextual analysis by yourself,
OpenType does not help in this at all. Of course, one can rather easily
write/generate OTPs that do this work but this means that just
"converting" OpenType tables into OTPs will rarely be enough. Additional
work, and a LOT of work, will need to be done to be able to fully support
OpenType.

4) There is an even deeper incompatibility between Omega/Aleph and
OpenType's philosophy. In fact, OpenType relies on a very clear (and
indeed justified) notion of separating characters and glyphs. The input
text is a character string, it is then converted into a parallel glyph
string so that glyph positioning and substitution are done entirely on
glyphs, not on characters. Omega, on the other hand, fails to make this
distinction. Original character codes (Unicode or other) are eventually
\emph{replaced} by glyph codes from which point there is no turning back,
i.e., there is no way to recover original character information. Let's
suppose you transformed your text with OTPs, you managed to do contextual
substitution and even positioning, everything is fine. Now comes paragraph
building and you need hyphenation. How are you going to do it if you do not
have your original characters anymore? Font-specific hyphenation patterns?
Yuck!

In fact, me and Yannis have been thinking about implementing OpenType
support into Omega since a long time. We have already come up with
temporary solutions that work to a certain degree (GSUB using OTPs, just
like you described) but these solutions are not acceptable on the long
run.  After having realised the problems I described above, we concluded
that full OpenType support is only possible with a drastical rewrite of
Omega itself, a serious undertaking---but it will be worth the effort. I
and my colleague (Atif Gulzar, atif.gulzar at gmail.com), under the
supervision of Yannis, have already started working in this direction. Of
course, our work is open and we will let you know about how we are
progressing. And of course, anyone willing to help us work on OpenType
support or on Omega in general is more than welcome. Just contact anyone
of us if you need more information.

Cheers,
Gábor