\chapter{Data model}
\label{sec:data-model}

Very many difficulties users have with font editing (both scripted and
interactive) come from an incomplete understanding of the data model
involved: what entities exist in a font and a font editor and what
relationships those entities have with each other.  The distinction between
glyphs and characters seems to be an especially frequent cause of confusion. 
This chapter attempts to describe FontAnvil's data model in a way that will
be useful to script programmers.

\section{Fonts in memory}

Because PE script was originally designed for controlling a GUI font
editor, it treats fonts as documents to be opened and closed, much as
a GUI editor might.

At any given time there exists a global set of fonts that are \emph{open}. 
These are stored in RAM.  Open fonts are associated with filenames, even if
the filenames do not actually exist on disk; the interpreter will assign
temporary filenames (usually similar to ``\texttt{Untitled1.sfd}'') to fonts
that were created in memory and not loaded from files.  The filenames should
be unique (no more than one open font sharing a filename), and it's not easy
to create a situation where they are non-unique, but I suspect that having
distinct open fonts with duplicate filenames may be technically possible and
likely to trigger bugs if attempted.

The set of open fonts is technically a sequence (with a specific order), not
an unordered set, and the order is visible to scripts through the
\verb|$firstfont| and \verb|$nextfont| built-in variables, but this fact is
seldom important.

At most one of the open fonts may be the \emph{current font}.  This state is
global.  Most font-editing operations implicitly apply to the current font. 
The \texttt{Open()} built-in function sets the current font, but its exact
behaviour is context-sensitive.  If the specified filename is already an
open font, then \texttt{Open()} just sets the current font to that one.  If
the specified filename is \emph{not} already an open font, then
\texttt{Open()} loads it from disk, causing it to become an open font, before
setting the current font to that one.

When the interepreter starts up, the set of open fonts is empty and there is
no current font.  In this state, any operation that implicitly refers to the
current font will fail; scripts can only use a small subset of the language
to create or open a font and make it current.  The \texttt{Close()} built-in
function removes the current font from the set of open fonts (without saving
it to disk---that must be done as a separate operation if saving is desired)
and places the interpreter back into the state of having no current font. 

The \FFdiff existence of a no-font state is a difference between PE script
interpreters (both FontAnvil and FontForge when run as a command-line
interpreter) and the FontForge GUI.  The GUI insists on always having at
least one open font and always having a current font, enforcing this rule by
automatically loading fonts from earlier editing sessions, automatically
creating a new empty font if the load fails, choosing another open font to
be current when one is closed, and terminating the program when the last
font is closed.

\section{Glyphs and slots}

Here are two pictures of Don Quixote.\footnote{Left: Gustave
Dor\'{e}, 1863.  Right: Honor\'{e} Daumier, 1868.}

\begin{center}
\includegraphics[width=1.3in]{quixote-dore.jpg}\qquad
\includegraphics[width=1.3in]{quixote-daumier.jpg}
\end{center}

The pictures are different, but they are pictures of the same fictional
character.  In the same way, we can have several pictures of the same
character in a writing system.

\begin{center}
\scalebox{4}{a}\qquad
\scalebox{4}{\textit{a}}\qquad
\scalebox{4}{\texttt{a}}
\end{center}

What these three ``a''s share is the \emph{character}; what they do not
share is the \emph{glyph}.  Fonts contain collections of glyphs, concrete
things, which are pictures of characters, abstract things.  It is often the
case that in any given font, each glyph corresponds to exactly one character
and vice versa; but there are many important exceptions to that rule.  To
understand how FontAnvil processes fonts it is important to bear in mind
that fonts are collections of glyphs, and glyphs are not the same thing as
characters despite being closely connected with characters.

This glyph (the classic ff-ligature) is not associated with one character,
but with a sequence of two characters:

\begin{center}
\scalebox{5}{ff}
\end{center}

There is no double-f character, as a separate abstract entity distinct from
just two ordinary ``f''s in a row, in the English language.  There is no
standard character code for such a thing.\footnote{Actually, there is one in
Unicode, but you're not supposed to really use it; the details of that are
beyond the scope of this discussion.} Nonetheless a font for high-quality
typesetting of English must contain a glyph for this double-f entity
that is not quite a character.  Despite such exceptions, one glyph per
character is true most of the time in English.  In some other languages, the
conceit of glyph-character equivalence breaks down entirely.  In Arabic, for
example, many characters have four or more visual forms requiring
separate glyphs in a font, depending on how they connect to neighbouring
characters.

FontAnvil represents a font in memory as including a zero-based array of
\emph{glyph slots}.  The array is variable-sized, but it is a true array
data structure, not a list: all slots from index 0 up to one less than the
number of slots exist as long as the in-memory font does.  Creating a new
glyph means overwriting the possibly-blank previous contents of some slot. 
Destroying a glyph means filling the slot with a blank glyph, but the slot
continues to exist.  Changing the number of slots can only be done by
increasing or decreasing the length of the array, and encodings (described
in the next section) may constrain the number of slots.

Glyph slots in a font always have \emph{glyph numbers} which are their
indices in the array.  Glyphs in a font cannot meaningfully be said to be in
any specific order other than the order determined by the array indices. 
Every glyph has a number, and every number has a glyph slot.  No two glyphs
can have the same slot; two slots may contain identical glyphs, or even a
``reference'' from one to the other, but the two slots' contents will not
truly be the same entity.  A glyph slot might be \emph{blank}, that is
devoid of outlines and other data, and people usually think of such slots as
not really being glyphs.  But some attributes of a glyph slot (such as its
name) are required to always have non-null values, even on blank slots.

\begin{framed}
FontAnvil's in-memory glyph slots can be blank but never truly empty. 
However, most on-disk file formats \emph{do} have a concept of a glyph
failing to exist at all.  Blank slots are usually not written when saving
from memory to disk, and loading a file from disk to memory that does not
fill all slots will usually leave the others blank.
\end{framed}

Every open font has a \emph{selection}, which specifies a set of the glyph
slots (more about glyph slots in the next section).  Many editing operations
implicitly operate on the selection of the current font.  Although every
open font has a selection, usually only the selection of the current font is
relevant.  Open fonts retain their selections through changes in which open
font is current.

The selection is actually a sequence, not a set, of glyph slots.  That means
the slots in it can be selected in a specific order.  This distinction is
relevant in the FontForge GUI, where you can select several glyphs in a
specific order, open a ``Metrics'' window, and have them come up in the
order you selected them.  However, each glyph slot can appear at most once
in the selection, and the order of the selection is seldom if ever
observable or controllable from the scripting language.  It is usually
better to think of it as a set with no specific order, not as a sequence.

The selection is applied at the level of glyph slots.  It is perfectly
possible for the selection to include blank glyph slots, because it is
defined as a set of slots, not a set of non-blank glyphs.  Nonetheless, one
often only cares about the non-blank glyphs, and some commands for
manipulating the selection will automatically limit themselves to slots that
are not blank.

Glyph slot numbers \emph{may or may not} be closely connected to Unicode,
ASCII, or other character codes, depending on issues discussed in the next
section.  It is important to be aware that glyph slot numbers are not the
same type of entity as character codes, despite sometimes having equal
numerical values.

\section{Encodings}

Fonts do not exist solely to be edited with FontAnvil.  To be useful, a font
must eventually be saved in some format understandable by a word processor
or similar application; and the external software must be able to associate
the glyphs in the font with the characters in text that will be typeset
using the font.  There will necessarily exist some \emph{code} that
associates numbers called \emph{code points} with the different characters
that might appear in text, and a font file must explicitly or implicitly
describe which code points go with which glyphs.  It is worth emphasizing
that code points refer to characters, \emph{not glyphs}.

The ASCII code is familiar to many computer users, especially in the
English-speaking world, but is inadequate for languages other than English. 
Today, nearly everybody uses a code called Unicode.  Unicode's code points
coincide with ASCII for all the characters covered by ASCII; but it also
covers many thousands of other characters.  It is supposed to be a universal
code for all text in all human languages.  But Unicode did not always exist and its use was not always
universal.  Most common font formats are designed to also accomodate
character encoding schemes predating Unicode or simply other than Unicode. 
Frequently, a font file will include translation mappings for several
different codes, in the hope that software using the font can find its own
preferred code among the choices.  There needs to be a way to translate code
points (which identify characters) into glyph numbers (which identify
glyphs), even in cases like ligatures and variant glyphs where this
translation is more complicated than a simple one-to-one lookup.

FontAnvil includes several mechanisms for addressing these issues, but the
most significant is that of the \emph{encoding}.  The encoding is a
property of each font (global to the font) and describes a set of
assumptions and constraints about the relationships between code points and
glyph numbers.

\begin{framed}
Every glyph slot in a font always has a glyph number, no two slots can have
the same number, and the set of glyph numbers that exist is always the set
of integers from 0 up to one less than the number of glyphs in the font. 
Every code point designates at most one glyph slot.  But a glyph slot might
have zero, one, or more than one code point; a code point might have no
glyph slot; and the set of code points that exist might not be a simple
interval of the integers.  Nonetheless, the font's encoding may trigger the
enforcement of constraints that make the code point situation less
complicated.
\end{framed}

Most of the possible values of the encoding field are associated with
standardized character codes defined by external organizations.  Each code
defines a meaning for a range of code points from 0 up to some number that
depends on the code.  \emph{When the encoding is associated with an
externally standardized code other than Unicode}, FontAnvil enforces the
following constraints:
\begin{itemize}
\item There must be at least as many glyph slots as there are code points in
the code.
\item Glyph slots in the range 0 through one less than the number of code
points in the code correspond one-to-one with the code points.
\item Any glyph slots with greater indices have no code points.
\end{itemize}

The case of \emph{unencoded glyphs}, those in glyph slots beyond the end of
the code point range specified by the encoding, is important.  Glyphs like
the ff-ligature mentioned earlier, or the alternate forms of letters in a
script like Arabic, usually fall into this category.  When a word processor
typesets text using a font, it starts out by translating the code points
into glyphs one-for-one according to the basic code points of the glyph. 
The result of that translation cannot contain unencoded glyphs.  But the
basic one-for-one translation of code points to glyphs is only a starting
point.  Further processes can merge and split glyphs so that more than one
character can be typeset with one glyph, one character can be typeset with
more than one glyph, and which glyph goes with which character can be
different in different contexts.  These further processes can bring
unencoded glyphs into play.  The encoding does not specify the number of
unencoded glyph slots that may exist after the range of encoded glyphs.  The
unencoded glyph slots may be manipulated by built-in functions like
\texttt{SetCharCnt()}; encoded glyph slots may not be added or removed.

Each glyph slot has a property called the \emph{Unicode number}.  This is a
code point (in the code that is named Unicode), but I am going to call the
number in this field just a ``number'' to distinguish it from the code
points that exist in non-Unicode encodings.  When the encoding is one of the
standardized non-Unicode encodings, the constraint is enforced that the
Unicode number must be the correct translation (using the iconv library) of
the glyph number for encoded glyphs, or the null value of -1 for unencoded
glyphs.  For example, if the font's encoding is KOI8-R (commonly used for
Russian text), then glyph slot number 241 is for the letter ``ya,'' which
looks like a backwards R.  FontAnvil will enforce the constraint that this
slot's Unicode number is 0x042F, which is the Unicode code point for that
letter.  ``The constraint is enforced'' means that if you try to change the
value of the Unicode number field, the font's encoding will be immediately
changed to Custom.  Editing the Unicode numbers is not compatible with
keeping the encoding and its fixed mapping.

But not every value for the font's global encoding field is associated with
an external standard other than Unicode.  When the font's encoding field
refers to some form of Unicode, or does not refer to an external standard,
then additional special considerations apply; and in fact, these special
cases are the most popular and useful values for the encoding field.

When the encoding is set to Custom, few encoding-related constraints are
enforced.  There may be any number of glyph slots.  Any slot may have a
Unicode number, or not, and there is not necessarily any relationship
between the Unicode numbers and the glyph slots.

There are two Unicode encoding options, Unicode (BMP) and Unicode (Full). 
These each behave more or less like the non-Unicode standardized encodings. 
One difference is that it appears sometimes possible to set the Unicode
number of a glyph slot such that it does not match its glyph number.  This
may be a bug.  FIXME investigate further - this description may possibly be
simplified if Unicode and non-Unicode turn out to really behave the same.

FIXME investigate and document the ``Original'' encoding option.

Glyph slots have names.  All glyph slots have non-empty names, including
blank slots, and all names must be unique within a font.  The names are
sometimes automatically assigned and may also be manipulated by script
commands; but constraints (including the requirement for uniqueness) will be
enforced on such manipulation.

\begin{framed}
People who think they want to edit glyph slot names are often wrong.
\end{framed}

FIXME document name lists

\section{The \texttt{.notdef} glyph}

FIXME

\section{The clipboard}

There is a global entity called the \emph{clipboard}, which holds glyph data
of the kind that might be stored in glyph slots, such as outlines, anchors,
and references.  The clipboard is like a font in that it can store a bunch
of slots' worth of data, in a definite order, but the clipboard is unlike a
font in that the slots do not have meaningful numbers and it does not store
slot attributes other than glyph data, such as slot names and code points. 

The usual way of using the clipboard is somewhat like using the clipboard in
any common GUI document editor: select some slots, do a cut or copy
operation, select some other slots (even in a different font), and do a
paste operation.  Here is typical code to copy the uppercase ASCII alphabet
from an existing font into a new font (leaving many things in the new font
empty or default, which may cause problems later):

\begin{verbatim}
Open("font1.sfd");
Select('A','Z');
Copy();
New();
Select('A','Z');
Paste();
Save("font2.sfd");
\end{verbatim}

One thing to be aware of is that \texttt{Paste()} always writes into the
selection, and so you must create a nonempty selection for \texttt{Paste()}
to be meaningful.  This differs from a word processor that can ``insert''
text; FontAnvil treats a font as fixed framework of glyph slots that can
only be changed by overwriting.  Inserting or deleting in the middle, in a
way that changes the number of slots that exist, would disrupt the framework
of the encoding and is rarely, if ever, what you really want.

\begin{framed}
A glyph slot's name is associated with the glyph slot, not with the glyph
data stored in the slot.  The slot name will not move with the glyph data
when the glyph data is cut and pasted into a new slot.  Unicode code points,
and any other encoding numbers, are also parts of the glyph slot and will
not move with cut and pasted glyph data.
\end{framed}

\section{Look-up tables}

FIXME

\section{Syntax, semantics, round trips, and reproducibility}

It is a very common complaint from users of font editors that someone loaded
a file, saved it without making any other changes, and ``the result was not
the same.''  Such complaints are often filed as bugs in the FontForge bug
tracker.  Dealing with them is not so easy as it might appear on the
surface, because what it means for two files to be ``the same'' is not so
simple as whether they contain the same sequence of bytes.  This section is
intended to clarify some of the relevant issues as they relate to FontAnvil,
and let users know what are and are not reasonable expectations about the
system's behaviour.

Consider the two ASCII strings ``\texttt{1.5}'' and
``\texttt{1.50000000000}''; are those the same?  As strings, they are not. 
But if these strings are representations of a real number, they represent
the same real number.  So if an editor loads one and saves the other, the
bytes in the file will have changed but the number will not have changed. 
We can say that the \emph{syntax} has changed but the \emph{semantics} have
not.  These are terms borrowed from the field of linguistics, where they
describe different levels of abstraction in the study of a language: syntax
refers to questions like what sequences of words are grammatically valid
sentences, while semantics refers to the meanings of words.

Note that if ASCII strings represent floating point numbers in low machine
precision, we might even have ``\texttt{1.50000000000}'' and
``\texttt{1.49999999999}'' referring to the same number.  The ``number''
that matters is the machine-precision binary number; two strings that both
result in the same machine-precision binary number are two strings with the
same meaning even if a human translating them instead to exact real numbers
would see a difference.

When FontAnvil, or any other font editor, loads a file from disk, the file
is converted into an in-memory representation that is designed to store the
semantics of the font.  When it saves the file, that in-memory
representation is converted into the disk file format.  As a result of this
process, in general it is reasonable to expect that loading and saving a
file will preserve the semantics of the data in the file, but in general
\emph{it is not reasonable to expect aspects of syntax beyond semantics to
be preserved.}

The result of a round-trip through any editor will in general be a file with
the same semantics, but not necessarily the same syntax.  In general that
statement is true of all software that edits complicated file formats,
including word processors and image editors as well as all font editors, not
only FontAnvil.  For reasons unknown to me, users of font editors seem
especially inclined to be surprised by the loss of non-semantic information
and to report it as a claimed ``bug,'' even though this is the normal,
expected, and pretty much inevitable behaviour of most computer
software.

Further confusion may ensue when a file is round-tripped through more than
one on-disk format, because different formats may define different
semantics, sometimes at the ontological level; that is, different formats
may be based on different models of what are the important concepts in the
problem domain.  For instance, some font formats make heavy use of
``references,'' which allow part of one glyph to automatically incorporate
another glyph.  Other font formats just do not contain references at all. 
Some font formats use quadratic splines to represent curves, while others
use cubic splines, and each of these is only approximately capable of
representing data from the other; a loss of precision necessarily occurs
when converting between them.  In general, information from one
format may need to be changed in irreversible semantic ways or even
discarded in order to convert a font to another format.  Converting back and
getting even the same semantics, let alone the same syntax, is not always a
reasonable expectation.  This issue is somewhat analogous to what occurs
when translating text successively through two or more very different human
languages.

The FontAnvil in-memory format attempts to be able to represent all the
semantic information in all the on-disk formats that FontAnvil supports; but
it still represents semantics only.  Since FontAnvil descends from FontForge
which descends from PfaEdit which was originally an editor for Postscript
only, the FontAnvil in-memory format may have some bias toward a Postscript
data model.  It nonetheless is intended to cover the semantics of all the
formats FontAnvil supports.

The FontAnvil SFD ``native'' format is a special on-disk format intended to
represent the semantics of the in-memory format as accurately as possible;
so, by design, it should be possible to round-trip between a font in memory
and a font on disk in SFD format, with no loss of semantics.  However, it is
still possible to find syntactic distinctions in SFD files that will not be
preserved by loading and saving.  For instance, one can mess around with
external software to re-arrange the sequence of glyphs in an on-disk SFD
file, only to have one's tampering automatically undone the next time the
file is loaded and saved because the order of entries in the file is not
semantic and all that really counts is the glyph numbers stored in the
appropriate fields.

From this general picture there emerge several points that describe how
FontAnvil relates to the syntax and semantics of font files.

\begin{itemize}
\item FontAnvil attempts to preserve semantics of font files through
a round trip of loading from and saving to any one supported foreign format. 
\item FontAnvil attempts to preserve semantics of font files through 
a round trip of loading a foreign format, saving and loading the data in SFD
format, and then saving back to the same foreign format.
\item When loading from one foreign format and saving to another, FontAnvil
attempts to preserve whatever semantics make sense in both formats; but a
round trip from one foreign format to another and back may not be lossless.
\item FontAnvil does not attempt to preserve syntax of any file format
beyond the semantics of the font inside, and in general it should be
expected that it usually \emph{will} change syntax.
\item FontAnvil does not attempt to preserve SFD file semantics specific to
features FontForge includes and FontAnvil does not; for instance, pickled
Python objects.
\item FontAnvil does not support SFD-like files written or
edited by software other than FontAnvil itself, except versions of FontForge
close to the Git repository version current as of March 2014.
\item FontAnvil does not support FontForge SFDir (SFD split into a directory
instead of a single file) format.
\end{itemize}

The Debian Project has recently embarked on a misguided project to demand
``reproducibility'' of everything in their world, which they define as
having the ability to compile software many times under different
circumstances and get byte for byte syntactically identical output.  I call
this \emph{extreme reproducibility} to allow distinguishing it from weaker
properties that might also be called reproducibility, and to emphasize what
an outlandish and cumbersome form this one actually is.  Several purported
reasons for extreme reproducibility to be desirable are put forward by the
project.  The claim that extreme reproducibility would be nice for security
is true, because multiple parties can independently compile something, check
that they got the same result, and thus be sure that they are all either
uncompromised or compromised in the same way.  But not everything that would
be nice to have is worth what it costs.

Absent rigorous audits of everything else that can go wrong, an extreme
reproducibility demand is far from a guarantee of security.  Demanding
extreme reproducibility precludes embedding metadata about the circumstances
of a build in the built file; as soon as there's so much as a timestamp
inside a file, byte for byte comparisons break.  Metadata is especially
important and useful in fonts and typesetting; and third-party file formats
in this application domain often have specific requirements for timestamps
and even per-save random ID numbers to be included in the files, making it
difficult to obey the file format definitions while also having extreme
reproducibility.

Randomized algorithms, and computations in which multiple threads work
simultaneously, may often generate differing but in some sense close enough
results when run more than once; anything of that nature is forbidden by
extreme reproducibility.  Cryptographic signature algorithms often require
the opposite of extreme reproducibility, with one-time nonce values that (on
pain of destroying the security guarantees) must \emph{never} be the same
from one run to the next; any part of a build process that involves such a
signature must be carefully segregated from any parts that are attempting to
guarantee extreme reproducibility.

With all that in mind, FontAnvil rejects the extreme form of
reproducibility, but embraces the weaker common-sense reproducibility
principle that running the same process more than once should produce
functionally identical results.

\begin{itemize}
\item FontAnvil scripts are intended to be \emph{semantically} reproducible:
running the same script on the same inputs with the same FontAnvil binary
should produce output with the same meaning.
\item It is not the policy of the FontAnvil project to make any attempt
at \emph{syntactic} reproducibility, it is not reasonable to expect
output to be byte for byte identical from one run to another, and in
general, the contrary should be expected.
\end{itemize}
