Unicode sequences, @typeface, and the font/typeface distinction

A run of Unicode code points now needs one pair of carets, not one pair
each: ^9053 53EF 9053^ renders the characters adjacently, and a single
character inside such a run stands for itself.  An invalid code point is
an error naming the position that wrote it, rather than an apology
rendered into the finished document; noncharacters are refused alongside
surrogates; and a code point that lands on a Klammertext special is safe
through the re-reads that assemble a document.  The character tables now
produce the precomposed form, so a diacritic written as ^a" and one typed
directly are the same text to a search or a copy-paste.

"kdesc -u" describes how to write a Unicode character, with tables of
glyphs beside the codes that produce them.  It is new and not yet
complete: it predates the sequence form and the typeface klammer below.

A typeface and a style are no longer both called "font".  A typeface is a
family resolved by name; a style code is the one letter naming a type
style.  So @document takes :serif_typeface and its siblings, a table takes
:style, captions take :caption_style and :caption_size, the uniform scale
is :text_scale, and the command is "kdesc --typeface".  This renames
arguments a document may already use; an old name is a located error
rather than a silent omission.

@typeface sets a passage in another typeface for the length of that
passage:

    @typeface Noto Sans Egyptian Hieroglyphs | ^132AA 133CF^ :scale 1.5 @

Writing the klammer is the whole declaration -- the document resolves the
typeface, embeds it in what it produces, and carries its license along.
Two typefaces for scripts no text font covers now ship with Klammertext:
Noto Sans Egyptian Hieroglyphs and Noto Sans Cuneiform.  Every typeface in
the distribution now carries its license file.

(from dev 61e598698891)

sks/document/document.k

@@ -40,10 +40,10 @@
 :include_sks_js.bool true
 :include_fonts.bool true
 
-:serif_font Crimson Pro
-:sans_font Open Sans
-:mono_font Inconsolata
-:font_scale 1.0
+:serif_typeface Crimson Pro
+:sans_typeface Open Sans
+:mono_typeface Inconsolata
+:text_scale 1.0
 
 :landscape.bool.tex false
 :paper_size..tex a4paper

sks/document/document_class.cpp

@@ -31,6 +31,10 @@ Document_class::Document_class(Machine& machine) : Klammer_base(machine)
     // show("document in main SKS");
     m_machine = machine;
     sloc = get("K_loc");
+    // The klammer's own position, so an argument error points at the
+    // @document application rather than at document_class.cpp: a
+    // default-constructed Locator captures the C++ source location.
+    loc = Locator::from_string(sloc);
     m_title = get("title");
     m_subtitle = get("subtitle");
     m_page_title = get("page_title");
@@ -77,17 +81,17 @@ Document_class::Document_class(Machine& machine) : Klammer_base(machine)
             s = s.substr(1, s.size() - 2);
         return s;
     };
-    m_serif_font = strip_quotes(get("serif_font"));
-    m_sans_font = strip_quotes(get("sans_font"));
-    m_mono_font = strip_quotes(get("mono_font"));
-    m_font_scale = stof(get("font_scale"));
+    m_serif_typeface = strip_quotes(get("serif_typeface"));
+    m_sans_typeface = strip_quotes(get("sans_typeface"));
+    m_mono_typeface = strip_quotes(get("mono_typeface"));
+    m_text_scale = stof(get("text_scale"));
 
-    if (!m_serif_font.empty())
-        m_resolved_serif = resolve_font(m_serif_font);
-    if (!m_sans_font.empty())
-        m_resolved_sans = resolve_font(m_sans_font);
-    if (!m_mono_font.empty())
-        m_resolved_mono = resolve_font(m_mono_font);
+    if (!m_serif_typeface.empty())
+        m_resolved_serif = resolve_font(m_serif_typeface, loc);
+    if (!m_sans_typeface.empty())
+        m_resolved_sans = resolve_font(m_sans_typeface, loc);
+    if (!m_mono_typeface.empty())
+        m_resolved_mono = resolve_font(m_mono_typeface, loc);
     paper_size = get("paper_size");
     landscape = strbool(get("landscape"), loc);
     leading = stof(get("leading"));
@@ -207,6 +211,135 @@ void write_file(const std::string& filename, const std::string& contents, bool w
 }
 
 
+// ---------------------------------------------------------------------
+// Typeface markers (@typeface; see the marker note in sks/font/font.py)
+//
+// Hand-rolled scanning, no std::regex: these run over the whole assembled
+// document and libstdc++'s regex executor recurses per character (the
+// documented SIGSEGV hazard).
+// ---------------------------------------------------------------------
+
+namespace {
+
+const std::string typeface_tag { "__TYPEFACE__" };
+
+// The next marker at or after `from`.  Returns false when there is none.
+// A marker is __TYPEFACE__<name>__<scale>__ : the two fields are taken up
+// to the next "__", so neither may contain one (a typeface name with a
+// double underscore is not a thing, and the scale is a number).
+bool next_typeface_marker(const std::string& text, size_t from,
+                          size_t& start, size_t& end,
+                          std::string& name, std::string& scale)
+{
+    start = text.find(typeface_tag, from);
+    if (start == std::string::npos) return false;
+    size_t name_at = start + typeface_tag.size();
+    size_t name_end = text.find("__", name_at);
+    if (name_end == std::string::npos) return false;
+    size_t scale_at = name_end + 2;
+    size_t scale_end = text.find("__", scale_at);
+    if (scale_end == std::string::npos) return false;
+    name = text.substr(name_at, name_end - name_at);
+    scale = text.substr(scale_at, scale_end - scale_at);
+    end = scale_end + 2;
+    return true;
+}
+
+// The fontspec command name for the nth distinct (typeface, scale) pair.
+// A TeX command name may hold only letters, and a typeface name may hold
+// anything -- so the pairs are NUMBERED rather than transliterated, and
+// the number is turned into letters.  Nothing outside this file needs to
+// know the rule: the klammer leaves a marker and this replaces it, so
+// there is no second implementation to keep in step.
+std::string typeface_command(size_t index)
+{
+    std::string suffix {};
+    do {
+        suffix = char('a' + (index % 26)) + suffix;
+        index /= 26;
+    } while (index > 0);
+    return "\\ktface" + suffix;
+}
+
+} // namespace
+
+void Document_class::collect_typefaces(const std::string& text)
+{
+    size_t pos = 0, start = 0, end = 0;
+    std::string name {}, scale {};
+    while (next_typeface_marker(text, pos, start, end, name, scale)) {
+        std::pair<std::string, std::string> entry { name, scale };
+        if (std::find(m_extra_typefaces.begin(), m_extra_typefaces.end(), entry)
+            == m_extra_typefaces.end()) {
+            m_extra_typefaces.push_back(entry);
+            // Resolve ONCE per distinct typeface: resolve_font() throws a
+            // located error naming the available typefaces if the store
+            // does not have it, which is the whole error path for a
+            // misspelled or uninstalled typeface.
+            bool seen = false;
+            for (const auto& rf : m_resolved_extra)
+                if (rf.family_name == name) seen = true;
+            if (!seen) {
+                // What a @typeface klammer asked for, and what the store
+                // resolved it to: a decision the user cannot read off
+                // their own input, so "-v 1".
+                Resolved_font rf = resolve_font(name, loc);
+                (void)K::log(1, "Typeface \"" + name + "\" resolved to:",
+                             rf.font_dir);
+                m_resolved_extra.push_back(rf);
+            }
+        }
+        pos = end;
+    }
+}
+
+std::string Document_class::strip_typeface_markers(const std::string& text)
+{
+    std::string result {};
+    size_t pos = 0, start = 0, end = 0;
+    std::string name {}, scale {};
+    while (next_typeface_marker(text, pos, start, end, name, scale)) {
+        result += text.substr(pos, start - pos);
+        pos = end;
+    }
+    result += text.substr(pos);
+    return result;
+}
+
+std::string Document_class::resolve_typeface_markers(const std::string& text)
+{
+    std::string result {};
+    size_t pos = 0, start = 0, end = 0;
+    std::string name {}, scale {};
+    while (next_typeface_marker(text, pos, start, end, name, scale)) {
+        result += text.substr(pos, start - pos);
+        auto entry = std::make_pair(name, scale);
+        auto it = std::find(m_extra_typefaces.begin(), m_extra_typefaces.end(), entry);
+        if (it != m_extra_typefaces.end())
+            result += typeface_command(it - m_extra_typefaces.begin());
+        pos = end;
+    }
+    result += text.substr(pos);
+    return result;
+}
+
+std::string Document_class::typeface_preamble()
+{
+    std::stringstream ss {};
+    for (size_t i = 0; i < m_extra_typefaces.size(); i++) {
+        const auto& [name, scale] = m_extra_typefaces[i];
+        const Resolved_font* rf = nullptr;
+        for (const auto& f : m_resolved_extra)
+            if (f.family_name == name) rf = &f;
+        if (!rf) continue;
+        // \newfontfamily declares a switch command, unlike \setmainfont,
+        // which changes the document's font permanently.
+        ss << latex::font_command("newfontfamily" + typeface_command(i),
+                                  *rf, stof(scale));
+    }
+    return ss.str();
+}
+
 /*
 void output_msg(const std::string& msg)
 {

sks/document/document_class.h

@@ -121,14 +121,35 @@ public:
 
     strings_t m_font_dirs {};
 
-    std::string m_serif_font {};
-    std::string m_sans_font {};
-    std::string m_mono_font {};
-    float m_font_scale = 1.0f;
+    std::string m_serif_typeface {};
+    std::string m_sans_typeface {};
+    std::string m_mono_typeface {};
+    float m_text_scale = 1.0f;
     Resolved_font m_resolved_serif {};
     Resolved_font m_resolved_sans {};
     Resolved_font m_resolved_mono {};
 
+    // Typefaces requested by @typeface klammers inside the document, in
+    // first-appearance order, each with the scale it was asked for.  A
+    // klammer cannot declare a typeface itself (see font.py's
+    // typeface_marker), so it leaves a marker and @document collects them
+    // here, resolves and localizes each, declares it, and replaces every
+    // marker with the target's inline switch.
+    // SYNC: typeface_marker() in sks/font/font.py builds this marker.
+    std::vector<std::pair<std::string, std::string>> m_extra_typefaces {};
+    std::vector<Resolved_font> m_resolved_extra {};
+    // Collect the (typeface, scale) pairs of `text` into
+    // m_extra_typefaces, resolving each typeface once.  Returns nothing;
+    // the markers are replaced by the per-target functions below.
+    void collect_typefaces(const std::string& text);
+    // html: the marker carries no inline switch (the span holds the family
+    // name), so each is replaced by nothing.
+    std::string strip_typeface_markers(const std::string& text);
+    // tex: each distinct (typeface, scale) gets a \newfontfamily command in
+    // the preamble; the marker is replaced by that command.
+    std::string resolve_typeface_markers(const std::string& text);
+    std::string typeface_preamble();
+
     std::string prolog {};
     std::string paper_size {};
     bool m_two_column = false;

sks/document/document_html.cpp

@@ -93,24 +93,28 @@ strings_t Document_class::resolved_font_names()
     add(m_resolved_serif);
     add(m_resolved_sans);
     add(m_resolved_mono);
+    // Typefaces a @typeface klammer asked for: each needs its own <link>
+    // to the localized css, which is what carries the @font-face.
+    for (const auto& rf : m_resolved_extra)
+        add(rf);
     return result;
 }
 
 std::string Document_class::font_definitions()
 {
     std::stringstream ss {};
-    if (!m_serif_font.empty() || !m_sans_font.empty() || !m_mono_font.empty()) {
+    if (!m_serif_typeface.empty() || !m_sans_typeface.empty() || !m_mono_typeface.empty()) {
         // Family names are quoted: an unquoted name with a digit-initial
         // word ("Source Sans 3") is invalid CSS, and a font-family using
         // var() with such a value computes to inherit, silently losing
         // the font.
         ss << ":root {\n";
-        if (!m_serif_font.empty())
-            ss << "    --serif: \"" << m_serif_font << "\", serif;\n";
-        if (!m_sans_font.empty())
-            ss << "    --sans-serif: \"" << m_sans_font << "\", sans-serif;\n";
-        if (!m_mono_font.empty())
-            ss << "    --monospace: \"" << m_mono_font << "\", monospace;\n";
+        if (!m_serif_typeface.empty())
+            ss << "    --serif: \"" << m_serif_typeface << "\", serif;\n";
+        if (!m_sans_typeface.empty())
+            ss << "    --sans-serif: \"" << m_sans_typeface << "\", sans-serif;\n";
+        if (!m_mono_typeface.empty())
+            ss << "    --monospace: \"" << m_mono_typeface << "\", monospace;\n";
         ss << "}\n";
     }
     // Font-size normalization across families is no longer emitted from
@@ -120,11 +124,11 @@ std::string Document_class::font_definitions()
     // factors did from build-time OS/2 metrics, but applied to EVERY family
     // switch instead of the four CSS sites that remembered to multiply.
     // The metric extraction in the font store remains (the tex path and
-    // kdesc --font still use it).
+    // kdesc --typeface still use it).
     // Global font scale: applied to body font-size
-    if (m_font_scale != 1.0f) {
+    if (m_text_scale != 1.0f) {
         char buf[16];
-        std::snprintf(buf, sizeof(buf), "%.4f", m_font_scale);
+        std::snprintf(buf, sizeof(buf), "%.4f", m_text_scale);
         ss << "body { font-size: " << buf << "rem; }\n";
     }
     return ss.str();
@@ -743,6 +747,22 @@ std::string Document_class::html()
     std::string output_directory = create_html_output_directories();
     save_string_input_as_file();
     process_input_files();
+    // @typeface markers are collected AFTER processing (a klammer cannot
+    // declare a typeface itself) but BEFORE the page is assembled, so the
+    // localized files and the <link> to each typeface's css are in place
+    // when make_*_html_page() runs.  The markers survive into the page
+    // CACHE deliberately: a cached file must still declare its typefaces
+    // on a later run, so the stripping happens here, on the assembled
+    // components, not before write_to_cache().
+    for (auto& [basename, text] : m_file_components) {
+        collect_typefaces(text);
+    }
+    for (const auto& rf : m_resolved_extra) {
+        install_resolved_font(rf, output_directory);
+    }
+    for (auto& [basename, text] : m_file_components) {
+        text = strip_typeface_markers(text);
+    }
     auto headings = insert_section_numbers();
     insert_section_numbers("___NUM2___", false);
     insert_html_caption_numbers();

sks/document/document_latex.cpp

@@ -72,6 +72,14 @@ std::string Document_class::tex()
     }
     latex_text = join(lines, "\n");
     latex_text = string_replace(latex_text, "___AT___", "^@");
+    // @typeface: collect what the document asked for, declare each as a
+    // \newfontfamily switch command in the preamble, and replace every
+    // marker with its command.  The declarations are appended to the
+    // prolog, which latex::page emits AFTER \setmainfont -- so fontspec is
+    // loaded and \newfontfamily is valid there.
+    collect_typefaces(latex_text);
+    latex_text = resolve_typeface_markers(latex_text);
+    prolog += typeface_preamble();
     std::string structure_name =
         m_structure == docstruct_t::book    ? "book" :
         m_structure == docstruct_t::article ? "article" : "plain";
@@ -81,5 +89,5 @@ std::string Document_class::tex()
                        point_size, ragged_right,
                        cover, landscape, latex_text,
                        m_resolved_serif, m_resolved_sans, m_resolved_mono,
-                       m_font_scale);
+                       m_text_scale);
 }