Unicode sequences, @typeface, and the font/typeface distinction

A run of Unicode code points now needs one pair of carets, not one pair
each: ^9053 53EF 9053^ renders the characters adjacently, and a single
character inside such a run stands for itself.  An invalid code point is
an error naming the position that wrote it, rather than an apology
rendered into the finished document; noncharacters are refused alongside
surrogates; and a code point that lands on a Klammertext special is safe
through the re-reads that assemble a document.  The character tables now
produce the precomposed form, so a diacritic written as ^a" and one typed
directly are the same text to a search or a copy-paste.

"kdesc -u" describes how to write a Unicode character, with tables of
glyphs beside the codes that produce them.  It is new and not yet
complete: it predates the sequence form and the typeface klammer below.

A typeface and a style are no longer both called "font".  A typeface is a
family resolved by name; a style code is the one letter naming a type
style.  So @document takes :serif_typeface and its siblings, a table takes
:style, captions take :caption_style and :caption_size, the uniform scale
is :text_scale, and the command is "kdesc --typeface".  This renames
arguments a document may already use; an old name is a located error
rather than a silent omission.

@typeface sets a passage in another typeface for the length of that
passage:

    @typeface Noto Sans Egyptian Hieroglyphs | ^132AA 133CF^ :scale 1.5 @

Writing the klammer is the whole declaration -- the document resolves the
typeface, embeds it in what it produces, and carries its license along.
Two typefaces for scripts no text font covers now ship with Klammertext:
Noto Sans Egyptian Hieroglyphs and Noto Sans Cuneiform.  Every typeface in
the distribution now carries its license file.

(from dev 61e598698891)

mac/katom_list.cpp

@@ -281,11 +281,164 @@ void encode_nonascii_characters(katom_list& katoms)
 {
     for (Katom& k : katoms) {
         if (is_nonascii(k)) {
-            k.m_text = encode(k.m_text);
+            k.m_text = encode(k.m_text, k.m_loc);
         }
     }
 }
 
+// Unicode sequences
+
+// ^UUUU UUUU ...^ -- a whitespace-delimited sequence of Unicode code
+// points between ONE pair of carets: ^9053 53EF 9053^ renders the three
+// characters adjacently.  The single-token form ^UUUU^ is the katomizer's
+// nonascii katom, resolved by encode(); a sequence cannot be a katom at
+// all, because whitespace is preserved as separate katoms and no katom
+// spans it.  So the tokens arrive as word katoms -- the opener "^"+hex,
+// the closer hex+"^", bare hex between, separated by SPACE katoms only (a
+// newline aborts: a sequence is one line; so does any other katom, which
+// is also what enforces "no space inside the delimiters' elbows": a bare
+// "^" is not an opener) -- and are assembled here.  Each token is 4-6 hex
+// digits, the same rule as the single form.  The rendered characters
+// replace the opening katom's text; the other members are marked
+// replaced.  Skipped inside @@/@@@ definition spans and
+// @eval/@read/@cond argument spans (code keeps its carets -- the rule the
+// string-level single-form pass cannot honor) and for literal katoms
+// (verbatim content; mark_literal_* run before this).
+namespace {
+
+bool hex_token(const std::string& s)
+{
+    if (s.size() < 4 || s.size() > 6) return false;
+    for (char c : s) {
+        bool hex = (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f')
+                || (c >= 'A' && c <= 'F');
+        if (!hex) return false;
+    }
+    return true;
+}
+
+// A sequence token is a 4-6-digit hex code point or a SINGLE character
+// standing for itself ("^" excluded -- a lone caret cannot say which
+// caret closes).  The single-character reading exists only inside a
+// sequence; ^s^ alone stays the mnemonic.
+bool shape_token(const std::string& s)
+{
+    return hex_token(s) || (s.size() == 1 && s[0] != '^');
+}
+
+// Render one scanned token.  A hex token is validated -- an invalid code
+// point is a located error, not text in the document -- and a structural
+// character in the result is hidden as its KTESC marker.  A single
+// character behaves exactly as if quoted with "^": alphanumerics stand
+// bare, anything else becomes its marker.
+std::string render_token(const std::string& s, const Locator& loc)
+{
+    if (hex_token(s)) {
+        int cp = std::stoi(s, nullptr, 16);
+        if (!valid_codepoint(cp)) {
+            throw Parsing_error(
+                "Invalid Unicode code point \"" + s + "\" in a ^...^ sequence: "
+                + invalid_codepoint_reason(cp), loc);
+        }
+        return hide_structural_characters(utf8char(cp));
+    }
+    char c = s[0];
+    bool alnum = (c >= '0' && c <= '9') || (c >= 'a' && c <= 'z')
+              || (c >= 'A' && c <= 'Z');
+    return alnum ? s : Target::escape_marker(s);
+}
+
+} // namespace
+
+void process_unicode_sequences(katom_list& katoms)
+{
+    (void)K::log(4);
+    int definition_depth = 0;
+    int code_depth = 0;              // inside an @eval/@read/@cond span
+    std::vector<bool> apply_is_code; // one entry per open application
+    for (size_t i = 0; i < katoms.size(); ++i) {
+        Katom& k = katoms[i];
+        switch (k.m_type) {
+        case katom_t::define_begin:
+        case katom_t::machine_begin:
+            ++definition_depth;
+            continue;
+        case katom_t::define_end:
+        case katom_t::machine_end:
+            if (definition_depth > 0) --definition_depth;
+            continue;
+        case katom_t::eval_begin:
+        case katom_t::read_begin:
+        case katom_t::cond_begin:
+            apply_is_code.push_back(true);
+            ++code_depth;
+            continue;
+        case katom_t::apply_begin:
+            apply_is_code.push_back(false);
+            continue;
+        case katom_t::apply_end:
+            if (!apply_is_code.empty()) {
+                if (apply_is_code.back()) --code_depth;
+                apply_is_code.pop_back();
+            }
+            continue;
+        default:
+            break;
+        }
+        if (definition_depth > 0 || code_depth > 0) continue;
+        if (k.m_type != katom_t::word && k.m_type != katom_t::text) continue;
+        // The opener: "^" + one token, nothing else.
+        if (k.m_text.size() < 2 || k.m_text[0] != '^') continue;
+        if (!shape_token(k.m_text.substr(1))) continue;
+        // Phase 1 -- SCAN by shape only, no validation: an unclosed run
+        // must fall through untouched (it may be innocent prose about hex),
+        // so an invalid code point may only error once the sequence has
+        // actually closed.  Tokens are word katoms separated by SPACE
+        // katoms; any other katom (a newline included: one line only)
+        // aborts.
+        std::vector<std::pair<std::string, const Locator*>> tokens {
+            { k.m_text.substr(1), &k.m_loc } };
+        std::vector<size_t> members {};
+        bool closed = false;
+        size_t j = i + 1;
+        for (; j < katoms.size(); ++j) {
+            Katom& t = katoms[j];
+            if (t.m_type == katom_t::space) {
+                members.push_back(j);
+                continue;
+            }
+            if (t.m_type != katom_t::word && t.m_type != katom_t::text) break;
+            const std::string& s = t.m_text;
+            if (s.size() >= 2 && s.back() == '^'
+                && shape_token(s.substr(0, s.size() - 1))) {
+                tokens.push_back({ s.substr(0, s.size() - 1), &t.m_loc });
+                members.push_back(j);
+                closed = true;
+                ++j;
+                break;
+            }
+            if (shape_token(s)) {
+                tokens.push_back({ s, &t.m_loc });
+                members.push_back(j);
+                continue;
+            }
+            break;
+        }
+        if (!closed) continue;
+        // Phase 2 -- render, validating each code point at its own katom.
+        std::string rendered {};
+        for (const auto& [text, loc] : tokens) {
+            rendered += render_token(text, *loc);
+        }
+        k.m_text = rendered;
+        k.m_unparsed = false;
+        for (size_t m : members) {
+            mark_as_replaced(katoms[m]);
+        }
+        i = j - 1;
+    }
+}
+
 // Literal
 
 void mark_literal_katoms(katom_list& katoms)