Characters that killed a render, what may sit next to a klammer, and ref/

A character written in the ^...^ mnemonic form -- ^oe^, ^zz^, ^hello^ --
used to kill ktext outright: no message, no output, and an exit status
that looked like a crash because it was one.  Only the forms that happen
to spell hexadecimal digits escaped, and they were right by luck rather
than by reading.  Any caret form that names no character is now a located
error that quotes what was written.

A klammer may now sit inside square brackets and braces as it already
could inside parentheses, and may be followed by the punctuation that
ends a sentence: [@i word@], {@b word@}, @i word@'s.  The rule is a fixed
list of characters, because none of them can occur inside a klammer name
and so none of them is ambiguous next to a delimiter.  A letter still
cannot: "x@" is the named closing delimiter, so a letter before "@" has
two readings and no rule can separate them.

The abbreviations an author writes -- the hyphen form @i-word, a klammer
glued to its neighbouring punctuation -- are now recognised as author
conveniences only.  They are not applied to @@ definitions or @@@ system
commands, which are written by someone who needs no such shorthand and
whose forms are already whole.

Also fixed, each of them silent: an exported shell function in the
environment broke every @eval in the document; a line break written with
@nl was destroyed when it passed through a klammer that generates a
table, printing as \textbackslash{}newline in the finished PDF; and
"kdesc --coverage" shortened its file column by comparing the spelling of
a path rather than the path, so a klammer set reached by a symbolic link
showed a full path where its neighbours showed a short one.

New in the distribution: ref/, a small klammerset that describes the
Klammermachine's own tables in Klammertext.  It currently provides the
character tables -- the diacritics and the extended Latin letters -- as
klammers that render to any target, so the documentation of what can be
written cannot drift from what the machine accepts.  It is early, and
will grow.

The Klammertext taxonomy now has a section on targets, describes the
@@@klammerset system command alongside the other three, and states the
precedence of "^" -- what a caret means before a letter, a digit, a
punctuation character, and a quote, in one ordered list.

(from dev 6594c7ff8c1f)

mac/ktype.h

@@ -37,8 +37,8 @@ inline const std::string ws_newline_s { R"(#/\d*)" };
 inline const std::string definition_name = R"([a-zA-Z][a-zA-Z0-9_.]*)";
 // The name at the START of a definition may carry a comma-separated list of
 // targets ("@@table.html,tex"), so its katom runs through commas.  This is
-// deliberately looser than the grammar: the katomizer's job is to delimit the
-// token, and parse_name() in klammer.cpp validates the list, where a
+// deliberately looser than the grammar: katomization's job is to delimit
+// the katom, and parse_name() in klammer.cpp validates the list, where a
 // malformed one can be reported with its location.  The comma stays out of
 // definition_name itself -- that pattern also delimits klammer APPLICATIONS,
 // option names and "*arg*" variables, where a comma is ordinary writer text.
@@ -192,7 +192,22 @@ std::vector<Ktype> katom_types {
     Ktype(katom_t::ws_newline, "ws-newline", ws_newline_s, "Remove all whitespace, leaving <number> newlines (default: 1)"),
     Ktype(katom_t::special, "special-char", R"(\^[@|#^:*])", "Klammertext special character treated as regular text"),
     //Ktype(katom_t::nonascii, "non-ascii-char", R"(\^\w(?:.|\^))", "Non-ASCII character"),
-    Ktype(katom_t::nonascii, "non-ascii-char", R"((\^(\w(?:.|\^)))|(\^[A-Fa-f0-9]{1,6}\^))", "Non-ASCII character"),
+    // The second alternative is the "^...^" delimited form: a code point
+    // (^263A^, 4-6 hex digits) OR a mnemonic (^s^, ^oe^).  Its character
+    // class is [A-Za-z0-9], not [A-Fa-f0-9], because a mnemonic is not
+    // hex: "^oe^" matched NEITHER alternative and so was not a katom at
+    // all, fell through to the "non-diacritic non-ascii" rewrite rule,
+    // which pads it with whitespace and hands back the same single word --
+    // and make_katoms_from_word() then recursed on the identical string
+    // until the stack died (SIGSEGV, no message).  The two entries that
+    // appeared to work, ^ae^ and ^AE^, did so only because "ae" happens to
+    // be hex digits; they were being katomized as CODE POINTS and rendered
+    // correctly only because encode() runs process_extended_latin() before
+    // process_unicode_codepoint().  Which of the two a form means is
+    // decided there, by the tables, not here -- katomization's job is only to
+    // recognize the shape.  A shape that resolves to no character at all is
+    // a located error, raised at the end of encode().
+    Ktype(katom_t::nonascii, "non-ascii-char", R"((\^(\w(?:.|\^)))|(\^[A-Za-z0-9]{1,6}\^))", "Non-ASCII character"),
 
     // Ktype(katom_t::word, "word", R"([a-z][a-z0-9_]*)", "Lower-case letters, numbers, or underscore"),
     // Ktype(katom_t::text, "text",
@@ -210,52 +225,7 @@ std::vector<Ktype> katom_types {
 //    Ktype(katom_t::undefined, "undefined", "", "Undefined pattern")
 };
 
-class Rgx {
- public:
-    explicit Rgx(std::string pattern)
-    : m_pattern(pattern)
-    , m_regex(std::regex(pattern)) {}
-    std::string m_pattern;
-    std::regex m_regex;
-};
 
-inline const
-std::vector<std::tuple<std::string, Rgx, std::string>> katom_rewrite_rules {
-    { "bar precedence", Rgx(R"((.*?)(\^\|)(.*))"), "$1 $2 $3" },
-    { "literal without space",     Rgx(R"((\^')(.*?)('\^))"), "$1 $2 $3" },
-    { "literal start to the left", Rgx(R"((.+)(\^'))"), "$1 $2" },
-    { "literal start to the right", Rgx(R"((\^')(.+))"), "$1 $2" },
-    { "literal end to the left",  Rgx(R"(('\^)(.+))"), "$1 $2" },
-    { "literal end to the right",  Rgx(R"((.+)('\^))"), "$1 $2" },
-    //{ "successive non-ascii characters", Rgx(R"((.?)(\^\w[-\"^`'~hcbrdwa])(\^\w)(.?))"), "$1 $2 $3 $4" },
-    // { "special character", Rgx(R"((.*?)(\^[@\|\^#\*])(.*))"), "$1 $2 $3" },
-    { "special character", Rgx(R"((.*?)(\^[@|^#*:])(.*))"), "$1 $2 $3" },
-    { "non-diacritic non-ascii", Rgx(R"((.*?)(\^[a-zA-Z0-9]{5}\^)(.*?))"), "$1 $2 $3" },
-    { "non-diacritic non-ascii 2", Rgx(R"((.*?)(\^[a-zA-Z0-9]{1,4}\^)(.*?))"), "$1 $2 $3" },
-    { "diacritic non-ascii", Rgx(R"((.*?)(\^[a-zA-z][^^])(.*?))"), "$1 $2 $3" },
-    { "double bar separator", Rgx(R"(([^\s\^]+?)(\|\|)(.*))"), "$1 $2 $3" },
-    { "bar separator", Rgx(R"(([^\s\^]+?)(\|)(.*))"), "$1 $2 $3" },
-    { "embedded ignore begin", Rgx(R"((.*?)#\[(.*))"), "$1 #[ $2" },
-    { "embedded ignore end", Rgx(R"((.*?)\]#(.*))"), "$1 ]# $2" },
-    { "embedded remove whitespace", Rgx(R"((.*?)#-(.*))"), "$1 #- $2" },
-    { "single argument for special character", Rgx(R"((@\w+)-(\^[@\|:\*\^])@?)"), "$1 $2 @" },
-    // { "special character", Rgx(R"((.*?)(\^[@\|:\*\^])(.*))"), "$1 $2 $3" },
-    { "klammer body arguments", Rgx(R"((.*?)(\*\w+\*)(.*))"), "$1 $2 $3" },
-    { "trailing punctuation", Rgx(R"((\w?)@([().,:;?'!]))"), "$1@ $2" },
-    { "trailing punctuation", Rgx(R"(@([().,:;?'!].*))"), "@ $1" },
-    { "trailing punctuation, shortcut", Rgx(R"((@\w+)-(\w+)([().,:;?'!][^\s]*))"), "$1 $2 @ $3" },
-    // { "single argument shortcut", Rgx(R"((@\w+)-([^\s@]+)@?)"), "$1 $2 @" },
-    { "no argument klammer", Rgx(R"((@\w+)@)"), "$1 @" },
-    { "left parenthesis", Rgx(R"(([(])@(\w+))"), "$1 @$2" },
-    { "embedded klammer", Rgx(R"(([^@]+)(@[^@]+@)([^@]+))"), "$1 $2 $3" },
-    { "embedded klammer to the right", Rgx(R"(([^@]+)(@[^@]+@))"), "$1 $2" },
-    { "embedded klammer to the left", Rgx(R"((@[^@]+@)([^@]+))"), "$1 $2" },
-    { "embedded add whitespace", Rgx(R"(([^\s])(#\+\d*)([^\s]))"), "$1 $2 $3" },
-    { "embedded add whitespace to the left", Rgx(R"(([^\s])(#\+\d*))"), "$1 $2" },
-    { "embedded add whitespace to the right", Rgx(R"((#\+\d*)([^\s]))"), "$1 $2" },
-    { "embedded ignore on the left", Rgx(R"(#([^-+/\s]))"), "# $1" },
-    { "embedded ignore on the right", Rgx(R"(([^\s])#)"), "$1 #" },
-};
 
 
 inline const
@@ -308,7 +278,6 @@ bool is_active(katom_t t);
 bool is_printable(katom_t type);
 
 void describe_katoms(bool show_regex = false);
-void describe_rewrite_patterns();
 
 bool is_deftype(katom_t type);