A rest argument keeps an empty first element; a removal block keeps its neighbour; the Markdown converter reads paragraphs whole; Smart App Control on the Windows page (from dev 1083db946f05)

- A rest argument (a list's or a table's items) now strips exactly ONE
  leading bar -- the separator from what precedes it -- so a second one is
  an empty first element: a table with an empty header corner is written
  `@table |` then ` | Maskulinum | ...`. Before, every leading bar was
  dropped and such a header row shifted one column left. New shipped suite
  tst/rest_test.sh.
- Text written hard against a closing ]# (a period, a letter) is no longer
  removed with the block; a following space or line break still is.
- sks/tns/md_to_sks.py converts inline Markdown per paragraph, so emphasis
  that wraps is converted and not mis-paired; bold may contain italic;
  tables are unnumbered; links to sibling .md files converted alongside
  become links to their html pages; the arrow becomes @rightarrow@.
- doc/install/windows_source_install.md: Smart App Control must be off --
  it blocks MSYS2's unsigned toolchain, and its Evaluation state turns
  itself on later.
This commit is contained in:
ack committed 2026-10-04 13:18:06 +02:00
1 parent 6e9889f7ae
commit bc2b3d8461
8 files changed
+244 -23

No files matched your search

+1 -1
View File
@@ -54,7 +54,7 @@ are regenerated on each release — patches cannot be merged directly.
Report problems (or send patches) to the author; accepted changes are Report problems (or send patches) to the author; accepted changes are
applied to the development tree and appear in a following snapshot. applied to the development tree and appear in a following snapshot.
This snapshot was assembled from development commit `1ce461b48a7d`. This snapshot was assembled from development commit `1083db946f05`.
## License ## License
+23
View File
@@ -28,6 +28,24 @@ special rights. Allow an hour or two the first time: the downloads are
about a gigabyte in all, TeX Live alone takes twenty to thirty minutes, about a gigabyte in all, TeX Live alone takes twenty to thirty minutes,
and the build a few minutes. Nothing here needs to be done twice. and the build a few minutes. Nothing here needs to be done twice.
**Smart App Control must be off.** A clean installation of Windows 11
runs Smart App Control, a security feature that blocks any program
Microsoft has not seen signed and widespread — and the MSYS2 compiler,
`make` and the `bash` shell on which this page depends are not signed, so
with it on the MSYS2 shell does not open and the build cannot start
("An Application Control policy has blocked this file"). Check it first:
Windows Security → **App & browser control** → **Smart App Control
settings**. It is in one of three states. *Off* is what you need. *On*
blocks MSYS2; switch it off. *Evaluation* lets everything run for now,
but Windows turns it on by itself when it decides the machine is used
"safely", at which point a working installation stops without warning —
so switch it off at once rather than wait. Switching it off cannot be
undone short of reinstalling Windows, which is why Microsoft asks you to
confirm; the rest of Windows Security (Defender, SmartScreen for
downloads) stays on and is not affected. A machine upgraded from Windows
10, or one on which an unsigned program has already been installed, has
it off already.
## Prerequisites ## Prerequisites
@@ -403,6 +421,11 @@ it (a new UCRT64 shell does, once the line is in `~/.bashrc`) and run
`MSYS2_PATH_TYPE` is not set to `inherit`, or the shell was opened before `MSYS2_PATH_TYPE` is not set to `inherit`, or the shell was opened before
you set it. Set it and open a new UCRT64 shell. you set it. Set it and open a new UCRT64 shell.
**"An Application Control policy has blocked this file", or the MSYS2
shell does not open.** Smart App Control is on; see *Before you start*.
It can turn itself on after the installation, in which case the
installation is intact and only the setting needs changing.
**The build is slow.** Windows Defender scans every file the compiler **The build is slow.** Windows Defender scans every file the compiler
writes. Excluding the repository folder from real-time scanning speeds the writes. Excluding the repository folder from real-time scanning speeds the
build up considerably; it is not required. build up considerably; it is not required.
+46 -8
View File
@@ -225,6 +225,17 @@ katom_list trim_part(katom_list part)
return trim(part, {katom_t::space, katom_t::newline, katom_t::bar}); return trim(part, {katom_t::space, katom_t::newline, katom_t::bar});
} }
// The trailing half of trim_part(): the leading side of a rest argument is
// decided bar by bar in argument_split().
static katom_list trim_trailing_part(katom_list part)
{
const std::set<katom_t> trim_types {katom_t::space, katom_t::newline, katom_t::bar};
while (!part.empty() && trim_types.contains(part.back().m_type)) {
part.pop_back();
}
return part;
}
std::tuple<katom_lists,katom_lists> std::tuple<katom_lists,katom_lists>
parameter_split(katom_list::const_iterator kbegin, katom_list::const_iterator kend) parameter_split(katom_list::const_iterator kbegin, katom_list::const_iterator kend)
{ {
@@ -330,24 +341,51 @@ argument_split(katom_list::const_iterator kbegin, katom_list::const_iterator ken
katom_lists optional {}; katom_lists optional {};
katom_list rest {}; katom_list rest {};
for (auto p : parts) { // function_symbol_parts() opens the first part with a bar of its own
// when the arguments do not begin with an option. Without positional
// parameters that bar is the first katom of the rest, and it is not
// the writer's.
bool rest_opens_synthetic = false;
for (std::size_t i = 0; i < parts.size(); i++) {
const auto& p = parts[i];
if (p[0].m_type == katom_t::option_name) { if (p[0].m_type == katom_t::option_name) {
optional.push_back(p); optional.push_back(p);
} else if (positional.size() < positional_limit) { } else if (positional.size() < positional_limit) {
positional.push_back(trim_part(p)); positional.push_back(trim_part(p));
} else { } else {
// Parts were trimmed, so adjacent parts would abut their bar if (rest.empty()) {
// katoms (a row separator "||" next to an empty cell's "|" rest_opens_synthetic = (i == 0) && positional_limit == 0;
// would serialize as "|||"). A space keeps the writer's } else {
// bar/double-bar distinction parseable. // Parts were trimmed, so adjacent parts would abut their bar
if (!rest.empty()) { // katoms (a row separator "||" next to an empty cell's "|"
// would serialize as "|||"). A space keeps the writer's
// bar/double-bar distinction parseable.
rest.push_back(Katom(" ", katom_t::space, p[0].m_loc)); rest.push_back(Katom(" ", katom_t::space, p[0].m_loc));
} }
rest.insert(rest.end(), p.begin(), p.end()); rest.insert(rest.end(), p.begin(), p.end());
} }
} }
return {positional, optional, trim_part(rest)};
} // Exactly ONE written leading bar is stripped from the rest (decided
// 2026-10-01): it is the bar that separates the rest from what
// precedes it -- the last positional, the options, or the klammer name
// in the opening-bar idiom `@table |` with the rows on the lines below.
// Every further leading bar delimits an EMPTY FIRST ELEMENT the writer
// meant (`@table | | M | F ||` has an empty corner cell), mirroring
// kutil.rest_split, which strips one trailing delimiter and keeps all
// other empty elements. Trailing bars and whitespace are trimmed as
// before.
rest = trim_whitespace(rest);
if (rest_opens_synthetic && !rest.empty() && rest[0].m_type == katom_t::bar) {
rest.erase(rest.begin());
rest = trim_whitespace(rest);
}
if (!rest.empty() && rest[0].m_type == katom_t::bar && rest[0].m_text == "|") {
rest.erase(rest.begin());
rest = trim_whitespace(rest);
}
return {positional, optional, trim_trailing_part(rest)};
}
// Parameter/argument mapping // Parameter/argument mapping
+9 -6
View File
@@ -554,13 +554,16 @@ void mark_ignored_katoms(katom_list& katoms)
//auto is_ignore_begin = [](const Katom& k) { return k.m_type == katom_t::ignore_begin; }; //auto is_ignore_begin = [](const Katom& k) { return k.m_type == katom_t::ignore_begin; };
//auto is_ignore_end = [](const Katom& k) { return k.m_type == katom_t::ignore_end; }; //auto is_ignore_end = [](const Katom& k) { return k.m_type == katom_t::ignore_end; };
for (auto [op,cl] : find_spans(katoms, begin_ignore, end_ignore, true, "ignored")) { for (auto [op,cl] : find_spans(katoms, begin_ignore, end_ignore, true, "ignored")) {
// find_span_katoms() returns a half-open range: `end` is the katom
// AFTER the closing "]#", as every other caller reads it. This one
// read it as inclusive until 2026-10-02 and so removed the katom
// abutting the close: "#[...]#." lost its period, "#[...]#x" its x.
// A following space or newline is removed by design (the loop
// below, and here), which is what hid it.
auto [begin, end] = find_span_katoms(katoms, op, cl); auto [begin, end] = find_span_katoms(katoms, op, cl);
// std::for_each(begin, end + 1, [](Katom& k) { k.m_type = katom_t::ignored; }); std::for_each(begin, end, mark_as_ignored);
std::for_each(begin, end + 1, mark_as_ignored); if (end != katoms.end() && is_newline(*end)) {
//if (end + 1 != katoms.end() && (end+1)->m_type == katom_t::newline) { mark_as_ignored(*end);
if (end + 1 != katoms.end() && is_newline(*(end + 1))) {
//(end+1)->m_type = katom_t::ignored;
mark_as_ignored(*(end + 1));
} }
} }
bool ignore_line = false; bool ignore_line = false;
+58 -8
View File
@@ -17,15 +17,25 @@ grep. A file with no markers is inside the convertible subset.
First run: the Rectify user guide (1,805 lines), 2026-08-07. It rendered to First run: the Rectify user guide (1,805 lines), 2026-08-07. It rendered to
html and pdf without error or warning, with 99 markers -- 91 of them internal html and pdf without error or warning, with 99 markers -- 91 of them internal
anchor links, the one gap that appears at volume. anchor links, the one gap that appears at volume. Second run: three German
vocabulary lists (4,200 lines, 152 tables), 2026-09-30, which rendered
without an error and with two silently wrong results -- an emphasis that
wrapped and a table's empty header corner -- so "renders clean" is not
"converted"; the fixes are in the spec's "What the second conversion
taught".
Usage: python3 md_to_sks.py <input.md> <output.kt> Usage: python3 md_to_sks.py <input.md> <output.kt> [sibling.md ...]
The siblings are Markdown files converted alongside this one: a link to
one of them becomes a link to its html page (../<stem>/index.html, where
ktext writes it); a link to any other .md file is a gap.
""" """
import re, sys, os, datetime import re, sys, os, datetime
from collections import Counter from collections import Counter
SPECIALS = "^@#|*" # ^ first: the other quotings introduce ^ SPECIALS = "^@#|*" # ^ first: the other quotings introduce ^
marks = Counter() marks = Counter()
SIBLINGS = set() # stems of the .md files converted together
def mark(kind, text): def mark(kind, text):
"""A #[MD ... ]# marker. """A #[MD ... ]# marker.
@@ -56,7 +66,9 @@ def quote(s):
CODE = re.compile(r'`([^`\n]+)`') CODE = re.compile(r'`([^`\n]+)`')
LINK = re.compile(r'(?<!\!)\[([^\]]*)\]\(([^)]+)\)') LINK = re.compile(r'(?<!\!)\[([^\]]*)\]\(([^)]+)\)')
BOLD = re.compile(r'\*\*([^*]+)\*\*|__([^_]+)__') # A bold run may hold italic runs ("**only for *sein* and *haben***"): the
# group admits a balanced *...* inside, so the closing ** is found past it.
BOLD = re.compile(r'\*\*((?:[^*]|\*[^*\s][^*]*\*)+)\*\*|__([^_]+)__')
# Markdown's underscore emphasis does not open or close INSIDE a word, and # Markdown's underscore emphasis does not open or close INSIDE a word, and
# technical prose is full of identifiers like *name*_rectified_1.*ext* where # technical prose is full of identifiers like *name*_rectified_1.*ext* where
# the underscores are part of the name. So the closing _ must not be # the underscores are part of the name. So the closing _ must not be
@@ -99,10 +111,22 @@ def inline(text):
def link(m): def link(m):
label, target = m.group(1), m.group(2) label, target = m.group(1), m.group(2)
# A link that cannot be converted keeps its words and records the
# loss. The marker goes BEFORE the words: a link is usually
# followed by punctuation, and a marker between the words and the
# period leaves the space before the marker standing ("words .").
if target.startswith("#"): if target.startswith("#"):
# An anchor into this document: no klammer gives an arbitrary # An anchor into this document: no klammer gives an arbitrary
# heading an anchor, so keep the words and record the loss. # heading an anchor.
return emphasis(label) + " " + keep(mark("gap", f"link to {target}")) return keep(mark("gap", f"link to {target}")) + " " + emphasis(label)
if target.endswith(".md") and "/" not in target:
# Another Markdown file. "@link X.md" would be read as an
# anchor (href="#X.md"). Converted alongside this one, its html
# page is X/index.html beside ours; otherwise it is a gap.
stem = target[:-3]
if stem not in SIBLINGS:
return keep(mark("gap", f"link to {target}, not converted")) + " " + emphasis(label)
target = f"../{stem}/index.html"
return opening("@link " + quote(target) + " :text ") + emphasis(label) + closing() return opening("@link " + quote(target) + " :text ") + emphasis(label) + closing()
def emphasis(t): def emphasis(t):
@@ -115,6 +139,9 @@ def inline(text):
the wrong list: IndexError, found by converting a second document. the wrong list: IndexError, found by converting a second document.
""" """
t = LINK.sub(link, t) t = LINK.sub(link, t)
# An arrow is the SKS klammer (Andy, 2026-10-01): U+2192, and the
# halfwidth U+FFEB his documents use.
t = re.sub(r'[\u2192\uffeb]', lambda m: keep(OPEN + "@rightarrow@" + CLOSE), t)
t = STRIKE.sub(lambda m: emphasis(m.group(1)) + " " + t = STRIKE.sub(lambda m: emphasis(m.group(1)) + " " +
keep(mark("gap", "strikethrough")), t) keep(mark("gap", "strikethrough")), t)
t = BOLD.sub(lambda m: opening("@b ") + (m.group(1) or m.group(2)) + closing(), t) t = BOLD.sub(lambda m: opening("@b ") + (m.group(1) or m.group(2)) + closing(), t)
@@ -230,7 +257,9 @@ def convert(path):
rows.append([inline(c) for c in cells(src[i])]) rows.append([inline(c) for c in cells(src[i])])
i += 1 i += 1
body = " ||\n".join(" | ".join(r) for r in rows) body = " ||\n".join(" | ".join(r) for r in rows)
opts = f" :cell_hpos {' '.join(align)}" if align and set(align) != {"l"} else "" # A Markdown table has no caption and no number.
opts = " :number false"
opts += f" :cell_hpos {' '.join(align)}" if align and set(align) != {"l"} else ""
# A Markdown table says nothing about column widths, and # A Markdown table says nothing about column widths, and
# Klammertext's default "fit" never wraps -- a prose column then # Klammertext's default "fit" never wraps -- a prose column then
# runs off the page (the pdf target warns). Give the widest # runs off the page (the pdf target warns). Give the widest
@@ -263,8 +292,28 @@ def convert(path):
out.append("") out.append("")
continue continue
out.append(inline(line) if line.strip() else "") if not line.strip():
i += 1 out.append(""); i += 1
continue
# A PARAGRAPH goes through inline() whole, its line breaks kept.
# Converting line by line left a literal "*" on each line of an
# emphasis that wrapped, and the leftovers then paired with EACH
# OTHER, so the wrong words came out italic -- 33 paragraphs of the
# vocabulary lists, visible only in a diff.
para = [line]; i += 1
while i < n and src[i].strip() and not (HEADING.match(src[i]) or
TABLEROW.match(src[i]) or FENCE.match(src[i]) or
BULLET.match(src[i])):
if ORDERED.match(src[i]):
# A numbered line hard against the line above is a wrapped
# line that happens to begin with a number ("1. and 3.
# Sg."). CommonMark reads it as a list; the writer did not
# mean one. Kept in the paragraph, and said.
out.append(mark("judgment", "a line beginning with a number "
"was kept in its paragraph, not read as a list"))
para.append(src[i]); i += 1
out.append(inline("\n".join(para)))
return out, title, stripped_numbers, dropped_toc return out, title, stripped_numbers, dropped_toc
@@ -312,6 +361,7 @@ def convert_list(block):
def main(): def main():
src_path, dst_path = sys.argv[1], sys.argv[2] src_path, dst_path = sys.argv[1], sys.argv[2]
SIBLINGS.update(os.path.splitext(os.path.basename(a))[0] for a in sys.argv[3:])
body, title, stripped, dropped_toc = convert(src_path) body, title, stripped, dropped_toc = convert(src_path)
today = datetime.date.today().isoformat() today = datetime.date.today().isoformat()
head = [f"#[MD source: {os.path.abspath(src_path)}", head = [f"#[MD source: {os.path.abspath(src_path)}",
+1
View File
@@ -19,6 +19,7 @@ test:
./katomize_test.sh ./katomize_test.sh
./filename_test.sh ./filename_test.sh
./alone_test.sh ./alone_test.sh
./rest_test.sh
./argtype_test.sh ./argtype_test.sh
./state_test.sh ./state_test.sh
./target_test.sh ./target_test.sh
+12
View File
@@ -151,6 +151,18 @@ check_warn "26. ... and is reported, not split" '@@w@@'
# the first version of the guard broke it. # the first version of the guard broke it.
check "27. @@ glued to #- still splits" '<x>' '@@v.* : <x> @@#-@v@' check "27. @@ glued to #- still splits" '<x>' '@@v.* : <x> @@#-@v@'
echo
echo "-- what abuts a closing ]# --"
# A removal block ends at its "]#"; what follows is the writer's. Until
# 2026-10-02 mark_ignored_katoms() read find_span_katoms()'s half-open end
# as inclusive and removed the katom abutting the close as well: a sentence
# ending "#[ note ]#." lost its period. A following space or newline is
# removed by design, which is what hid it.
check "28. punctuation after ]# survives" 'a . c' 'a #[gone]#. c'
check "29. a letter after ]# survives" 'a x c' 'a #[gone]#x c'
check "30. a space after ]# is removed" 'a . c' 'a #[gone]# . c'
check "31. nested blocks, punctuation after" 'a . c' 'a #[ #[in]# out ]#. c'
echo echo
echo "=================================================================" echo "================================================================="
echo "${bold}Results: $PASS passed, $FAIL failed${reset}" echo "${bold}Results: $PASS passed, $FAIL failed${reset}"
+94
View File
@@ -0,0 +1,94 @@
#!/bin/bash
#
# rest_test.sh — Regression tests for the bar structure of a rest argument.
#
# A rest argument collects everything after the declared positional
# arguments, bars included, and hands the structure on; a klammer reads the
# bar count as dimension (| elements, || rows). The one thing the ENGINE
# decides is where the structure begins and ends:
#
# * exactly ONE written leading bar is stripped — the bar that separates
# the rest from what precedes it: the last positional argument, the
# options, or the klammer name in the opening-bar idiom (`@table |` with
# the rows on the lines below). Every further leading bar delimits an
# EMPTY FIRST ELEMENT the writer meant: `@table | | M | F ||` has an
# empty corner cell. Decided 2026-10-01; before it every leading bar
# was trimmed, so an empty first element could not be written at all,
# and a Markdown table's empty header corner shifted the header row one
# column left.
# * trailing bars and whitespace are trimmed (the customary dangling
# separator before a closing @), as before.
# * an empty element between two bars is kept wherever it stands.
#
# The rest(N) cast (kutil.rest_split) is SKS Python and is tested at that
# tier; here the value is read back raw through *name* substitution.
#
# Engine tier: --klammersets none, fixtures inline.
#
# Usage: ./rest_test.sh (LSan suppressions come from env/runtime.env)
# Exit code: 0 if all tests pass, 1 otherwise.
PASS=0
FAIL=0
KTEXT=ktext
K=${KLAMMERTEXT_HOME:?KLAMMERTEXT_HOME must be set}
red=$'\033[31m'
green=$'\033[32m'
reset=$'\033[0m'
trim() { awk '{ sub(/[ \t\r]+$/, "") } { line[NR]=$0 } END { f=1; while (f<=NR && line[f]=="") f++; l=NR; while (l>=1 && line[l]=="") l--; for (i=f;i<=l;i++) print line[i] }'; }
# check_eq TEST_NAME EXPECTED INPUT
# The fixtures: @zz takes only a rest, @yy one positional then a rest.
# Each prints its arguments in brackets, so the bar structure is visible.
check_eq() {
local test_name="$1" expected="$2" input="$3"
local output status
output=$("$KTEXT" --klammersets none -d -s "@@zz items.rest :x 0 : [*items*] @@ @@yy first | second.rest : [*first*][*second*] @@ $input" 2>/tmp/rest_test_err.$$)
status=$?
output=$(printf '%s' "$output" | trim)
if [ $status -ne 0 ]; then
echo "${red}FAIL${reset} $test_name — ktext exited $status"
echo " stderr: $(head -3 /tmp/rest_test_err.$$)"
FAIL=$((FAIL + 1))
return
fi
if [ "$output" = "$expected" ]; then
echo "${green}PASS${reset} $test_name"
PASS=$((PASS + 1))
else
echo "${red}FAIL${reset} $test_name"
echo " expected: [$expected]"
echo " got: [$output]"
FAIL=$((FAIL + 1))
fi
}
echo "-- the leading bar --"
check_eq " 1. no leading bar: the elements as written" '[M | F]' '@zz M | F @'
check_eq " 2. the opening-bar idiom: one bar stripped" '[M | F]' '@zz | M | F @'
check_eq " 3. a second leading bar is an empty first element" '[| M | F]' '@zz | | M | F @'
check_eq " 4. the idiom with the rows below, empty corner cell" '[| M | F ||
Nom | der]' '@zz |
| M | F ||
Nom | der @'
check_eq " 5. options first, then the idiom bar" '[M | F]' '@zz :x 1 | M | F @'
check_eq " 6. options first, then an empty first element" '[| M | F]' '@zz :x 1 | | M | F @'
echo
echo "-- after a positional argument --"
check_eq " 7. the bar after the positional is the separator" '[a][b | c]' '@yy a | b | c @'
check_eq " 8. a second bar there is an empty first element" '[a][| b | c]' '@yy a | | b | c @'
echo
echo "-- the middle and the end --"
check_eq " 9. an empty element between bars is kept" '[M | | F]' '@zz M | | F @'
check_eq "10. one trailing bar is trimmed" '[M | F]' '@zz M | F | @'
check_eq "11. a row separator is never the stripped bar" '[|| M | F]' '@zz || M | F @'
echo
echo "================================"
echo "Passed: $PASS Failed: $FAIL"
rm -f /tmp/rest_test_err.$$
[ $FAIL -eq 0 ]