A rest argument keeps an empty first element; a removal block keeps its neighbour; the Markdown converter reads paragraphs whole; Smart App Control on the Windows page (from dev 1083db946f05)
- A rest argument (a list's or a table's items) now strips exactly ONE leading bar -- the separator from what precedes it -- so a second one is an empty first element: a table with an empty header corner is written `@table |` then ` | Maskulinum | ...`. Before, every leading bar was dropped and such a header row shifted one column left. New shipped suite tst/rest_test.sh. - Text written hard against a closing ]# (a period, a letter) is no longer removed with the block; a following space or line break still is. - sks/tns/md_to_sks.py converts inline Markdown per paragraph, so emphasis that wraps is converted and not mis-paired; bold may contain italic; tables are unnumbered; links to sibling .md files converted alongside become links to their html pages; the arrow becomes @rightarrow@. - doc/install/windows_source_install.md: Smart App Control must be off -- it blocks MSYS2's unsigned toolchain, and its Evaluation state turns itself on later.
This commit is contained in:
1 parent
6e9889f7ae
commit
bc2b3d8461
8 files changed
+239
-18
No files matched your search
@@ -54,7 +54,7 @@ are regenerated on each release — patches cannot be merged directly.
|
||||
Report problems (or send patches) to the author; accepted changes are
|
||||
applied to the development tree and appear in a following snapshot.
|
||||
|
||||
This snapshot was assembled from development commit `1ce461b48a7d`.
|
||||
This snapshot was assembled from development commit `1083db946f05`.
|
||||
|
||||
## License
|
||||
|
||||
|
||||
@@ -28,6 +28,24 @@ special rights. Allow an hour or two the first time: the downloads are
|
||||
about a gigabyte in all, TeX Live alone takes twenty to thirty minutes,
|
||||
and the build a few minutes. Nothing here needs to be done twice.
|
||||
|
||||
**Smart App Control must be off.** A clean installation of Windows 11
|
||||
runs Smart App Control, a security feature that blocks any program
|
||||
Microsoft has not seen signed and widespread — and the MSYS2 compiler,
|
||||
`make` and the `bash` shell on which this page depends are not signed, so
|
||||
with it on the MSYS2 shell does not open and the build cannot start
|
||||
("An Application Control policy has blocked this file"). Check it first:
|
||||
Windows Security → **App & browser control** → **Smart App Control
|
||||
settings**. It is in one of three states. *Off* is what you need. *On*
|
||||
blocks MSYS2; switch it off. *Evaluation* lets everything run for now,
|
||||
but Windows turns it on by itself when it decides the machine is used
|
||||
"safely", at which point a working installation stops without warning —
|
||||
so switch it off at once rather than wait. Switching it off cannot be
|
||||
undone short of reinstalling Windows, which is why Microsoft asks you to
|
||||
confirm; the rest of Windows Security (Defender, SmartScreen for
|
||||
downloads) stays on and is not affected. A machine upgraded from Windows
|
||||
10, or one on which an unsigned program has already been installed, has
|
||||
it off already.
|
||||
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -403,6 +421,11 @@ it (a new UCRT64 shell does, once the line is in `~/.bashrc`) and run
|
||||
`MSYS2_PATH_TYPE` is not set to `inherit`, or the shell was opened before
|
||||
you set it. Set it and open a new UCRT64 shell.
|
||||
|
||||
**"An Application Control policy has blocked this file", or the MSYS2
|
||||
shell does not open.** Smart App Control is on; see *Before you start*.
|
||||
It can turn itself on after the installation, in which case the
|
||||
installation is intact and only the setting needs changing.
|
||||
|
||||
**The build is slow.** Windows Defender scans every file the compiler
|
||||
writes. Excluding the repository folder from real-time scanning speeds the
|
||||
build up considerably; it is not required.
|
||||
+41
-3
@@ -225,6 +225,17 @@ katom_list trim_part(katom_list part)
|
||||
return trim(part, {katom_t::space, katom_t::newline, katom_t::bar});
|
||||
}
|
||||
|
||||
// The trailing half of trim_part(): the leading side of a rest argument is
|
||||
// decided bar by bar in argument_split().
|
||||
static katom_list trim_trailing_part(katom_list part)
|
||||
{
|
||||
const std::set<katom_t> trim_types {katom_t::space, katom_t::newline, katom_t::bar};
|
||||
while (!part.empty() && trim_types.contains(part.back().m_type)) {
|
||||
part.pop_back();
|
||||
}
|
||||
return part;
|
||||
}
|
||||
|
||||
std::tuple<katom_lists,katom_lists>
|
||||
parameter_split(katom_list::const_iterator kbegin, katom_list::const_iterator kend)
|
||||
{
|
||||
@@ -330,23 +341,50 @@ argument_split(katom_list::const_iterator kbegin, katom_list::const_iterator ken
|
||||
katom_lists optional {};
|
||||
katom_list rest {};
|
||||
|
||||
for (auto p : parts) {
|
||||
// function_symbol_parts() opens the first part with a bar of its own
|
||||
// when the arguments do not begin with an option. Without positional
|
||||
// parameters that bar is the first katom of the rest, and it is not
|
||||
// the writer's.
|
||||
bool rest_opens_synthetic = false;
|
||||
for (std::size_t i = 0; i < parts.size(); i++) {
|
||||
const auto& p = parts[i];
|
||||
if (p[0].m_type == katom_t::option_name) {
|
||||
optional.push_back(p);
|
||||
} else if (positional.size() < positional_limit) {
|
||||
positional.push_back(trim_part(p));
|
||||
} else {
|
||||
if (rest.empty()) {
|
||||
rest_opens_synthetic = (i == 0) && positional_limit == 0;
|
||||
} else {
|
||||
// Parts were trimmed, so adjacent parts would abut their bar
|
||||
// katoms (a row separator "||" next to an empty cell's "|"
|
||||
// would serialize as "|||"). A space keeps the writer's
|
||||
// bar/double-bar distinction parseable.
|
||||
if (!rest.empty()) {
|
||||
rest.push_back(Katom(" ", katom_t::space, p[0].m_loc));
|
||||
}
|
||||
rest.insert(rest.end(), p.begin(), p.end());
|
||||
}
|
||||
}
|
||||
return {positional, optional, trim_part(rest)};
|
||||
|
||||
// Exactly ONE written leading bar is stripped from the rest (decided
|
||||
// 2026-10-01): it is the bar that separates the rest from what
|
||||
// precedes it -- the last positional, the options, or the klammer name
|
||||
// in the opening-bar idiom `@table |` with the rows on the lines below.
|
||||
// Every further leading bar delimits an EMPTY FIRST ELEMENT the writer
|
||||
// meant (`@table | | M | F ||` has an empty corner cell), mirroring
|
||||
// kutil.rest_split, which strips one trailing delimiter and keeps all
|
||||
// other empty elements. Trailing bars and whitespace are trimmed as
|
||||
// before.
|
||||
rest = trim_whitespace(rest);
|
||||
if (rest_opens_synthetic && !rest.empty() && rest[0].m_type == katom_t::bar) {
|
||||
rest.erase(rest.begin());
|
||||
rest = trim_whitespace(rest);
|
||||
}
|
||||
if (!rest.empty() && rest[0].m_type == katom_t::bar && rest[0].m_text == "|") {
|
||||
rest.erase(rest.begin());
|
||||
rest = trim_whitespace(rest);
|
||||
}
|
||||
return {positional, optional, trim_trailing_part(rest)};
|
||||
}
|
||||
|
||||
// Parameter/argument mapping
|
||||
|
||||
@@ -554,13 +554,16 @@ void mark_ignored_katoms(katom_list& katoms)
|
||||
//auto is_ignore_begin = [](const Katom& k) { return k.m_type == katom_t::ignore_begin; };
|
||||
//auto is_ignore_end = [](const Katom& k) { return k.m_type == katom_t::ignore_end; };
|
||||
for (auto [op,cl] : find_spans(katoms, begin_ignore, end_ignore, true, "ignored")) {
|
||||
// find_span_katoms() returns a half-open range: `end` is the katom
|
||||
// AFTER the closing "]#", as every other caller reads it. This one
|
||||
// read it as inclusive until 2026-10-02 and so removed the katom
|
||||
// abutting the close: "#[...]#." lost its period, "#[...]#x" its x.
|
||||
// A following space or newline is removed by design (the loop
|
||||
// below, and here), which is what hid it.
|
||||
auto [begin, end] = find_span_katoms(katoms, op, cl);
|
||||
// std::for_each(begin, end + 1, [](Katom& k) { k.m_type = katom_t::ignored; });
|
||||
std::for_each(begin, end + 1, mark_as_ignored);
|
||||
//if (end + 1 != katoms.end() && (end+1)->m_type == katom_t::newline) {
|
||||
if (end + 1 != katoms.end() && is_newline(*(end + 1))) {
|
||||
//(end+1)->m_type = katom_t::ignored;
|
||||
mark_as_ignored(*(end + 1));
|
||||
std::for_each(begin, end, mark_as_ignored);
|
||||
if (end != katoms.end() && is_newline(*end)) {
|
||||
mark_as_ignored(*end);
|
||||
}
|
||||
}
|
||||
bool ignore_line = false;
|
||||
|
||||
+58
-8
@@ -17,15 +17,25 @@ grep. A file with no markers is inside the convertible subset.
|
||||
|
||||
First run: the Rectify user guide (1,805 lines), 2026-08-07. It rendered to
|
||||
html and pdf without error or warning, with 99 markers -- 91 of them internal
|
||||
anchor links, the one gap that appears at volume.
|
||||
anchor links, the one gap that appears at volume. Second run: three German
|
||||
vocabulary lists (4,200 lines, 152 tables), 2026-09-30, which rendered
|
||||
without an error and with two silently wrong results -- an emphasis that
|
||||
wrapped and a table's empty header corner -- so "renders clean" is not
|
||||
"converted"; the fixes are in the spec's "What the second conversion
|
||||
taught".
|
||||
|
||||
Usage: python3 md_to_sks.py <input.md> <output.kt>
|
||||
Usage: python3 md_to_sks.py <input.md> <output.kt> [sibling.md ...]
|
||||
|
||||
The siblings are Markdown files converted alongside this one: a link to
|
||||
one of them becomes a link to its html page (../<stem>/index.html, where
|
||||
ktext writes it); a link to any other .md file is a gap.
|
||||
"""
|
||||
import re, sys, os, datetime
|
||||
from collections import Counter
|
||||
|
||||
SPECIALS = "^@#|*" # ^ first: the other quotings introduce ^
|
||||
marks = Counter()
|
||||
SIBLINGS = set() # stems of the .md files converted together
|
||||
|
||||
def mark(kind, text):
|
||||
"""A #[MD ... ]# marker.
|
||||
@@ -56,7 +66,9 @@ def quote(s):
|
||||
|
||||
CODE = re.compile(r'`([^`\n]+)`')
|
||||
LINK = re.compile(r'(?<!\!)\[([^\]]*)\]\(([^)]+)\)')
|
||||
BOLD = re.compile(r'\*\*([^*]+)\*\*|__([^_]+)__')
|
||||
# A bold run may hold italic runs ("**only for *sein* and *haben***"): the
|
||||
# group admits a balanced *...* inside, so the closing ** is found past it.
|
||||
BOLD = re.compile(r'\*\*((?:[^*]|\*[^*\s][^*]*\*)+)\*\*|__([^_]+)__')
|
||||
# Markdown's underscore emphasis does not open or close INSIDE a word, and
|
||||
# technical prose is full of identifiers like *name*_rectified_1.*ext* where
|
||||
# the underscores are part of the name. So the closing _ must not be
|
||||
@@ -99,10 +111,22 @@ def inline(text):
|
||||
|
||||
def link(m):
|
||||
label, target = m.group(1), m.group(2)
|
||||
# A link that cannot be converted keeps its words and records the
|
||||
# loss. The marker goes BEFORE the words: a link is usually
|
||||
# followed by punctuation, and a marker between the words and the
|
||||
# period leaves the space before the marker standing ("words .").
|
||||
if target.startswith("#"):
|
||||
# An anchor into this document: no klammer gives an arbitrary
|
||||
# heading an anchor, so keep the words and record the loss.
|
||||
return emphasis(label) + " " + keep(mark("gap", f"link to {target}"))
|
||||
# heading an anchor.
|
||||
return keep(mark("gap", f"link to {target}")) + " " + emphasis(label)
|
||||
if target.endswith(".md") and "/" not in target:
|
||||
# Another Markdown file. "@link X.md" would be read as an
|
||||
# anchor (href="#X.md"). Converted alongside this one, its html
|
||||
# page is X/index.html beside ours; otherwise it is a gap.
|
||||
stem = target[:-3]
|
||||
if stem not in SIBLINGS:
|
||||
return keep(mark("gap", f"link to {target}, not converted")) + " " + emphasis(label)
|
||||
target = f"../{stem}/index.html"
|
||||
return opening("@link " + quote(target) + " :text ") + emphasis(label) + closing()
|
||||
|
||||
def emphasis(t):
|
||||
@@ -115,6 +139,9 @@ def inline(text):
|
||||
the wrong list: IndexError, found by converting a second document.
|
||||
"""
|
||||
t = LINK.sub(link, t)
|
||||
# An arrow is the SKS klammer (Andy, 2026-10-01): U+2192, and the
|
||||
# halfwidth U+FFEB his documents use.
|
||||
t = re.sub(r'[\u2192\uffeb]', lambda m: keep(OPEN + "@rightarrow@" + CLOSE), t)
|
||||
t = STRIKE.sub(lambda m: emphasis(m.group(1)) + " " +
|
||||
keep(mark("gap", "strikethrough")), t)
|
||||
t = BOLD.sub(lambda m: opening("@b ") + (m.group(1) or m.group(2)) + closing(), t)
|
||||
@@ -230,7 +257,9 @@ def convert(path):
|
||||
rows.append([inline(c) for c in cells(src[i])])
|
||||
i += 1
|
||||
body = " ||\n".join(" | ".join(r) for r in rows)
|
||||
opts = f" :cell_hpos {' '.join(align)}" if align and set(align) != {"l"} else ""
|
||||
# A Markdown table has no caption and no number.
|
||||
opts = " :number false"
|
||||
opts += f" :cell_hpos {' '.join(align)}" if align and set(align) != {"l"} else ""
|
||||
# A Markdown table says nothing about column widths, and
|
||||
# Klammertext's default "fit" never wraps -- a prose column then
|
||||
# runs off the page (the pdf target warns). Give the widest
|
||||
@@ -263,8 +292,28 @@ def convert(path):
|
||||
out.append("")
|
||||
continue
|
||||
|
||||
out.append(inline(line) if line.strip() else "")
|
||||
i += 1
|
||||
if not line.strip():
|
||||
out.append(""); i += 1
|
||||
continue
|
||||
|
||||
# A PARAGRAPH goes through inline() whole, its line breaks kept.
|
||||
# Converting line by line left a literal "*" on each line of an
|
||||
# emphasis that wrapped, and the leftovers then paired with EACH
|
||||
# OTHER, so the wrong words came out italic -- 33 paragraphs of the
|
||||
# vocabulary lists, visible only in a diff.
|
||||
para = [line]; i += 1
|
||||
while i < n and src[i].strip() and not (HEADING.match(src[i]) or
|
||||
TABLEROW.match(src[i]) or FENCE.match(src[i]) or
|
||||
BULLET.match(src[i])):
|
||||
if ORDERED.match(src[i]):
|
||||
# A numbered line hard against the line above is a wrapped
|
||||
# line that happens to begin with a number ("1. and 3.
|
||||
# Sg."). CommonMark reads it as a list; the writer did not
|
||||
# mean one. Kept in the paragraph, and said.
|
||||
out.append(mark("judgment", "a line beginning with a number "
|
||||
"was kept in its paragraph, not read as a list"))
|
||||
para.append(src[i]); i += 1
|
||||
out.append(inline("\n".join(para)))
|
||||
|
||||
return out, title, stripped_numbers, dropped_toc
|
||||
|
||||
@@ -312,6 +361,7 @@ def convert_list(block):
|
||||
|
||||
def main():
|
||||
src_path, dst_path = sys.argv[1], sys.argv[2]
|
||||
SIBLINGS.update(os.path.splitext(os.path.basename(a))[0] for a in sys.argv[3:])
|
||||
body, title, stripped, dropped_toc = convert(src_path)
|
||||
today = datetime.date.today().isoformat()
|
||||
head = [f"#[MD source: {os.path.abspath(src_path)}",
|
||||
|
||||
@@ -19,6 +19,7 @@ test:
|
||||
./katomize_test.sh
|
||||
./filename_test.sh
|
||||
./alone_test.sh
|
||||
./rest_test.sh
|
||||
./argtype_test.sh
|
||||
./state_test.sh
|
||||
./target_test.sh
|
||||
|
||||
@@ -151,6 +151,18 @@ check_warn "26. ... and is reported, not split" '@@w@@'
|
||||
# the first version of the guard broke it.
|
||||
check "27. @@ glued to #- still splits" '<x>' '@@v.* : <x> @@#-@v@'
|
||||
|
||||
echo
|
||||
echo "-- what abuts a closing ]# --"
|
||||
# A removal block ends at its "]#"; what follows is the writer's. Until
|
||||
# 2026-10-02 mark_ignored_katoms() read find_span_katoms()'s half-open end
|
||||
# as inclusive and removed the katom abutting the close as well: a sentence
|
||||
# ending "#[ note ]#." lost its period. A following space or newline is
|
||||
# removed by design, which is what hid it.
|
||||
check "28. punctuation after ]# survives" 'a . c' 'a #[gone]#. c'
|
||||
check "29. a letter after ]# survives" 'a x c' 'a #[gone]#x c'
|
||||
check "30. a space after ]# is removed" 'a . c' 'a #[gone]# . c'
|
||||
check "31. nested blocks, punctuation after" 'a . c' 'a #[ #[in]# out ]#. c'
|
||||
|
||||
echo
|
||||
echo "================================================================="
|
||||
echo "${bold}Results: $PASS passed, $FAIL failed${reset}"
|
||||
|
||||
Executable
+94
@@ -0,0 +1,94 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# rest_test.sh — Regression tests for the bar structure of a rest argument.
|
||||
#
|
||||
# A rest argument collects everything after the declared positional
|
||||
# arguments, bars included, and hands the structure on; a klammer reads the
|
||||
# bar count as dimension (| elements, || rows). The one thing the ENGINE
|
||||
# decides is where the structure begins and ends:
|
||||
#
|
||||
# * exactly ONE written leading bar is stripped — the bar that separates
|
||||
# the rest from what precedes it: the last positional argument, the
|
||||
# options, or the klammer name in the opening-bar idiom (`@table |` with
|
||||
# the rows on the lines below). Every further leading bar delimits an
|
||||
# EMPTY FIRST ELEMENT the writer meant: `@table | | M | F ||` has an
|
||||
# empty corner cell. Decided 2026-10-01; before it every leading bar
|
||||
# was trimmed, so an empty first element could not be written at all,
|
||||
# and a Markdown table's empty header corner shifted the header row one
|
||||
# column left.
|
||||
# * trailing bars and whitespace are trimmed (the customary dangling
|
||||
# separator before a closing @), as before.
|
||||
# * an empty element between two bars is kept wherever it stands.
|
||||
#
|
||||
# The rest(N) cast (kutil.rest_split) is SKS Python and is tested at that
|
||||
# tier; here the value is read back raw through *name* substitution.
|
||||
#
|
||||
# Engine tier: --klammersets none, fixtures inline.
|
||||
#
|
||||
# Usage: ./rest_test.sh (LSan suppressions come from env/runtime.env)
|
||||
# Exit code: 0 if all tests pass, 1 otherwise.
|
||||
|
||||
PASS=0
|
||||
FAIL=0
|
||||
KTEXT=ktext
|
||||
K=${KLAMMERTEXT_HOME:?KLAMMERTEXT_HOME must be set}
|
||||
|
||||
red=$'\033[31m'
|
||||
green=$'\033[32m'
|
||||
reset=$'\033[0m'
|
||||
|
||||
trim() { awk '{ sub(/[ \t\r]+$/, "") } { line[NR]=$0 } END { f=1; while (f<=NR && line[f]=="") f++; l=NR; while (l>=1 && line[l]=="") l--; for (i=f;i<=l;i++) print line[i] }'; }
|
||||
|
||||
# check_eq TEST_NAME EXPECTED INPUT
|
||||
# The fixtures: @zz takes only a rest, @yy one positional then a rest.
|
||||
# Each prints its arguments in brackets, so the bar structure is visible.
|
||||
check_eq() {
|
||||
local test_name="$1" expected="$2" input="$3"
|
||||
local output status
|
||||
output=$("$KTEXT" --klammersets none -d -s "@@zz items.rest :x 0 : [*items*] @@ @@yy first | second.rest : [*first*][*second*] @@ $input" 2>/tmp/rest_test_err.$$)
|
||||
status=$?
|
||||
output=$(printf '%s' "$output" | trim)
|
||||
if [ $status -ne 0 ]; then
|
||||
echo "${red}FAIL${reset} $test_name — ktext exited $status"
|
||||
echo " stderr: $(head -3 /tmp/rest_test_err.$$)"
|
||||
FAIL=$((FAIL + 1))
|
||||
return
|
||||
fi
|
||||
if [ "$output" = "$expected" ]; then
|
||||
echo "${green}PASS${reset} $test_name"
|
||||
PASS=$((PASS + 1))
|
||||
else
|
||||
echo "${red}FAIL${reset} $test_name"
|
||||
echo " expected: [$expected]"
|
||||
echo " got: [$output]"
|
||||
FAIL=$((FAIL + 1))
|
||||
fi
|
||||
}
|
||||
|
||||
echo "-- the leading bar --"
|
||||
check_eq " 1. no leading bar: the elements as written" '[M | F]' '@zz M | F @'
|
||||
check_eq " 2. the opening-bar idiom: one bar stripped" '[M | F]' '@zz | M | F @'
|
||||
check_eq " 3. a second leading bar is an empty first element" '[| M | F]' '@zz | | M | F @'
|
||||
check_eq " 4. the idiom with the rows below, empty corner cell" '[| M | F ||
|
||||
Nom | der]' '@zz |
|
||||
| M | F ||
|
||||
Nom | der @'
|
||||
check_eq " 5. options first, then the idiom bar" '[M | F]' '@zz :x 1 | M | F @'
|
||||
check_eq " 6. options first, then an empty first element" '[| M | F]' '@zz :x 1 | | M | F @'
|
||||
|
||||
echo
|
||||
echo "-- after a positional argument --"
|
||||
check_eq " 7. the bar after the positional is the separator" '[a][b | c]' '@yy a | b | c @'
|
||||
check_eq " 8. a second bar there is an empty first element" '[a][| b | c]' '@yy a | | b | c @'
|
||||
|
||||
echo
|
||||
echo "-- the middle and the end --"
|
||||
check_eq " 9. an empty element between bars is kept" '[M | | F]' '@zz M | | F @'
|
||||
check_eq "10. one trailing bar is trimmed" '[M | F]' '@zz M | F | @'
|
||||
check_eq "11. a row separator is never the stripped bar" '[|| M | F]' '@zz || M | F @'
|
||||
|
||||
echo
|
||||
echo "================================"
|
||||
echo "Passed: $PASS Failed: $FAIL"
|
||||
rm -f /tmp/rest_test_err.$$
|
||||
[ $FAIL -eq 0 ]
|
||||
Reference in new issue
Block a user