ack/klammertext
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

@link :section links to a section by its heading, with an alone value (from dev 831455f36fcd)

Browse Source 
A section link names the TEXT of a heading: :section alone means the current
file, a value names the file of the same @document that has it. Resolved by
@document at assembly in html and tex; errors for a missing file, a missing
title, a doubled title and a link outside a @document. The Markdown
converter reads a Markdown anchor as a section link to the heading its slug
names. Also from dev: the converter's four silent defects fixed and --chapter
(9ae2c4cd), and the engine's alone-value error text.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
ackandClaude Fable 5.1 committed 2026-10-12 20:55:39 +02:00
1 parent 32aa47d41c
commit f4dcdf3ffd
15 files changed
+594 -114
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

sks/tns/md_to_sks.py
+282 -65
View File
@@ -11,9 +11,13 @@ correspondence between a Markdown construct and a klammer is a property of
the klammer set. That is why this lives under sks/ rather than in mac/.
What cannot be converted exactly is recorded in the output as a
#[MD <kind>: ... ]# marker -- lossy, gap, or judgment -- so a converted
document carries its own worklist and "how far from done is this file" is a
grep. A file with no markers is inside the convertible subset.
#[MD <kind>: ... ]# marker -- lossy, gap, or judgment; with --chapter also
section -- so a converted document carries its own worklist and "how far
from done is this file" is a grep. A file with no markers is inside the
convertible subset. The marker is the contract: a construct this program
does not handle must leave one, never pass through as prose. It does not
hold yet -- see "What the reference implementation does not do" in the
spec for the constructs that still pass through unmarked.
First run: the Rectify user guide (1,805 lines), 2026-08-07. It rendered to
html and pdf without error or warning, with 99 markers -- 91 of them internal
@@ -22,20 +26,100 @@ vocabulary lists (4,200 lines, 152 tables), 2026-09-30, which rendered
without an error and with two silently wrong results -- an emphasis that
wrapped and a table's empty header corner -- so "renders clean" is not
"converted"; the fixes are in the spec's "What the second conversion
taught".
taught". A review against a probe document (2026-10-11) found twelve more
ways the output was silently wrong and eight constructs passed through
with no marker; four were fixed first (a ragged nested list dropped items,
a loose list became one @ol per item, the "@@" separator ran over @c
literals, an escaped bar split a table cell), and the probe became the
fixture sks/tst/md_to_sks_test.sh: the conversion, in both modes, must
equal a golden file line for line and then render. The golden files pin
only what is right; the remaining defects are listed in the spec and join
the probe as they are fixed.
Usage: python3 md_to_sks.py <input.md> <output.kt> [sibling.md ...]
Usage: python3 md_to_sks.py [--chapter] <input.md> <output.kt> [sibling.md ...]
The siblings are Markdown files converted alongside this one: a link to
one of them becomes a link to its html page (../<stem>/index.html, where
ktext writes it); a link to any other .md file is a gap.
The siblings are Markdown files converted alongside this one. Without
--chapter each file is a document of its own, and a link to a sibling
becomes a link to its html page (../<stem>/index.html, where ktext writes
it); a link to any other .md file is a gap.
With --chapter (Andy, 2026-10-12) the file is one CHAPTER of a book whose
@document is written by hand and names the chapters through :files: no
@document wrapper is emitted, no :title is lifted, and every heading maps
one level deeper -- "#" is @s1, the chapter's own heading. A link to a
sibling is then a link to another chapter of the same book, written
"@link <its title> :section @" (:section finds a section by its heading;
implemented 2026-10-12), the title being the sibling's first "#" heading
read from the file. A Markdown link targets a FILE, so the title is a
guess, and a "section" marker records it. The same mechanism converts a
Markdown anchor, "#some-heading" or "other.md#some-heading": the slug is
matched against the headings of the file it points into, and the link
names that heading's text -- again marked, since the match is a guess.
The motivating case was ~/projects/kokkoja/docs/, five topic files
cross-linked as "[A host's Kokkoja](host.md)".
"""
import re, sys, os, datetime
import re, sys, os, datetime, argparse
from collections import Counter
SPECIALS = "^@#|*" # ^ first: the other quotings introduce ^
marks = Counter()
SIBLINGS = set() # stems of the .md files converted together
SIBLINGS = {} # stem -> path of the .md files converted together
CHAPTER = False # --chapter: a chapter of a hand-written book
STEM = "" # the file being converted
HEADINGS = {} # stem -> [(slug, level, title as Klammertext)], lazily
def slug(text):
"""GitHub's anchor for a heading: lower case, punctuation other than
hyphen and underscore removed, every whitespace a hyphen. A duplicate
heading gets -1, -2 ... appended, which lookups retry without."""
return re.sub(r'\s', '-', re.sub(r'[^\w\s-]', '', text.lower()).strip())
def headings_of(stem):
"""The headings of a file converted alongside (or this one): the anchor
each would have in Markdown, its level, and the title AS THE CONVERTER
EMITS IT -- manual number stripped, inline Markdown converted -- which
is the text a :section link must name. Headings inside fenced code
are not headings."""
if stem in HEADINGS:
return HEADINGS[stem]
found = []
try:
with open(SIBLINGS[stem], encoding="utf-8") as f:
fenced = False
for line in f:
line = line.rstrip("\n")
if FENCE.match(line):
fenced = not fenced
continue
m = HEADING.match(line)
if m and not fenced:
text = m.group(2)
title = MANUAL_NUMBER.sub("", text) if MANUAL_NUMBER.match(text) else text
found.append((slug(text), len(m.group(1)), title))
except (OSError, KeyError):
pass
HEADINGS[stem] = found
return found
def find_heading(stem, anchor):
"""The (level, title) a Markdown anchor names in STEM's file, or None.
An anchor made unique with -N is retried as its base."""
for s_, level, title in headings_of(stem):
if s_ == anchor:
return level, title
m = re.match(r'^(.*)-\d+$', anchor)
if m:
for s_, level, title in headings_of(stem):
if s_ == m.group(1):
return level, title
return None
def sibling_title(stem, fallback):
"""A sibling chapter's title: its first "#" heading."""
for _, level, title in headings_of(stem):
if level == 1:
return title
return fallback
def mark(kind, text):
"""A #[MD ... ]# marker.
@@ -115,17 +199,48 @@ def inline(text):
# loss. The marker goes BEFORE the words: a link is usually
# followed by punctuation, and a marker between the words and the
# period leaves the space before the marker standing ("words .").
def section_link(title, stem, note):
"""A link to a section by its title (@link :section, resolved by
@document; 2026-10-12). :section alone means this file; a
heading in another chapter names that file as the value. The
label is kept as :text only where it differs from the title.
The title is a GUESS from a Markdown anchor or a file, so the
marker stays."""
where = "" if stem == STEM else " " + stem
text = "" if label == title else " :text " + emphasis(label)
return (keep(mark("section", note)) + " " +
opening("@link " + inline(title) + " :section" + where + text) + closing())
if target.startswith("#"):
# An anchor into this document: no klammer gives an arbitrary
# heading an anchor.
return keep(mark("gap", f"link to {target}")) + " " + emphasis(label)
if target.endswith(".md") and "/" not in target:
# Another Markdown file. "@link X.md" would be read as an
# anchor (href="#X.md"). Converted alongside this one, its html
# page is X/index.html beside ours; otherwise it is a gap.
stem = target[:-3]
# An anchor into this file: Markdown's slug of a heading. The
# heading it names becomes a :section link to that heading's
# text; an anchor matching no heading is the gap it always was.
found = find_heading(STEM, target[1:])
if found:
return section_link(found[1], STEM, f'link to {target}, read as the heading "{found[1]}"')
return keep(mark("gap", f"link to {target}, no such heading")) + " " + emphasis(label)
m_file = re.match(r'^([^/#]+)\.md(?:#(.+))?$', target)
if m_file:
# Another Markdown file, with or without an anchor into it.
# "@link X.md" would be read as an anchor (href="#X.md").
stem, anchor = m_file.group(1), m_file.group(2)
if stem not in SIBLINGS:
return keep(mark("gap", f"link to {target}, not converted")) + " " + emphasis(label)
if CHAPTER:
# Another chapter of the same book: a link to a section by
# its title -- the heading the anchor names, or the file's
# own title (its first "#") when there is no anchor.
if anchor:
found = find_heading(stem, anchor)
if not found:
return keep(mark("gap", f"link to {target}, no such heading")) + " " + emphasis(label)
return section_link(found[1], stem, f'link to {target}, read as the heading "{found[1]}"')
title = sibling_title(stem, label)
return section_link(title, stem, f'link to {target}, read as its title "{title}"')
if anchor:
# Converted as separate documents, the other page's ids are
# not known here.
return keep(mark("gap", f"link to {target}: a heading on another page")) + " " + emphasis(label)
target = f"../{stem}/index.html"
return opening("@link " + quote(target) + " :text ") + emphasis(label) + closing()
@@ -155,6 +270,34 @@ def inline(text):
text = "".join(p if re.match(r'^[\x00\x01]', p) else quote(p) for p in parts)
text = re.sub(r'\x01(\d+)\x01', lambda m: verbatim[int(m.group(1))], text)
# The code spans stay placeholders through the two fixes below, so that
# neither touches their content: a code span is LITERAL, and the "@@"
# rule once rewrote a Klammertext example inside one ("`@@name.k`"
# rendered as "@ @name.k"). Only the delimiter sentinels go on now; the
# content comes back last.
text = re.sub(r'(\x00\d+\x00)', OPEN + r'\1' + CLOSE, text)
# Klammertext delimiters need whitespace around them, but Markdown
# emphasis abuts its neighbours: "un**bold**ed" and "*name*_rectified"
# both put a word character hard against a delimiter, which the
# katomizer then reads as an unparsable word. A space makes it parse,
# and "#-" removes that space again from the OUTPUT, so intra-word
# emphasis converts exactly rather than approximately.
# EXCEPT where the katomizer splits the neighbour off itself (kdesc
# --rewrite, the "punctuation" and "klammer" categories): trailing
# punctuation after a closing "@" -- "It is @i italicized @." -- and
# an opening bracket before an opening "@name". Those are legal as
# written, and the "#-" was noise (Andy, 2026-10-12). Everything else
# that is not a space needs the fix: a word character, a quote, an
# underscore ("*name*_x"), a digit.
text = re.sub(r'(?<=[^\s(\[{])' + OPEN, ' #-', text).replace(OPEN, "")
text = re.sub(CLOSE + r"(?=[^\s\]})(.,:;?'!])", '#- ', text).replace(CLOSE, "")
# A closing "@" abutting the next klammer's opening "@" spells "@@",
# which is a DEFINITION delimiter -- the engine then reports a span that
# ends without a beginning, pointing at text the writer never wrote.
# Adjacent klammers are ordinary in converted prose (*a*_b_ produces
# two), so separate them. The code spans are still placeholders here,
# on purpose (see above): a "@@" inside one is the author's.
text = re.sub(r'@(?=@)', '@ ', text)
# Inline code goes in RAW, and is closed with the named delimiter. "@c"
# took an ordinary string parameter until 2026-08-16, so every Klammertext
# special inside a code span had to be quoted -- and about 40% of the
@@ -164,24 +307,8 @@ def inline(text):
# source reads as the author wrote it. A literal klammer cannot be closed
# with a bare "@" -- the machine cannot find the end by counting -- so the
# close is "c@".
text = re.sub(r'\x00(\d+)\x00',
lambda m: OPEN + "@c " + spans[int(m.group(1))] + " c@" + CLOSE, text)
# Klammertext delimiters need whitespace around them, but Markdown
# emphasis abuts its neighbours: "un**bold**ed" and "*name*_rectified"
# both put a word character hard against a delimiter, which the
# katomizer then reads as an unparsable word. A space makes it parse,
# and "#-" removes that space again from the OUTPUT, so intra-word
# emphasis converts exactly rather than approximately.
# Any non-space neighbour, not just a word character: "*name*_x" and
# "_1.*ext*" put an underscore or a period against the delimiter.
text = re.sub(r'(?<=\S)' + OPEN, ' #-', text).replace(OPEN, "")
text = re.sub(CLOSE + r'(?=[^\s])', '#- ', text).replace(CLOSE, "")
# A closing "@" abutting the next klammer's opening "@" spells "@@",
# which is a DEFINITION delimiter -- the engine then reports a span that
# ends without a beginning, pointing at text the writer never wrote.
# Adjacent klammers are ordinary in converted prose (*a*_b_ produces
# two), so separate them.
return re.sub(r'@(?=@)', '@ ', text)
return re.sub(r'\x00(\d+)\x00',
lambda m: "@c " + spans[int(m.group(1))] + " c@", text)
# ---- blocks ---------------------------------------------------------------
@@ -193,8 +320,30 @@ ORDERED = re.compile(r'^(\s*)(\d+)[.)]\s+(.*)$')
TABLEROW = re.compile(r'^\s*\|(.*)\|\s*$')
ALIGNROW = re.compile(r'^\s*\|[\s:|-]+\|\s*$')
def list_marker(line):
"""The marker of an item line -- its bullet character, or the delimiter
after its number ("." or ")") -- and None for a line that is no item.
CommonMark starts a NEW list when the marker changes."""
m = BULLET.match(line)
if m:
return line[len(m.group(1))]
m = ORDERED.match(line)
if m:
return line[len(m.group(1)) + len(m.group(2))]
return None
def cells(line):
return [c.strip() for c in TABLEROW.match(line).group(1).split("|")]
"""The cells of one table row.
A bar escaped as "\\|" is cell CONTENT -- GFM requires the escape even
inside a code span -- so the split is on the unescaped bars only, and
the backslash is then dropped: inline() quotes a prose bar to "^|", and
a bar inside a code span goes into the @c literal as it is. Splitting
on every bar shifted the header of any table that documents Klammertext
(whose own tables are full of "\\|") one column right, silently.
"""
inner = TABLEROW.match(line).group(1)
return [c.strip().replace("\\|", "|") for c in re.split(r'(?<!\\)\|', inner)]
def convert(path):
src = open(path, encoding="utf-8").read().split("\n")
@@ -224,8 +373,18 @@ def convert(path):
m = HEADING.match(line)
if m:
level, text = len(m.group(1)), m.group(2)
if level == 1 and title is None:
if CHAPTER:
level += 1 # "#" is the chapter's heading, @s1
elif level == 1 and title is None:
title = inline(text); i += 1; continue
elif level == 1:
# A second "#": not a title. Probably a part, or the file
# is two documents; neither is the converter's call, so it
# is a top-level section and the marker says so. (It was
# "@s0", which does not exist, and failed at render.)
out.append(mark("judgment", "a second top-level heading was "
"made @s1; it may be a @part, or a second document"))
level = 2
if MANUAL_NUMBER.match(text):
text = MANUAL_NUMBER.sub("", text); stripped_numbers += 1
if re.match(r'^contents$', text, re.I):
@@ -284,10 +443,32 @@ def convert(path):
continue
if BULLET.match(line) or ORDERED.match(line):
block, base = [], None
while i < n and (BULLET.match(src[i]) or ORDERED.match(src[i]) or
(src[i].strip() and src[i].startswith((" ", "\t")))):
block.append(src[i]); i += 1
block = []
while i < n:
if (BULLET.match(src[i]) or ORDERED.match(src[i]) or
(src[i].strip() and src[i].startswith((" ", "\t")))):
block.append(src[i]); i += 1
elif not src[i].strip():
# A blank line ends the list UNLESS the next non-blank
# line is another item of it: a loose list (items
# separated by blank lines) is one list, and ending it
# here made every item its own @ol, each numbered 1.
# "Of it" is CommonMark's rule: an indented item (a
# nested one) or one with the same marker; a different
# marker starts a new list. An indented non-item after
# the blank (a code block inside the item) still ends
# the list, as it always did.
j = i
while j < n and not src[j].strip():
j += 1
if j < n and list_marker(src[j]) and (
src[j].startswith((" ", "\t")) or
list_marker(src[j]) == list_marker(block[0])):
i = j
else:
break
else:
break
out.append(convert_list(block))
out.append("")
continue
@@ -329,39 +510,70 @@ def convert_list(block):
parts.append(body)
return klammer + " " + " | ".join(parts) + " @"
def parse(lines, indent):
def parse(lines):
"""The items at one level, each with its children.
An item's children are the item lines that begin at or past its
CONTENT column (where the text after the marker starts, as in
CommonMark); an item line that begins before that is the next item
at THIS level, however it is indented. The loop never stops on an
indent: the caller has already bounded the lines, so there is
nothing an indent could hand back to. The earlier rule took the
first child's indent as the sublist's and broke off at anything
shallower, so " - a1" followed by " - a2" DROPPED a2 -- and the
document rendered without a word about it.
"""
items, i = [], 0
ordered = bool(ORDERED.match(lines[0])) if lines else False
while i < len(lines):
m = BULLET.match(lines[i]) or ORDERED.match(lines[i])
if not m or len(m.group(1)) < indent:
break
if not m: # a stray wrapped line: previous item's
if items:
text, children = items[-1]
items[-1] = (text + " " + lines[i].strip(), children)
i += 1
continue
text = m.groups()[-1]
content_col = m.start(m.lastindex)
i += 1
child_lines = []
while i < len(lines):
m2 = BULLET.match(lines[i]) or ORDERED.match(lines[i])
if m2 and len(m2.group(1)) > indent:
if m2 and len(m2.group(1)) >= content_col:
child_lines.append(lines[i]); i += 1
elif not m2 and lines[i].strip():
text += " " + lines[i].strip(); i += 1
else:
elif m2:
break
children = None
if child_lines:
sub, sub_ordered = parse(child_lines, len(BULLET.match(child_lines[0]).group(1))
if BULLET.match(child_lines[0])
else len(ORDERED.match(child_lines[0]).group(1)))
children = (sub, sub_ordered)
elif child_lines: # a wrapped line of the last child
child_lines.append(lines[i]); i += 1
else: # a wrapped line of this item
text += " " + lines[i].strip(); i += 1
children = parse(child_lines) if child_lines else None
items.append((text, children))
return items, ordered
items, ordered = parse([l for l in block if l.strip()], 0)
items, ordered = parse([l for l in block if l.strip()])
return emit(items, ordered, 0)
def main():
src_path, dst_path = sys.argv[1], sys.argv[2]
SIBLINGS.update(os.path.splitext(os.path.basename(a))[0] for a in sys.argv[3:])
global CHAPTER
ap = argparse.ArgumentParser(
description="Convert a Markdown file to the Standard Klammer Set.",
epilog="Siblings are the .md files converted alongside; a link to one "
"is converted as a link to its page (or, with --chapter, to "
"its section), a link to any other .md file is a gap.")
ap.add_argument("--chapter", action="store_true",
help="the file is a chapter of a hand-written @document: "
"no wrapper, '#' maps to @s1, sibling links are :section links")
ap.add_argument("input", metavar="input.md")
ap.add_argument("output", metavar="output.kt")
ap.add_argument("siblings", metavar="sibling.md", nargs="*")
args = ap.parse_args()
global STEM
CHAPTER = args.chapter
src_path, dst_path = args.input, args.output
STEM = os.path.splitext(os.path.basename(src_path))[0]
SIBLINGS.update((os.path.splitext(os.path.basename(a))[0], a) for a in args.siblings)
SIBLINGS[STEM] = src_path # its own headings resolve "#anchor" links
body, title, stripped, dropped_toc = convert(src_path)
today = datetime.date.today().isoformat()
head = [f"#[MD source: {os.path.abspath(src_path)}",
@@ -372,15 +584,20 @@ def main():
'("2.1 ..."); the numbers were removed because @s1/@s2 '
"number the headings themselves"))
head.append("")
head += ["@document", ":structure plain"]
if title:
head.append(f":title {title}")
head += [":text", ""]
text = "\n".join(head + body + ["", "@", ""])
if CHAPTER:
tail = [""]
else:
head += ["@document", ":structure plain"]
if title:
head.append(f":title {title}")
head += [":text", ""]
tail = ["", "@", ""]
text = "\n".join(head + body + tail)
text = re.sub(r'\n{3,}', "\n\n", text)
open(dst_path, "w", encoding="utf-8").write(text)
print(f"{dst_path}: {len(text.splitlines())} lines")
for kind in ("source", "lossy", "gap", "judgment"):
print(f" {kind:9} {marks[kind]}")
for kind in ("lossy", "gap", "judgment", "section"):
if kind != "section" or CHAPTER:
print(f" {kind:9} {marks[kind]}")
main()