Initial commit: Chatter — assistive-writing app for reMarkable Paper Pro Move

Direct-framebuffer ink pipeline (stock-quality strokes), finger-wipe erase,
growable scrolling canvas with color-ghost cleanup, bidirectional toggle with a
persistent 4-finger return launcher, instant button feedback. Includes prebuilt
aarch64 binaries (dist/), build/deploy/install scripts, a user guide, and a
complete technical reference.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-27 16:34:28 +02:00
commit 5f21d9099c
43 changed files with 2878 additions and 0 deletions

View File

@@ -0,0 +1,79 @@
# Chatter — Stylus & Pen-Rendering Research
A record of what we learned trying to reproduce the reMarkable stock pen
behavior in Chatter, so we can resume this later. (Status 2026-06-26: the solid
ink pipeline is done and shipping; calligraphy is "close enough for now" per the
user and is **not essential** for the conversational-assistance use case.)
## 1. The render pipeline (solved)
The reMarkable epaper Qt backend is a **software scene-graph renderer**:
- Custom `QSGGeometryNode`s are silently dropped — they don't render.
- `QQuickPaintedItem` textures render but always via the e-ink **dashed
two-pass** refresh, with latency. Screen mode (Pen/Mono) and antialiasing do
not change this.
- **Solution:** bypass Qt for ink. Capture the framebuffer `QImage` by
interposing `EPFramebuffer::setBuffers` (cross-DSO, interposable; the
executable is linked `-Wl,--export-dynamic` so its symbol wins). Draw strokes
straight into that buffer (`InkEngine`) and refresh each segment with an
explicit `EPFramebuffer::swapBuffers(rect, …)` — which renders **solid,
single-pass**. Qt Quick is used only for the static UI.
- Buffer A is `960×1696 RGB32` (logical screen 954×1696, padded to 960);
`bytesPerLine 3840`. There is **no PNG image-format plugin** on the device
(only gif/ico/jpeg/svg) — save snapshots as **BMP**.
## 2. The Calligraphy pen model
reMarkable's Calligraphy is **not a fixed geometric/flat nib**. Confirmed by
experiment + reMarkable docs: it's a **dynamic** model combining:
- **Stroke direction** — downstrokes thick, upstrokes thin. (User's stock-pen
circles: thick on the descending side — right for clockwise, left for CCW. A
4-direction asterisk showed *no* variation, because quick uniform straight
strokes don't trigger it.)
- **Pressure** — heavy pressure ≈ doubles the width.
- **Speed** — faster = thinner.
- **Tilt/orientation** — simulates an angled nib.
The exact angle/algorithm is compiled into xochitl's proprietary brush engine
(assets `LS_Calligraphy_*` / `P_Calligraphy_*`, `rm-brushgfx`) — **not** in a
readable config, so exact duplication would need deep reverse-engineering.
## 3. Width calibration
- `LastPenSize` (from `xochitl.conf` `LastWritingTool`) is only a **category**:
1/2/3 = thin/thicker/thickest. Not a pixel width.
- Measured on the physical tablet: **thicker(2) ≈ 1 mm, thickest(3) ≈ 2 mm,
thinnest ≈ 2 px**. Pressure can **almost double** the width.
- Display is **264 PPI → 264/25.4 ≈ 10.4 px/mm.**
- Chatter mapping (`main.cpp`): `maxMm = max(0.2, (size-1)·1.0)`
size3 = 2 mm, size2 = 1 mm; `setWidthRange(2 px, maxMm·pxPerMm)`.
## 4. Chatter's current approximation (`InkEngine`, pen type 21)
```
f = 0.25·dirF + 0.50·pressure + 0.12·speedF + 0.13·tiltMag // clamp 0..1
width = minWidth + (maxWidth minWidth)·f
dirF = 0.5 + 0.5·(Δy/len) // downstroke → 1, upstroke → 0
speedF = 1 clamp(speed/3, 0, 1) // speed = len/dt (px/ms), slow → 1
tiltMag = clamp(hypot(tiltX, tiltY), 0, 1) // from event2 ABS_TILT_X/Y (±9000)
```
Uniform (non-calligraphy) pens: `f = 0.5 + 0.5·pressure`.
`PenDevice` emits `strokeMove(pos, pressure, tiltX, tiltY, eraser)`.
## 5. Tooling (in `tools/`)
- `setbufshim.cpp` — LD_PRELOAD shim; proved `setBuffers` capture (feasibility).
- `fbdump.cpp` — LD_PRELOAD into xochitl; snapshots the framebuffer to
`/home/root/fbdump.{png,bmp}` when `/tmp/fbdump` is touched (use BMP — no PNG
plugin). Note: `setBuffers` interposition into *xochitl* via LD_PRELOAD was
flaky/unconfirmed in one attempt; the in-process `--export-dynamic` capture in
Chatter itself is reliable.
## 6. Open questions / ways to go deeper later
- Recover exact reMarkable widths-per-size and the nib/pressure curves by
snapshotting stock strokes (fix the xochitl fbdump path) and measuring, or by
parsing the v6 `.rm` per-point width/direction data.
- Add a proper speed estimate (smoothed) and tilt-direction (not just magnitude).
- Confirm `LastPenSize` values for the thin/medium categories (only size 3 = 3.0
is confirmed).