Andy Kopra
5f21d9099c
Direct-framebuffer ink pipeline (stock-quality strokes), finger-wipe erase, growable scrolling canvas with color-ghost cleanup, bidirectional toggle with a persistent 4-finger return launcher, instant button feedback. Includes prebuilt aarch64 binaries (dist/), build/deploy/install scripts, a user guide, and a complete technical reference. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
80 lines
4.0 KiB
Markdown
80 lines
4.0 KiB
Markdown
# Chatter — Stylus & Pen-Rendering Research
|
||
|
||
A record of what we learned trying to reproduce the reMarkable stock pen
|
||
behavior in Chatter, so we can resume this later. (Status 2026-06-26: the solid
|
||
ink pipeline is done and shipping; calligraphy is "close enough for now" per the
|
||
user and is **not essential** for the conversational-assistance use case.)
|
||
|
||
## 1. The render pipeline (solved)
|
||
|
||
The reMarkable epaper Qt backend is a **software scene-graph renderer**:
|
||
- Custom `QSGGeometryNode`s are silently dropped — they don't render.
|
||
- `QQuickPaintedItem` textures render but always via the e-ink **dashed
|
||
two-pass** refresh, with latency. Screen mode (Pen/Mono) and antialiasing do
|
||
not change this.
|
||
- **Solution:** bypass Qt for ink. Capture the framebuffer `QImage` by
|
||
interposing `EPFramebuffer::setBuffers` (cross-DSO, interposable; the
|
||
executable is linked `-Wl,--export-dynamic` so its symbol wins). Draw strokes
|
||
straight into that buffer (`InkEngine`) and refresh each segment with an
|
||
explicit `EPFramebuffer::swapBuffers(rect, …)` — which renders **solid,
|
||
single-pass**. Qt Quick is used only for the static UI.
|
||
- Buffer A is `960×1696 RGB32` (logical screen 954×1696, padded to 960);
|
||
`bytesPerLine 3840`. There is **no PNG image-format plugin** on the device
|
||
(only gif/ico/jpeg/svg) — save snapshots as **BMP**.
|
||
|
||
## 2. The Calligraphy pen model
|
||
|
||
reMarkable's Calligraphy is **not a fixed geometric/flat nib**. Confirmed by
|
||
experiment + reMarkable docs: it's a **dynamic** model combining:
|
||
- **Stroke direction** — downstrokes thick, upstrokes thin. (User's stock-pen
|
||
circles: thick on the descending side — right for clockwise, left for CCW. A
|
||
4-direction asterisk showed *no* variation, because quick uniform straight
|
||
strokes don't trigger it.)
|
||
- **Pressure** — heavy pressure ≈ doubles the width.
|
||
- **Speed** — faster = thinner.
|
||
- **Tilt/orientation** — simulates an angled nib.
|
||
|
||
The exact angle/algorithm is compiled into xochitl's proprietary brush engine
|
||
(assets `LS_Calligraphy_*` / `P_Calligraphy_*`, `rm-brushgfx`) — **not** in a
|
||
readable config, so exact duplication would need deep reverse-engineering.
|
||
|
||
## 3. Width calibration
|
||
|
||
- `LastPenSize` (from `xochitl.conf` `LastWritingTool`) is only a **category**:
|
||
1/2/3 = thin/thicker/thickest. Not a pixel width.
|
||
- Measured on the physical tablet: **thicker(2) ≈ 1 mm, thickest(3) ≈ 2 mm,
|
||
thinnest ≈ 2 px**. Pressure can **almost double** the width.
|
||
- Display is **264 PPI → 264/25.4 ≈ 10.4 px/mm.**
|
||
- Chatter mapping (`main.cpp`): `maxMm = max(0.2, (size-1)·1.0)` →
|
||
size3 = 2 mm, size2 = 1 mm; `setWidthRange(2 px, maxMm·pxPerMm)`.
|
||
|
||
## 4. Chatter's current approximation (`InkEngine`, pen type 21)
|
||
|
||
```
|
||
f = 0.25·dirF + 0.50·pressure + 0.12·speedF + 0.13·tiltMag // clamp 0..1
|
||
width = minWidth + (maxWidth − minWidth)·f
|
||
dirF = 0.5 + 0.5·(Δy/len) // downstroke → 1, upstroke → 0
|
||
speedF = 1 − clamp(speed/3, 0, 1) // speed = len/dt (px/ms), slow → 1
|
||
tiltMag = clamp(hypot(tiltX, tiltY), 0, 1) // from event2 ABS_TILT_X/Y (±9000)
|
||
```
|
||
Uniform (non-calligraphy) pens: `f = 0.5 + 0.5·pressure`.
|
||
`PenDevice` emits `strokeMove(pos, pressure, tiltX, tiltY, eraser)`.
|
||
|
||
## 5. Tooling (in `tools/`)
|
||
|
||
- `setbufshim.cpp` — LD_PRELOAD shim; proved `setBuffers` capture (feasibility).
|
||
- `fbdump.cpp` — LD_PRELOAD into xochitl; snapshots the framebuffer to
|
||
`/home/root/fbdump.{png,bmp}` when `/tmp/fbdump` is touched (use BMP — no PNG
|
||
plugin). Note: `setBuffers` interposition into *xochitl* via LD_PRELOAD was
|
||
flaky/unconfirmed in one attempt; the in-process `--export-dynamic` capture in
|
||
Chatter itself is reliable.
|
||
|
||
## 6. Open questions / ways to go deeper later
|
||
|
||
- Recover exact reMarkable widths-per-size and the nib/pressure curves by
|
||
snapshotting stock strokes (fix the xochitl fbdump path) and measuring, or by
|
||
parsing the v6 `.rm` per-point width/direction data.
|
||
- Add a proper speed estimate (smoothed) and tilt-direction (not just magnitude).
|
||
- Confirm `LastPenSize` values for the thin/medium categories (only size 3 = 3.0
|
||
is confirmed).
|