chatter/doc/Chatter_stylus_research.md

Chatter — Stylus & Pen-Rendering Research

A record of what we learned trying to reproduce the reMarkable stock pen behavior in Chatter, so we can resume this later. (Status 2026-06-26: the solid ink pipeline is done and shipping; calligraphy is "close enough for now" per the user and is not essential for the conversational-assistance use case.)

1. The render pipeline (solved)

The reMarkable epaper Qt backend is a software scene-graph renderer:

• Custom QSGGeometryNodes are silently dropped — they don't render.
• QQuickPaintedItem textures render but always via the e-ink dashed two-pass refresh, with latency. Screen mode (Pen/Mono) and antialiasing do not change this.
• Solution: bypass Qt for ink. Capture the framebuffer QImage by interposing EPFramebuffer::setBuffers (cross-DSO, interposable; the executable is linked -Wl,--export-dynamic so its symbol wins). Draw strokes straight into that buffer (InkEngine) and refresh each segment with an explicit EPFramebuffer::swapBuffers(rect, …) — which renders solid, single-pass. Qt Quick is used only for the static UI.
• Buffer A is 960×1696 RGB32 (logical screen 954×1696, padded to 960); bytesPerLine 3840. There is no PNG image-format plugin on the device (only gif/ico/jpeg/svg) — save snapshots as BMP.

2. The Calligraphy pen model

reMarkable's Calligraphy is not a fixed geometric/flat nib. Confirmed by experiment + reMarkable docs: it's a dynamic model combining:

• Stroke direction — downstrokes thick, upstrokes thin. (User's stock-pen circles: thick on the descending side — right for clockwise, left for CCW. A 4-direction asterisk showed no variation, because quick uniform straight strokes don't trigger it.)
• Pressure — heavy pressure ≈ doubles the width.
• Speed — faster = thinner.
• Tilt/orientation — simulates an angled nib.

The exact angle/algorithm is compiled into xochitl's proprietary brush engine (assets LS_Calligraphy_* / P_Calligraphy_*, rm-brushgfx) — not in a readable config, so exact duplication would need deep reverse-engineering.

3. Width calibration

• LastPenSize (from xochitl.conf LastWritingTool) is only a category: 1/2/3 = thin/thicker/thickest. Not a pixel width.
• Measured on the physical tablet: thicker(2) ≈ 1 mm, thickest(3) ≈ 2 mm, thinnest ≈ 2 px. Pressure can almost double the width.
• Display is 264 PPI → 264/25.4 ≈ 10.4 px/mm.
• Chatter mapping (main.cpp): maxMm = max(0.2, (size-1)·1.0) → size3 = 2 mm, size2 = 1 mm; setWidthRange(2 px, maxMm·pxPerMm).

4. Chatter's current approximation (InkEngine, pen type 21)

f      = 0.25·dirF + 0.50·pressure + 0.12·speedF + 0.13·tiltMag   // clamp 0..1
width  = minWidth + (maxWidth − minWidth)·f
  dirF    = 0.5 + 0.5·(Δy/len)     // downstroke → 1, upstroke → 0
  speedF  = 1 − clamp(speed/3, 0, 1)   // speed = len/dt (px/ms), slow → 1
  tiltMag = clamp(hypot(tiltX, tiltY), 0, 1)   // from event2 ABS_TILT_X/Y (±9000)

Uniform (non-calligraphy) pens: f = 0.5 + 0.5·pressure. PenDevice emits strokeMove(pos, pressure, tiltX, tiltY, eraser).

5. Tooling (in tools/)

• setbufshim.cpp — LD_PRELOAD shim; proved setBuffers capture (feasibility).
• fbdump.cpp — LD_PRELOAD into xochitl; snapshots the framebuffer to /home/root/fbdump.{png,bmp} when /tmp/fbdump is touched (use BMP — no PNG plugin). Note: setBuffers interposition into xochitl via LD_PRELOAD was flaky/unconfirmed in one attempt; the in-process --export-dynamic capture in Chatter itself is reliable.

6. Open questions / ways to go deeper later

• Recover exact reMarkable widths-per-size and the nib/pressure curves by snapshotting stock strokes (fix the xochitl fbdump path) and measuring, or by parsing the v6 .rm per-point width/direction data.
• Add a proper speed estimate (smoothed) and tilt-direction (not just magnitude).
• Confirm LastPenSize values for the thin/medium categories (only size 3 = 3.0 is confirmed).