chatter/doc/Chatter_technical_reference.md

Chatter — Technical Reference (master document)

Audience: a programmer who needs to understand, build, modify, or maintain Chatter — including how the reMarkable standard software works and how Chatter fits alongside it.

This is the index + complete architecture. Several topics have their own deep-dive documents; this file summarizes each and links to it, then fills in everything not covered elsewhere. Read this first; follow the links for detail.

Document map

Document	What it covers
Chatter_application_proposal.md	The why: the assistive-speech use case, user-experience goals.
Chatter_implementation.md	The phased plan and its running changelog; device facts, toolchain, decisions.
Chatter_phase1_findings.md	On-device investigation results (input devices, display, doc store).
Chatter_stylus_research.md	Pen rendering, the Calligraphy model, width calibration, tooling.
Chatter_user_guide.md	The end-user-facing instructions.
developer_mode_screen.md	Enabling developer mode on the device.
tablet_access.md	SENSITIVE (root password + IPs) — gitignored, not in the repo.

---

1. The device

reMarkable Paper Pro Move — model RM03A, codename "Chiappa".

• SoC: NXP i.MX93 (Arm Cortex-A55, aarch64), 2 GB RAM, 64 GB storage.
• OS: "Codex" Linux 5.7.121 (Yocto scarthgap), image 3.27.1.0. SSH server is dropbear (not OpenSSH).
• Display: E Ink Gallery 3 — color (ACeP, "acep2"), 7.3", 954×1696 logical / 960×1696 framebuffer, 264 PPI (≈10.4 px/mm). Driven by DRM/KMS only (imx-drm); there is no /dev/fb*.
• Input: pen = /dev/input/event2 (Elan marker: ABS_X 0–6760, ABS_Y 0–11960, ABS_PRESSURE 0–4096, ABS_TILT_X/Y ±9000, BTN_TOOL_PEN/ BTN_TOOL_RUBBER for flip-to-erase). Finger = /dev/input/event3 (10-point capacitive, ABS_MT_*, grid 1248×2208). Power = event0, hall/folio = event1. Pen and finger are separate evdev devices — Chatter tells them apart natively.

Full investigation: Chatter_phase1_findings.md.

1.1 Filesystem / persistence model (critical)

• / (rootfs, /dev/mmcblk0p3) — ext4, mounted read-only, but persistent. Remount rw (mount -o remount,rw /) to write; survives reboots, lost on an OS update (A/B partition swap).
• /etc, /run, /var/volatile — VOLATILE overlays (upperdir on tmpfs). Anything written here is lost on reboot. (This is why a systemd unit dropped in /etc/systemd/system vanished after a reboot.)
• /home — encrypted, fully persistent, ~45 GB free. Everything Chatter installs lives here (/home/root/chatter/). SSH keys persist because they are under /home/root/.ssh.

Consequence: the launcher's systemd unit is installed onto the rootfs (/usr/lib/systemd/system) so it survives reboots; the binary and scripts live under /home. An OS update wipes rootfs changes — disable OS auto-updates on a delivered device, and keep scripts/install-launcher.sh to reinstall.

---

2. The standard reMarkable software stack

Understanding the stock stack is necessary because Chatter reuses its display plumbing and hands off to it.

• xochitl — the stock note app. Qt 6.8.2 / Qt Quick, proprietary. Owns the display while running. Started/stopped as the xochitl systemd service.
• The epaper QPA platform plugin (libqsgepaper.so, in /usr/lib/plugins/). A Qt platform + software scene-graph renderer for the e-paper. A Qt app runs on it with -platform epaper (+ QT_QUICK_BACKEND=epaper). Key facts learned by reverse engineering:
  • It is a software renderer: custom QSGGeometryNodes are silently dropped; only textures/rects/glyphs render. (This is why Chatter cannot draw ink via the scene graph — see §3.)
  • EPFramebuffer is the panel singleton (actually EPFramebufferAcep2 on this color device), guarded by an flock at /tmp/epframebuffer.lock — only one process may hold the panel. Success marker in the journal: SWTCON initialized \o/. A stranded holder → Failed to lock epframebuffer / Failed to initialize SWTCON and a blank panel. Always stop the previous holder first.
  • Relevant EPFramebuffer methods (mangled symbols bound in src/epfb.h):
    • instance() → the singleton.
    • setBuffers(std::tuple<QImage,QImage>, QImage*) — sets the front/back buffers; cross-DSO and interposable (see §3).
    • swapBuffers(QRect, EPContentType, EPScreenMode, QFlags<UpdateFlag>) — pushes a region to the panel. A single explicit call renders solid (no dashed two-pass). Screen modes (from EPScreenModeItem::Mode): Pen=0, Mono=1, Animation=2, UI=3, Content=4, Sleep=5. Chatter uses Pen (0) for fast ink and Content (4, "full update, STD") for full refreshes.
    • ghostControl(GhostControlMode) — the panel's anti-ghosting API. Modes 0/3 do an immediate full-screen de-ghost via the region swapBuffers; mode 1 schedules one. Chatter does NOT call it — its region-swap path uses internal EPContentMap/EPScreenModeMap members the stock app maintains and we do not, so calling it corrupts state and hangs after a few calls. Chatter de-ghosts manually instead (§6.3).
  • The panel uses ACeP color waveforms (acep2_lut, get_waveform_data, software TCON "SWTCON"). Color ghosting is real and only cleared by a full-update waveform driven to the dark extreme (§6.3).
• Pen styles live in ~/.config/remarkable/xochitl.conf, key LastWritingTool (a @Variant QVariantMap, readable via QSettings): LastPen (tool id, e.g. 21 = Calligraphy, 16 = a fineliner), LastPenSize (category 1/2/3 = thin/thicker/thickest), LastPenColorCode (0xAARRGGBB). xochitl writes this to disk only when you leave a document (return to the document list) — not on toolbar taps, and not reliably on an abrupt stop.

---

3. Chatter architecture (the core idea)

Problem: the make-or-break requirement is that ink look like the stock pen — solid, crisp, low-latency. The epaper software scene graph cannot do this: geometry nodes don't render, and QQuickPaintedItem textures always come out as the e-ink dashed two-pass refresh, regardless of screen mode or antialiasing.

Solution — a direct-framebuffer ink pipeline that bypasses the Qt scene:

1. FbCapture (src/FbCapture.{h,cpp}) interposes EPFramebuffer::setBuffers in-process. The executable is linked -Wl,--export-dynamic, so its definition of that symbol wins the cross-DSO call from the plugin; we capture the real framebuffer's pixel memory and wrap it as a QImage (FbCapture::framebuffer()) with no copy (constBits). (Intra-DSO calls like swapBuffers are not LD_PRELOAD-interposable due to direct binding — but in-process --export-dynamic on setBuffers works.)
2. InkEngine (src/InkEngine.{h,cpp}) draws strokes with QPainter straight into that framebuffer image, then calls EPFramebuffer::swapBuffers (via src/epfb.h) on just the dirty rectangle. One explicit swap → solid, single-pass ink.
3. Qt Quick renders only the static UI (the two buttons). The scene never recomposites over the ink because nothing in it animates.

This is what gives stock-quality strokes. The reverse-engineering trail and the dead-ends (geometry nodes, screen modes, antialiasing) are in Chatter_stylus_research.md and the Chatter_implementation.md changelog (v0.7).

3.1 Process / run model

• Chatter is a single Qt 6 executable run as a transient systemd service (systemd-run --unit=chatter …), with QT_QUICK_BACKEND=epaper, LD_LIBRARY_PATH=/usr/lib/plugins/scenegraph, -platform epaper.
• It requires xochitl to be stopped (single DRM master + the panel flock).
• Managed via systemctl {stop,status} chatter and journalctl -u chatter.

---

4. Source components

src/
  main.cpp          Entry point: reads pen style, wires PenDevice → InkEngine,
                    exposes `ink`/`appControl` to QML, loads the QML UI.
  FbCapture.{h,cpp} Interposes EPFramebuffer::setBuffers; exposes the live
                    framebuffer QImage + FbCapture::ready().
  InkEngine.{h,cpp} THE core. Virtual canvas, ink drawing, finger-erase, buttons
                    (paint/hit-test/actions), two-finger scroll, de-ghosting.
  PenDevice.{h,cpp} evdev reader for the pen (event2); maps to screen coords;
                    emits strokeStart / strokeMove(pos,pressure,tiltX,tiltY,eraser)
                    / strokeEnd.
  AppControl.{h,cpp} returnToStandard(): transient unit stops chatter, starts xochitl.
  epfb.h            asm-label bindings to the private EPFramebuffer symbols
                    (instance, swapBuffers, ghostControl).
  EPScreenModeItem.{h,cpp}  Legacy/unused: binding to the private screen-mode item
                    from the scene-graph era. Kept for reference.
  InkCanvas.*, Experiment.*  Legacy from the Qt-Quick-canvas spike; NOT built.

qml/Main.qml        White Window; the Back/Clear buttons (TopButton); a
                    MultiPointTouchArea for finger erase (1) and scroll (2).

tools/
  chatter_launcher.c  The return-to-Chatter daemon (4-finger watcher on event3).
  grabtest.c          Probe whether an input device is EVIOCGRAB-exclusive.
  fbdump.cpp          LD_PRELOAD framebuffer snapshot (BMP; device has no PNG plugin).
  swapshim.cpp, setbufshim.cpp  Feasibility shims used during RE.

scripts/
  build.sh            Source the SDK env, cmake build.
  deploy-and-run.sh   Stop xochitl+chatter, scp, relaunch as a transient unit.
  to-chatter.sh       Switch standard → Chatter (run by the launcher).
  install-launcher.sh Persistently install the launcher unit on the rootfs.
  chatter-launcher.service  The systemd unit (installed to rootfs).
  restore-xochitl.sh  Return to the standard GUI.

---

5. Input & gesture model

Two independent input streams, never confused:

• Pen (event2) — read by PenDevice and delivered to InkEngine. The epaper QPA does not deliver the pen to Qt, so the pen never triggers QML.
• Finger (event3) — delivered by the epaper QPA to Qt as touch, handled in QML (MultiPointTouchArea, and the buttons' MouseAreas).

Gesture map:

Input	Action	Where handled
Stylus draw	Ink (flip = erase via BTN_TOOL_RUBBER)	PenDevice → InkEngine
Stylus tap on a button	Button action	InkEngine hit-tests (m_buttons)
1 finger drag	Erase wipe (~12 mm; a tap erases nothing)	QML MultiPointTouchArea → InkEngine::erase*
2 fingers drag	Vertical scroll	QML → InkEngine::panBy/panEnd
Finger tap on a button	Button action (gray feedback)	QML MouseArea → InkEngine::flashButton/activateButton
4 fingers hold (~700 ms)	Return to Chatter (only while xochitl is front)	tools/chatter_launcher.c

The MultiPointTouchArea latches the gesture type until all fingers lift, so a two-finger scroll never degrades into an erase when one finger is raised. Erase requires movement past a ~1.5 mm threshold (tap-safe).

Buttons (Back, Clear) are a single source of truth in InkEngine: QML registers their geometry (registerButton) so the engine can exclude ink, redraw them after a blit, and hit-test stylus taps. Press feedback (flashButton) and actions (activateButton) are drawn directly to the framebuffer (instant, no scene flashing); both finger and stylus route through them.

---

6. The virtual canvas, scrolling, and de-ghosting

6.1 Growable raster canvas

InkEngine holds a QImage canvas larger than the screen (starts 2× tall, grows downward as you write near the bottom). The screen is a viewport into it at vertical offset m_panY. Drawing maps screen→canvas (+m_panY); a dirty canvas rect is blitted back to the framebuffer (blitRegion) and the buttons are repainted on top. (Storage model chosen: raster, to preserve the exact ink quality; trade-off is no crisp zoom-in. Horizontal/zoom are future work.)

6.2 Scrolling

Two-finger drag → panBy(dy) (natural: content follows fingers), clamped to the canvas, fast-blitted per step. panEnd() debounces a de-ghost (§6.3).

6.3 De-ghosting (color ACeP ghosting)

Fast Pen-waveform swaps leave color residue ("faint red duplicate") that accumulates while scrolling and is not cleared by a white redraw — it is panel retention, cleared only by a full-update waveform driven to black (white/gray do not clear it; this was tested). Chatter's fullRefresh(): fills the screen black + full-update swap (screenMode=Content), waits ~220 ms, then full-updates the real content. ghostControl() would be the "proper" API but corrupts state (§2), so this manual flash is used.

To keep the flash from being intrusive:

• It runs only after scrolling, debounced ~1 s after the last scroll (not on every finger-lift).
• Clear has two methods: if there was no scrolling since the last clear (the common fill-one-screen case) it uses the gentle fast clear; if there was scrolling, it does the black de-ghost clear. (m_scrolled flag.)
• Tunables via env: CHATTER_FULL_SM (full-update screen mode, default 4), CHATTER_FLASH_GRAY (flash level 0=black..255; black is what actually clears).

A black flash is intrinsic to clearing color ghosting (the stock UI flashes on its full refreshes too); we minimized when it happens rather than eliminating it.

---

7. Pen-style matching, width calibration, calligraphy

• Chatter reads xochitl.conf LastWritingTool at startup (readPenStyle in main.cpp) and applies tool type, size, and color — no style menu in Chatter. Because each switch-to-Chatter restarts the process, it re-reads the current pen. The flush sequence matters (§2): set the pen, use it, leave the document, then switch to Chatter.
• Width is calibrated to measured widths on the 264-PPI panel: size 3 ≈ 2 mm, size 2 ≈ 1 mm, thinnest ≈ 2 px; pressure ≈ doubles width.
• Calligraphy (tool 21) is approximated with a dynamic direction/pressure/speed/tilt width model. Full detail, measurements, and the formula: Chatter_stylus_research.md.

---

8. Toggle & the return launcher

• Chatter → standard: Back → InkEngine::backRequested → AppControl::returnToStandard() → a transient unit stops chatter and starts xochitl (sequenced so the panel lock is released first).
• standard → Chatter: tools/chatter_launcher.c runs always as a rootfs systemd service. It reads event3 without grabbing it (verified possible via tools/grabtest.c — xochitl does not hold an exclusive grab), counts multitouch slots, and on a 4-finger hold ≥700 ms while xochitl is the front app runs to-chatter.sh. It reopens the device on any read interruption (sleep/ wake) so it never dies.
• Install persistently: scripts/install-launcher.sh (remounts the rootfs rw, places the unit + its multi-user.target.wants symlink under /usr/lib/systemd/system). See the persistence model in §1.1.

---

9. Build & deploy

Get the source: clone from the Gitea server (needs an account with read access to ack/chatter and your SSH key registered, or use the HTTPS URL):

git clone git@git.andykopra.com:ack/chatter.git
cd chatter

Toolchain: official reMarkable Chiappa SDK 3.27.0.97 (Qt 6.8.2 sysroot, x86_64 host). Install, then source environment-setup-cortexa55-remarkable-linux.

scripts/build.sh            # source SDK env + cmake build  -> build/chatter
scripts/deploy-and-run.sh   # stop xochitl+chatter, scp, relaunch as transient unit
scripts/restore-xochitl.sh  # back to the standard GUI
scripts/install-launcher.sh # one-time (and after each OS update): persist the launcher

CMakeLists.txt: Qt6 Quick app (qt_add_executable + qt_add_qml_module), links libqsgepaper.so, and crucially target_link_options(... -Wl,--export-dynamic) so the setBuffers interposition wins.

Device housekeeping: only one process may hold the panel — always stop the previous holder. Deploy under /home/root/chatter, never the rootfs (except the launcher unit). The device sleeps and DHCP may reassign its IP on wake; if you script a reconnect, rescan for the tablet's MAC address (shown on the device's GPLv3-compliance screen, alongside its IPs).

---

10. Status & open items

Working on hardware: solid stock-quality ink matching the selected pen; finger-wipe erase; whole-page Clear (two methods); stylus eraser; bidirectional toggle (Back + 4-finger launcher, persistent); vertical scroll on a growable canvas with debounced de-ghosting; instant button feedback (finger + stylus).

Open / future:

• Save (Phase 5) — write a timestamped transcript into a "Chatter" folder in xochitl's on-disk document format; importance to the user still unconfirmed. See Chatter_implementation.md §6 / §6a.
• Horizontal scroll & zoom-out overview — deferred extensions of the canvas.
• Calligraphy fidelity — "close enough" approximation; exact nib unknown.
• Erase / gesture thresholds — to be tuned with the end user (Matt's field test).
• OS auto-update would wipe the rootfs launcher unit (and could break paths) — disable it on a delivered device.

---

This document is the technical entry point. When the design changes, update this file and the relevant component doc; keep Chatter_implementation.md's changelog as the chronological record.