Initial commit: Chatter — assistive-writing app for reMarkable Paper Pro Move

Direct-framebuffer ink pipeline (stock-quality strokes), finger-wipe erase, growable scrolling canvas with color-ghost cleanup, bidirectional toggle with a persistent 4-finger return launcher, instant button feedback. Includes prebuilt aarch64 binaries (dist/), build/deploy/install scripts, a user guide, and a complete technical reference. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-27 16:34:28 +02:00
commit 5f21d9099c
43 changed files with 2878 additions and 0 deletions
--- a/doc/Chatter_implementation.md
+++ b/doc/Chatter_implementation.md
@@ -0,0 +1,381 @@
+# Chatter — Implementation Plan & Method
+
+**Status:** Living document — revised as we learn. Read the Changelog at the bottom for what changed.
+**Version:** 0.7 (2026-06-26)
+**Audience:** The Chatter development team (Andreas, Matt, and Claude as development assistant).
+
+> This document is the shared source of truth for *how* we build Chatter. It is
+> deliberately incremental: early phases will answer questions that later phases
+> depend on, and this document will be updated as those answers come in. If
+> something here is marked **(unknown / to confirm on-device)**, treat it as an
+> open question, not a decision.
+
+---
+
+## 1. What Chatter is
+
+Chatter is a minimal, purpose-built interface for a reMarkable tablet used as an
+**assistive-speech device** by a user who has difficulty speaking — she writes,
+and her conversation partner reads. Chatter **augments** the tablet; it does not
+replace the standard reMarkable GUI.
+
+The design priority is **intuitiveness under intermittent recall**: the user
+should not have to remember technical steps. See the full motivation and the
+user-experience reasoning in [`Chatter_application_proposal.md`](Chatter_application_proposal.md).
+
+Chatter's core actions, by expected frequency:
+
+1. **Erase part of the text** — a simple, intuitive gesture (e.g. wiping a
+   fingertip back and forth over an area).
+2. **Erase the whole page** — a single gesture (e.g. a diagonal finger stroke,
+   upper-right → lower-left).
+3. **Toggle to/from the standard GUI** — *necessary*; Chatter must be able to
+   hand off to the stock interface and be returned to.
+4. **Save the current page** — *importance to the user is not yet confirmed*; a
+   single button writes a timestamped transcript into a "Chatter" folder.
+
+---
+
+## 2. Target device and key constraints
+
+**Device: reMarkable Paper Pro Move — model RM03A.** (Identified from the
+official user-guide PDF linked in the proposal.)
+
+| Property | Value | Why it matters |
+|---|---|---|
+| Display | E Ink Gallery 3, **color**, 7.3", 954×1696 (264 PPI), backlit | Color refresh pipeline differs from older grayscale models |
+| Touch | Multi-point capacitive | Finger gestures (erase) are viable |
+| Pen | Active Marker / Marker Plus (flip-to-erase on Plus) | Pen and touch are **separate input streams** → we can tell finger from stylus natively |
+| SoC / RAM | ARM 1.7 GHz dual A55, 2 GB RAM, 64 GB | Standard ARM-Linux cross-compile target |
+| OS | "Codex" (custom Linux) | Stock note app is Xochitl (Qt/C++, proprietary) |
+| Dev access | Enable **developer mode**, which requires a **factory reset** | **Back up the device first** |
+
+**Key risk — this is the newest model.** The mature community frameworks
+(rmkit, libreMarkable, Plato, KOReader ports) were built against the older
+RM1/RM2 grayscale framebuffer and refresh ioctls. They likely **do not support
+the Paper Pro Move yet**, or only partially. We should expect to do
+**first-principles work** — discovering this model's framebuffer format, color
+refresh path, and input device nodes by probing the hardware over SSH — rather
+than relying on an existing framework. This raises the value of hands-on device
+access early and argues for a small, well-proven first milestone.
+
+---
+
+## 3. Roles
+
+- **Andreas** — development, with Claude's assistance, on his own Paper Pro Move
+  over SSH.
+- **Matt** — installation and QA, including on-site testing with the end user on
+  her tablet. Matt has the same model (Paper Pro Move). Matt may also take on
+  development; the split will be worked out as the project progresses.
+- **Claude** — development assistant: writes/reviews code, drives the
+  build-deploy loop over SSH where possible, and maintains this document.
+
+This document is how Matt stays in sync. When the design changes, the change
+lands here first.
+
+---
+
+## 4. Development phases
+
+The phases are ordered so each de-risks the next. **Toggle (necessary) precedes
+Save (provisional).**
+
+### Phase 0 — Safety & access
+- **Back up first** (critical on the user's device; skippable on a new tablet
+  with nothing saved). Enabling developer mode **performs a factory reset** — on
+  the Paper Pro family, SSH is unavailable until developer mode is on, and
+  turning it on wipes the device. Sync to the reMarkable cloud account before
+  enabling. Document this backup procedure so Matt can repeat it.
+- **Enable developer mode:** Hamburger menu (upper left) → Settings → Software →
+  Advanced → Developer mode → Accept. This factory-resets the device (deletes
+  all local files; user data cleared on reboot). Re-pair with the reMarkable
+  account afterward to access cloud content. (Disabling later requires
+  reMarkable's recovery application.)
+- **Reconnect wifi** after the reset (the wifi config is wiped).
+- **Get SSH credentials:** Menu → Settings → Help → Copyright and licenses;
+  the root username (`root`), password, and device IP addresses are under the
+  "GPLv3 Compliance" header. USB exposes the tablet at `10.11.99.1`.
+- **Enable SSH over wifi (it is OFF by default):** connect over USB, SSH in,
+  and run `rm-ssh-over-wlan on`. Only then is the listed `192.168.x.x` address
+  reachable; after that the USB cable is optional.
+- Install your workstation's SSH public key into the tablet's
+  `~/.ssh/authorized_keys` (user `root`) for passwordless access.
+- Confirm SSH login, `scp` of a binary, and remote execution.
+- Stand up the x86_64-hosted cross-toolchain for Codex OS; verify a trivial
+  ARM binary runs on-device.
+
+> Dev environment note: development host is **Jatke** (on the wired LAN); we
+> connect to the tablet **over wifi** (USB not in use).
+
+### Phase 1 — On-device investigation (no app code; produces a findings writeup)
+Answer the questions that can only be answered with the tablet over SSH. See the
+checklist in §5. Output: a short findings document added to `doc/`.
+**Status: substantially complete (2026-06-25)** — see
+[`Chatter_phase1_findings.md`](Chatter_phase1_findings.md). Key results: pen
+(`event2`) and finger (`event3`) are separate evdev devices; the display is
+DRM/KMS only (no fbdev) driven by `imx-drm`; and the stock app is a Qt 6.8.2 app
+rendering through a reusable **`epaper` QPA plugin** — so Chatter should be a
+Qt 6 app run with `-platform epaper`. Remaining sub-items (verifying `-platform
+epaper` from a third-party binary, input delivery, toggle hand-off) roll into
+the Phase 2 spike.
+
+### Phase 2 — Minimal canvas spike
+Read the pen, render strokes to the (color) display, and clear the page —
+nothing more. This proves the two hardest pieces (input + color refresh) end to
+end and de-risks everything after it.
+
+**Status (2026-06-25): display/toolchain half PROVEN on hardware.** A
+third-party Qt Quick app (`build/chatter`) cross-built with the Chiappa SDK
+renders crisp black-on-white on the e-paper via `-platform epaper`
+(`QT_QUICK_BACKEND=epaper`).
+
+**Status (2026-06-26): inking WORKS, with one known quality issue.** Custom
+evdev pen reader (`src/PenDevice.cpp`, reads `event2`) drives a
+`QQuickPaintedItem` canvas (`src/InkCanvas.cpp`): ink tracks the pen, orientation
+correct, eraser (BTN_TOOL_RUBBER) erases, finger-tap "Clear" works, latency good.
+- **Known issue:** strokes render as a "dashed line that fills in" (two-pass),
+  unlike the solid stock pen. **Root cause:** we use the panel's default
+  *grayscale* mode. **Fix path:** xochitl wraps its writing area in a QML
+  `ScreenModeItem` set to **Mono/FAST** with a dedicated **pen waveform**
+  (`/usr/share/remarkable/ct33_pen.bin`). The control is the private
+  `EPScreenModeItem` class in `libqsgepaper.so` (`setMode()`, modes
+  Mono/FAST/FastGrayscale) — not a public QML type, no SDK header — so Chatter
+  must integrate it directly (instantiate, link the plugin, drive `mode` via the
+  meta-object, wrap the canvas). Antialiasing on/off does not affect it.
+- **Pen-style decision:** Chatter uses the pen style set in the standard GUI
+  (xochitl.conf `LastPen`/`LastPenSize`/`LastPenColor`, readable via QSettings),
+  re-read so it tracks changes the user makes after toggling — no style menu in
+  Chatter.
+
+**Lessons learned (encoded in `scripts/`):**
+- The e-paper framebuffer is a **singleton guarded by an flock**
+  (`/tmp/epframebuffer.lock`, from `libqsgepaper`'s `EPFramebufferAcep2`). Only
+  ONE process may hold it; a stranded instance shows `Failed to lock
+  epframebuffer` / `Failed to initialize SWTCON` and the panel stays blank.
+  Always stop the previous instance first. Journal success marker:
+  `SWTCON initialized \o/`.
+- **Run as a transient systemd service** (`systemd-run --unit=chatter …`), not an
+  ssh background job (which holds the SSH channel open and can strand the
+  process). Manage with `systemctl stop chatter` / `journalctl -u chatter`.
+- A full-screen `Window` needs an **explicit size** (`width: Screen.width;
+  height: Screen.height`); `visibility: FullScreen` alone left it unsized/blank.
+- xochitl runs with **no special Qt env** — `-platform epaper` +
+  `QT_QUICK_BACKEND=epaper` suffices.
+
+**Run recipe:** `scripts/build.sh && scripts/deploy-and-run.sh`;
+`scripts/restore-xochitl.sh` to return to the standard GUI.
+
+### Phase 3 — Erase gestures
+- Region erase: finger wipe removes ink near the wiped area ("near" tuned
+  experimentally with the user).
+- Whole-page erase: single diagonal finger stroke (upper-right → lower-left).
+- Provide stylus fallbacks if finger/stylus separation proves unreliable.
+
+### Phase 4 — Toggle to/from the standard GUI  *(necessary)*
+
+**Status (2026-06-26): BIDIRECTIONAL toggle works on-device (verified).**
+- **Chatter → standard:** top-left **"Back"** button → `AppControl::returnToStandard()`
+  → transient unit stops chatter, starts xochitl.
+- **standard → Chatter:** a persistent **launcher daemon** (`tools/chatter_launcher.c`,
+  installed as a rootfs systemd service via `scripts/install-launcher.sh`) watches
+  the touch device and, on a **4-finger hold (~700 ms)** while xochitl is in front,
+  runs `to-chatter.sh` to switch to Chatter. Gesture is tunable; finalize the
+  memorable one with the user.
+- **Deployment note:** `/etc` and `/run` are volatile overlays (wiped on reboot);
+  persistent installs go to `/home` (app/binaries/scripts) or the rootfs
+  (`mount -o remount,rw /`, lost on OS update). Disable OS auto-updates on a
+  delivered device.
+
+- A gesture/button in Chatter hands off to the standard reMarkable GUI.
+- A **memorable** mechanism returns from the standard GUI to Chatter. The
+  proposal asks that the user help choose this; we will prototype options and
+  let her pick.
+- **Phase 1 insight:** because the display is DRM/KMS (single master) and
+  xochitl owns it while running, the toggle is a **hand-off**, not an overlay —
+  stop/pause `xochitl` to give Chatter the display, and reverse to return.
+  Mechanism (`systemctl stop/start` vs. `SIGSTOP/SIGCONT`) to be validated.
+
+### Phase 5 — Save the page  *(provisional — pending confirmation it matters to the user)*
+- A single visible button saves the current page.
+- Filename fully derived from date/time (year, month, day, hour, minute,
+  second).
+- Written into a folder named **"Chatter"**, using the same on-disk document
+  format the stock app uses, so transcripts appear as normal notebooks and can
+  be renamed/deleted/synced via the standard GUI (see §6).
+
+### Phase 6 — Field test & refine
+Matt installs and observes the user. Capture especially the "Why does this…" and
+"Why doesn't this…" questions; feed them back into this document and the phase
+plan.
+
+---
+
+## 5. On-device investigation checklist (Phase 1)
+
+These are the open questions the hands-on phase exists to answer.
+
+- **Input devices:** which `/dev/input/event*` is the pen vs. the capacitive
+  touch; event format, coordinate range and orientation, pressure; the
+  flip-to-erase signal on Marker Plus.
+- **Display:** how the color Gallery 3 panel is driven — framebuffer device and
+  pixel format, and the refresh mechanism (ioctls or this model's equivalent).
+  *(Highest-risk unknown.)*
+- **Document store:** where Xochitl stores notebooks; the
+  `.metadata` / `.content` / per-page `.rm` layout for this model; the `.rm`
+  format version (newer, color-capable); how a *collection* (folder) is
+  represented; and whether a hand-built document appears in the library after a
+  restart/sync.
+- **Coexistence & toggle:** how to launch our binary alongside / over Xochitl,
+  and what a reliable toggle and return-to-Chatter mechanism looks like.
+
+---
+
+## 6. The "save" approach (clarification)
+
+reMarkable exposes **no public callable on-device API** for creating notebooks.
+The stock app simply **writes files to an on-disk document store** in a specific
+format (roughly: a UUID per document, a `.metadata` JSON, a `.content` JSON, and
+one binary `.rm` line-file per page, plus the color/folder data and a sync
+layer). A "folder" such as the **Chatter** folder is itself a document of type
+*collection*.
+
+So "save using the same method the environment uses" means: **write our canvas
+into the exact on-disk format Xochitl reads, for this model's format version.**
+The open question for Phase 5 is confirming/replicating the Paper Pro Move's
+current `.rm` format. This is investigated in Phase 1.
+
+---
+
+## 6a. Confirmed device facts (from SSH, 2026-06-25)
+
+- **Internal codename:** "Chiappa" (`/proc/device-tree/model` = "reMarkable
+  Chiappa"; hostname `imx93-chiappa`). This is the Paper Pro Move (RM03A).
+- **SoC:** NXP **i.MX93** (Arm Cortex-A55). Relevant to the toolchain target and
+  to how the display is driven.
+- **OS:** Codex Linux **5.7.121** (Yocto **scarthgap**), image version
+  `3.27.1.0`.
+- **SSH server:** **dropbear** (not OpenSSH). Root login; `~/.ssh/authorized_keys`
+  honored. Wifi SSH enabled via `rm-ssh-over-wlan on`.
+- **Access from Jatke:** passwordless via ed25519 key as `root` over wifi (the
+  tablet's IP) or `root@10.11.99.1` (USB, fixed on every reMarkable). The tablet's
+  actual IP/MAC and credentials are in `tablet_access.md` (SENSITIVE, gitignored).
+
+**Phase 0 status:** access established (dev mode, SSH over USB + wifi, key-based
+login, scp verified). Remaining for Phase 0: stand up the cross-toolchain and
+run a trivial ARM binary on-device.
+
+## 6b. Toolchain / build & deploy (researched 2026-06-25)
+
+**Official SDK (cross-toolchain + Qt 6.8.2 sysroot), self-extracting `.sh`,
+x86_64 host only.** For this device (codename **chiappa**):
+
+```
+https://storage.googleapis.com/remarkable-codex-toolchain/3.27.0.97/chiappa/remarkable-production-image-5.7.119-chiappa-public-x86_64-toolchain.sh
+```
+
+- ~486 MB. Device image is `3.27.1.0` (Codex 5.7.121); newest published Chiappa
+  SDK is `3.27.0.97` (5.7.119) — same `3.27.x` family, so binaries built against
+  it run on the device. (An aarch64-host variant also exists; we use x86_64 for
+  Jatke.) Download links live on developer.remarkable.com/links; the GCS bucket
+  itself is not directly listable (403).
+- Build host **Jatke** (x86_64, ample disk). SDKs expect a Yocto-supported Linux
+  host.
+
+**Install & activate:**
+```
+chmod u+x remarkable-production-image-5.7.119-chiappa-public-x86_64-toolchain.sh
+./remarkable-production-image-5.7.119-chiappa-public-x86_64-toolchain.sh -d <install-dir>
+source <install-dir>/environment-setup-cortexa55-remarkable-linux   # exact filename confirmed post-install (i.MX93 = Cortex-A55)
+```
+
+**Build a Qt Quick app** (per developer.remarkable.com/documentation/qt_epaper):
+CMake with `find_package(Qt6 REQUIRED COMPONENTS Quick)` +
+`qt_add_executable` / `qt_add_qml_module`; `cmake . -B build && cmake --build
+build` after sourcing the SDK env.
+
+**Deploy & run on device:**
+```
+scp build/<app> chatter:/home/root/chatter/      # deploy under /home, not rootfs
+ssh chatter 'systemctl stop xochitl'             # release the display (pre-authorized)
+ssh chatter 'cd /home/root/chatter && QT_QUICK_BACKEND=epaper ./<app> -platform epaper'
+```
+
+Constraints (official + Phase 1): **Qt Quick/QML only, no Qt Widgets**; **xochitl
+must be stopped** (DRM single-master); **touch is automatic, pen is custom**
+(read `event2`). Some SDK/OS combos require copying `libqsgepaper.so` to the
+device — verify during the spike.
+
+## 7. Decisions & open questions
+
+**Decided (from Phase 1):**
+- **Implementation stack: C++ / Qt 6.8.2**, rendered via the device's **`epaper`
+  QPA plugin** (`-platform epaper`). Chosen because the stock app proves this
+  path works and gives us e-ink display + refresh for free; the older
+  fbdev-mmap approach is not available (DRM/KMS only). Cross-build with a Yocto
+  scarthgap aarch64 SDK matching Qt 6.8.2. *(Matt — flag any objection.)*
+- **Deploy location: under `/home`** (45 GB free), never the rootfs (~88 MB
+  free, overlay, reset by updates).
+- **Toggle is a display hand-off** with xochitl (see Phase 4), not an overlay.
+
+**Still open:**
+- **Save mechanism** — leaning **PDF-backed document** over authoring native v6
+  `.rm`; decide in Phase 5 (and gated by whether save matters to the user).
+- **Whether "save" matters to the user** — learned from observation.
+- **Toggle gesture + return-to-Chatter mechanism** — prototype and choose with
+  the user (Phase 4).
+- **Erase "closeness" thresholds** — tuned experimentally with the user
+  (Phase 3).
+
+---
+
+## Changelog
+
+- **0.6 (2026-06-26)** — Inking engine working on hardware: custom evdev pen
+  reader (`event2`) + `QQuickPaintedItem` canvas — pen draws, eraser erases,
+  finger-tap **Clear**, good latency. Added top control bar (**Back** left,
+  **Clear** right); Back hands off to the standard GUI (verified). Bound the
+  private `EPScreenModeItem` (key-function trick) to control panel refresh mode —
+  but the **dashed two-pass stroke refresh is the render loop's behavior,
+  independent of mode**, so it's deferred to a framebuffer-direct effort
+  (needs disassembly of `swapBuffers` for `EPContentType`/`UpdateFlag`).
+  Implemented **pen-style matching**: Chatter reads the standard GUI's selected
+  pen (type/size/color) from `xochitl.conf` `LastWritingTool` and applies width +
+  color — verified (Calligraphy, size 3, black).
+- **0.7 (2026-06-26)** — ⭐ **Solved stroke quality with a direct-framebuffer ink
+  pipeline.** After proving Qt Quick can't render stock-quality e-ink strokes
+  (geometry nodes unsupported by the software backend; painted-textures always
+  get the dashed two-pass; screen-mode and antialiasing irrelevant), switched
+  architecture: `FbCapture` interposes `EPFramebuffer::setBuffers` in-process
+  (`-Wl,--export-dynamic`) to capture the real framebuffer image; `InkEngine`
+  draws strokes straight into it and refreshes each segment with an explicit
+  `swapBuffers` — **solid, crisp, low-latency, correctly-aligned strokes on
+  hardware (verified), plus working Clear.** Qt Quick now only renders the static
+  Back/Clear UI. Key RE findings: `swapBuffers` renders solid (not dashed) on a
+  single explicit call; intra-DSO calls aren't LD_PRELOAD-interposable but the
+  cross-DSO `setBuffers` is. Remaining: calligraphic nib (uniform round width vs
+  the angled direction-dependent Calligraphy nib), return-to-Chatter trigger,
+  Save, field testing.
+- **0.5 (2026-06-25)** — Phase 2 spike: cross-built a Qt Quick hello-app with the
+  Chiappa SDK and rendered it on the e-paper via `-platform epaper` (proven on
+  hardware). Recorded lessons (panel flock singleton, run-as-systemd-service,
+  explicit Window size) and added `scripts/build.sh`, `deploy-and-run.sh`,
+  `restore-xochitl.sh`. Seeded the source tree (`CMakeLists.txt`, `src/main.cpp`,
+  `qml/Main.qml`).
+- **0.4 (2026-06-25)** — Researched the toolchain (§6b): identified the official
+  Chiappa SDK (`3.27.0.97` / Codex 5.7.119, x86_64 host, ~486 MB), install/
+  activate steps, the Qt Quick build + `-platform epaper` run commands, and the
+  deploy-and-stop-xochitl flow. Confirmed Qt-Quick-only + custom-pen constraints.
+- **0.3 (2026-06-25)** — Phase 1 on-device investigation substantially complete
+  (see `Chatter_phase1_findings.md`). Resolved the stack decision to C++/Qt 6 +
+  `epaper` QPA plugin; recorded display = DRM/KMS only, pen/finger as separate
+  evdev devices, toggle = display hand-off, deploy under `/home`, and the
+  document-store format for the Save phase.
+- **0.2 (2026-06-25)** — Phase 0 access established. Added confirmed device
+  facts (§6a): codename "Chiappa", NXP i.MX93 SoC, Codex Linux 5.7.121
+  (scarthgap), dropbear SSH, passwordless key access from Jatke. Corrected the
+  dev-mode menu path and documented that wifi SSH is off by default
+  (`rm-ssh-over-wlan on`).
+- **0.1 (2026-06-25)** — Initial version: scope, target device (RM03A) and its
+  constraints, roles, phase plan (with Toggle before Save), Phase 1 investigation
+  checklist, save-format clarification, and open decisions.