test(e2e): Playwright UI smoke + Go integration tests against MAS

- Playwright e2e against Vite dev server (LoginScreen renders, homeserver
  visible, version pinned). Fallback documented to swap to `wails dev
  -browser` via WAILS_DEV_URL when full backend bindings are required.
- Go integration tests (//go:build integration,goolm) for Start +
  ListRooms + LoadTimeline + SendText + GetDiagnostics against real
  Synapse. Skip cleanly if MATRIX_INTEGRATION_TOKEN/USER env vars unset
  so CI never breaks unattended.
- pnpm e2e/e2e:ui/e2e:report scripts. Per-run keyring namespace +
  t.TempDir XDG_CONFIG_HOME so integration tests don't clobber the
  production keyring entry.

E2E.md explains how to run each layer + limits (OIDC not automated).
This commit is contained in:
Egutierrez
2026-05-25 17:20:05 +02:00
parent 1a2e9b2cc0
commit 1d3744f2d7
8 changed files with 484 additions and 2 deletions
+6
View File
@@ -23,3 +23,9 @@ local_files/
# OS
.DS_Store
Thumbs.db
# Playwright (E2E)
frontend/playwright-report/
frontend/test-results/
frontend/blob-report/
frontend/.playwright/
+142
View File
@@ -0,0 +1,142 @@
# E2E tests — matrix_client_pc
Two layers cover the app end-to-end:
| Layer | Tool | What it proves | Where |
|---|---|---|---|
| **A. Backend integration** | `go test -tags 'integration goolm'` | `MatrixService` Start / ListRooms / LoadTimeline / SendText / GetDiagnostics work against a real Synapse + MAS | Repo root (`integration_test.go`) |
| **B. UI smoke** | Playwright + Vite dev server | The React frontend renders LoginScreen with the expected copy + button + version | `frontend/` |
Component-level tests (Vitest + RTL with mocked backend) live under `frontend/src/__tests__/` — they remain the primary feedback loop. The two layers below add the "did the wiring actually work" gate.
---
## Layer A — Go integration tests (real Synapse)
`integration_test.go` is gated by **two build tags**:
- `integration` — keeps it out of `go test ./...`.
- `goolm` — matches the runtime build tag the app uses for the cryptohelper backend.
Every test calls `readIntegrationEnv()` which **skips cleanly** when `MATRIX_INTEGRATION_TOKEN` / `MATRIX_INTEGRATION_USER` are unset. CI never breaks for lack of creds — it just shows `--- SKIP:`.
### Required env vars
| Var | Required | Example |
|---|---|---|
| `MATRIX_INTEGRATION_TOKEN` | yes | `syt_ZXhhbXBsZQ...` (MAS-issued access token) |
| `MATRIX_INTEGRATION_USER` | yes | `@egutierrez:matrix-af2f3d.organic-machine.com` |
| `MATRIX_INTEGRATION_DEVICE` | optional | `ABCDEFGH` (auto-discovered via `/whoami` if missing) |
| `MATRIX_INTEGRATION_TEST_ROOM` | optional | `!abcd1234:matrix-af2f3d.organic-machine.com` — enables the send/recv round-trip |
### Run
```bash
MATRIX_INTEGRATION_TOKEN=syt_... \
MATRIX_INTEGRATION_USER='@egutierrez:matrix-af2f3d.organic-machine.com' \
MATRIX_INTEGRATION_TEST_ROOM='!somefoo:matrix-af2f3d.organic-machine.com' \
go test -tags 'integration goolm' -timeout 120s -v \
./projects/element_agents/apps/matrix_client_pc/...
```
Verify the build without credentials:
```bash
go vet -tags 'integration goolm' \
./projects/element_agents/apps/matrix_client_pc/...
go test -tags 'integration goolm' -timeout 60s -v \
./projects/element_agents/apps/matrix_client_pc/...
# All 4 tests SKIP cleanly.
```
### What each test covers
| Test | Asserts |
|---|---|
| `TestIntegration_HomeserverReachable` | `whoami` against `https://matrix-af2f3d.organic-machine.com` returns the expected user_id (sanity gate; fastest fail) |
| `TestIntegration_StartAndListRooms` | `Start(user)` + `ListRooms()` returns `>=1` joined room |
| `TestIntegration_GetDiagnostics` | After `Start`: `Started`, `ClientReady`, `SyncActive` all true; `RoomsCount >= 1`; `LastError` empty |
| `TestIntegration_RoundtripMessage` | `SendText` + `LoadTimeline` round-trip: the new event surfaces within 10s of polling. Skipped if `MATRIX_INTEGRATION_TEST_ROOM` missing |
### Isolation
The tests use a **per-run keyring namespace** (`fn_registry.matrix_client_pc.test.<unix_nano>`) so they never touch the production keyring entry. `XDG_CONFIG_HOME` is overridden to a `t.TempDir()`, so the crypto store lands in a throwaway directory that disappears with the test process.
If the host has no D-Bus session (rare for headless CI), `buildService` calls `t.Skip()` instead of failing — the keyring is a hard dependency the test acknowledges it can't synthesise.
### Limits
- **OIDC login is NOT automated.** The MAS browser-redirect flow needs a real human at a real browser. Integration tests assume a token already exists; if it doesn't, get one with `wails dev` + click "Sign in" once, then extract from `~/.config/matrix_client_pc/...` keyring entry.
- **Device binding.** Some MAS sessions issue tokens without a device_id; the app's `Start()` retries `/whoami` and surfaces a clear error. Integration tests rely on the env var or on `/whoami` succeeding — if both miss, the test fails fast with a readable message.
---
## Layer B — Playwright UI smoke
Playwright drives a **plain Vite dev server** (not full `wails dev`) on a pinned port. The React app's only Wails runtime call on mount (`GetLastUserID`) throws synchronously because `window.go` is undefined, the `finally` clears the loading overlay, and LoginScreen renders identically to a fresh production launch. This catches HTML / CSS / bundle / Mantine theming regressions without needing the Go backend compiled with `goolm` + cgo on the test host.
### Why not `wails dev -browser`?
Two reasons:
1. `wails dev` (with or without `-browser`) compiles the full Go binary — `goolm` requires cgo, libolm transitive deps, and our SQLite driver. In a WSL / headless CI environment that's slow (60-90s warmup) and noisy.
2. The Go backend listens to OS keyring + opens browsers for OIDC. Both of those break in containerised / headless contexts.
If/when you do want the real bound runtime (e.g., to test that bindings actually expose the right names), set `WAILS_DEV_URL` to a running `wails dev -browser` URL and rerun:
```bash
# Terminal 1
cd projects/element_agents/apps/matrix_client_pc
wails dev -browser -loglevel Warning
# Note the URL it prints, usually http://localhost:34115
# Terminal 2
cd projects/element_agents/apps/matrix_client_pc/frontend
WAILS_DEV_URL=http://localhost:34115 pnpm e2e
```
The `playwright.config.ts` honours `WAILS_DEV_URL` and skips spawning its own webServer if you've started one externally (via `reuseExistingServer`).
### Run
```bash
cd projects/element_agents/apps/matrix_client_pc/frontend
pnpm e2e # headless run
pnpm e2e:ui # interactive UI for debugging
pnpm e2e:report # open last HTML report
```
First run on a fresh machine:
```bash
pnpm install
pnpm exec playwright install chromium
```
### Current specs
- `e2e/login_screen.spec.ts`
- `LoginScreen renders with Sign-in button` — finds the `matrix_client_pc` heading + the `Sign in with Matrix` button.
- `LoginScreen mentions homeserver + MAS context` — verifies the `matrix-af2f3d` homeserver + "Matrix Authentication Service" copy.
- `LoginScreen shows version tag` — pins the `v0.1.0 (issue 0147)` string.
### Limits
- **No post-login flow.** OIDC requires a real browser session against MAS; Playwright can't drive that headlessly without storing credentials. If `WAILS_E2E_TOKEN` is ever added, the suite can grow a "logged-in HomeScreen" spec — see issue tracker.
- **No real Wails IPC.** The smoke verifies the static initial render only. Anything that depends on `window.go.*` is excluded by design.
---
## Combined run (local dev)
```bash
# 1. Layer A (skips if no creds)
go test -tags 'integration goolm' -timeout 60s -v \
./projects/element_agents/apps/matrix_client_pc/...
# 2. Layer B (no creds required)
cd projects/element_agents/apps/matrix_client_pc/frontend
pnpm e2e
```
Both layers should be green in <3 minutes on a warm cache.
+33
View File
@@ -0,0 +1,33 @@
import { test, expect } from "@playwright/test";
// UI smoke against the React frontend served by Vite. We don't need a real
// Wails backend: the only runtime call on mount (GetLastUserID) throws because
// window.go is undefined, the catch path drops the loading overlay, and
// LoginScreen renders identically to what the production app shows on first
// launch. This catches HTML/CSS/asset regressions + Mantine bundle errors
// without dragging the full Go+goolm compile into CI.
test("LoginScreen renders with Sign-in button", async ({ page }) => {
// Swallow uncaught Wails-runtime errors so they don't fail the test.
page.on("pageerror", () => {});
await page.goto("/");
await expect(page.getByRole("heading", { name: /matrix_client_pc/i })).toBeVisible();
await expect(
page.getByRole("button", { name: /sign in with matrix/i }),
).toBeVisible();
});
test("LoginScreen mentions homeserver + MAS context", async ({ page }) => {
page.on("pageerror", () => {});
await page.goto("/");
await expect(page.getByText(/matrix-af2f3d/)).toBeVisible();
await expect(
page.getByText(/matrix authentication service/i),
).toBeVisible();
});
test("LoginScreen shows version tag", async ({ page }) => {
page.on("pageerror", () => {});
await page.goto("/");
await expect(page.getByText(/v0\.1\.0 \(issue 0147\)/i)).toBeVisible();
});
+5 -1
View File
@@ -9,7 +9,10 @@
"preview": "vite preview",
"test": "vitest run",
"test:watch": "vitest",
"test:ui": "vitest --ui"
"test:ui": "vitest --ui",
"e2e": "playwright test",
"e2e:ui": "playwright test --ui",
"e2e:report": "playwright show-report"
},
"dependencies": {
"@mantine/core": "^7.13.0",
@@ -20,6 +23,7 @@
"react-dom": "^18.3.1"
},
"devDependencies": {
"@playwright/test": "^1.60.0",
"@testing-library/jest-dom": "^6.9.1",
"@testing-library/react": "^16.3.2",
"@testing-library/user-event": "^14.6.1",
+1 -1
View File
@@ -1 +1 @@
a27c7df4927b39f755bc0add9ca73805
9a66d5a5186912b91bb20602c66c7f8e
+35
View File
@@ -0,0 +1,35 @@
import { defineConfig } from "@playwright/test";
// E2E config — runs Vite dev server (not full `wails dev`) on a fixed port so
// Playwright can hit the React UI without needing the Go backend compiled with
// goolm. LoginScreen + initial App boot render fine because the only Wails
// runtime call (GetLastUserID) throws inside a try/finally that flips the
// loading overlay off. See E2E.md for the rationale + how to swap to `wails
// dev -browser` if/when the full backend is available.
const DEV_PORT = Number(process.env.WAILS_E2E_PORT || 5180);
const BASE_URL = process.env.WAILS_DEV_URL || `http://localhost:${DEV_PORT}`;
export default defineConfig({
testDir: "./e2e",
timeout: 30_000,
fullyParallel: false,
forbidOnly: !!process.env.CI,
retries: process.env.CI ? 2 : 0,
workers: 1,
reporter: process.env.CI ? "github" : "list",
use: {
baseURL: BASE_URL,
headless: true,
trace: "on-first-retry",
screenshot: "only-on-failure",
},
webServer: {
command: `pnpm dev --port ${DEV_PORT} --strictPort`,
url: BASE_URL,
timeout: 60_000,
reuseExistingServer: !process.env.CI,
stdout: "pipe",
stderr: "pipe",
},
});
+38
View File
@@ -27,6 +27,9 @@ importers:
specifier: ^18.3.1
version: 18.3.1(react@18.3.1)
devDependencies:
'@playwright/test':
specifier: ^1.60.0
version: 1.60.0
'@testing-library/jest-dom':
specifier: ^6.9.1
version: 6.9.1
@@ -417,6 +420,11 @@ packages:
peerDependencies:
react: ^18.x || ^19.x
'@playwright/test@1.60.0':
resolution: {integrity: sha512-O71yZIbAh/PxDMNGns37GHBIfrVkEVyn+AXyIa5dOTfb4/xNvRWV+Vv/NMbNCtODB/pO7vLlF2OTmMVLhmr7Ag==}
engines: {node: '>=18'}
hasBin: true
'@polka/url@1.0.0-next.29':
resolution: {integrity: sha512-wwQAWhWSuHaag8c4q/KN/vCoeOJYshAIvMQwD4GpSb3OiZklFfvAgmj0VCBBImRpuF/aFgIRzllXlVX93Jevww==}
@@ -795,6 +803,11 @@ packages:
flatted@3.4.2:
resolution: {integrity: sha512-PjDse7RzhcPkIJwy5t7KPWQSZ9cAbzQXcafsetQoD7sOJRQlGikNbx7yZp2OotDnJyrDcbyRq3Ttb18iYOqkxA==}
fsevents@2.3.2:
resolution: {integrity: sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==}
engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0}
os: [darwin]
fsevents@2.3.3:
resolution: {integrity: sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==}
engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0}
@@ -906,6 +919,16 @@ packages:
resolution: {integrity: sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==}
engines: {node: '>=12'}
playwright-core@1.60.0:
resolution: {integrity: sha512-9bW6zvX/m0lEbgTKJ6YppOKx8H3VOPBMOCFh2irXFOT4BbHgrx5hPjwJYLT40Lu+4qtD36qKc/Hn56StUW57IA==}
engines: {node: '>=18'}
hasBin: true
playwright@1.60.0:
resolution: {integrity: sha512-hheHdokM8cdqCb0lcE3s+zT4t4W+vvjpGxsZlDnikarzx8tSzMebh3UiFtgqwFwnTnjYQcsyMF8ei2mCO/tpeA==}
engines: {node: '>=18'}
hasBin: true
postcss@8.5.15:
resolution: {integrity: sha512-FfR8sjd4em2T6fb3I2MwAJU7HWVMr9zba+enmQeeWFfCbm+UOC/0X4DS8XtpUTMwWMGbjKYP7xjfNekzyGmB3A==}
engines: {node: ^10 || ^12 || >=14}
@@ -1553,6 +1576,10 @@ snapshots:
dependencies:
react: 18.3.1
'@playwright/test@1.60.0':
dependencies:
playwright: 1.60.0
'@polka/url@1.0.0-next.29': {}
'@rolldown/pluginutils@1.0.0-beta.27': {}
@@ -1901,6 +1928,9 @@ snapshots:
flatted@3.4.2: {}
fsevents@2.3.2:
optional: true
fsevents@2.3.3:
optional: true
@@ -1994,6 +2024,14 @@ snapshots:
picomatch@4.0.4: {}
playwright-core@1.60.0: {}
playwright@1.60.0:
dependencies:
playwright-core: 1.60.0
optionalDependencies:
fsevents: 2.3.2
postcss@8.5.15:
dependencies:
nanoid: 3.3.12
+224
View File
@@ -0,0 +1,224 @@
//go:build integration && goolm
// +build integration,goolm
// Integration tests against a real Synapse + MAS. Build tag `integration` keeps
// them out of the default `go test ./...` run; `goolm` matches the runtime
// build tag the app uses (cryptohelper backend). All tests skip cleanly when
// MATRIX_INTEGRATION_* env vars are absent so CI never breaks unattended.
//
// Run with:
//
// MATRIX_INTEGRATION_TOKEN=syt_... \
// MATRIX_INTEGRATION_USER=@egutierrez:matrix-af2f3d.organic-machine.com \
// MATRIX_INTEGRATION_DEVICE=DEVICEID \
// MATRIX_INTEGRATION_TEST_ROOM='!abcd:matrix-af2f3d.organic-machine.com' \
// go test -tags 'integration goolm' -timeout 120s -v \
// ./projects/element_agents/apps/matrix_client_pc/...
//
// MATRIX_INTEGRATION_TEST_ROOM is optional — without it the round-trip test
// auto-skips. MATRIX_INTEGRATION_DEVICE is also optional; if missing the test
// calls /whoami to discover the device_id, mirroring what the app does.
package main
import (
"context"
"crypto/rand"
"encoding/hex"
"fmt"
"os"
"path/filepath"
"testing"
"time"
"fn-registry/projects/element_agents/apps/matrix_client_pc/internal/infra"
"maunium.net/go/mautrix/id"
)
const integrationHomeserverURL = "https://matrix-af2f3d.organic-machine.com"
// integrationEnv reads the required env vars + returns them. If MATRIX_INTEGRATION_TOKEN
// or MATRIX_INTEGRATION_USER is missing the test should t.Skip — that is the
// canonical "no creds available" signal.
type integrationEnv struct {
Token string
User string
Device string
TestRoom string
}
func readIntegrationEnv(t *testing.T) integrationEnv {
t.Helper()
env := integrationEnv{
Token: os.Getenv("MATRIX_INTEGRATION_TOKEN"),
User: os.Getenv("MATRIX_INTEGRATION_USER"),
Device: os.Getenv("MATRIX_INTEGRATION_DEVICE"),
TestRoom: os.Getenv("MATRIX_INTEGRATION_TEST_ROOM"),
}
if env.Token == "" || env.User == "" {
t.Skipf("skipping integration test: set MATRIX_INTEGRATION_TOKEN + MATRIX_INTEGRATION_USER to run (got token=%v user=%v)",
env.Token != "", env.User != "")
}
return env
}
// buildService wires up a MatrixService against the real Synapse, persisting
// the test token in a throwaway keyring namespace so we don't pollute the user's
// production credentials. It returns the service + a cleanup that calls Stop.
func buildService(t *testing.T, env integrationEnv) (*MatrixService, func()) {
t.Helper()
tmpDir := t.TempDir()
// Override user config dir so userStoreDir() lands inside the tmpDir
// instead of clobbering ~/.config/matrix_client_pc/<user>.
t.Setenv("XDG_CONFIG_HOME", tmpDir)
// Use a test-scoped keyring service to keep production creds untouched.
testKeyringService := fmt.Sprintf("fn_registry.matrix_client_pc.test.%d", time.Now().UnixNano())
s := &MatrixService{
store: infra.NewKeyringTokenStore(testKeyringService),
}
s.SetContext(context.Background())
// Persist the token + pickle key under the test keyring so Start() can load
// them the same way the production code does.
pickle := make([]byte, 32)
if _, err := rand.Read(pickle); err != nil {
t.Fatalf("rand: %v", err)
}
tok := infra.Token{
AccessToken: env.Token,
UserID: env.User,
DeviceID: env.Device,
HomeserverURL: integrationHomeserverURL,
PickleKeyHex: hex.EncodeToString(pickle),
}
if err := s.store.Save(env.User, tok); err != nil {
t.Skipf("keyring unavailable (likely no D-Bus session): %v", err)
}
cleanup := func() {
s.Stop()
_ = s.store.Delete(env.User)
// Belt-and-braces: blow away any crypto.db files left behind.
_ = os.RemoveAll(filepath.Join(tmpDir, "matrix_client_pc"))
}
return s, cleanup
}
func TestIntegration_StartAndListRooms(t *testing.T) {
env := readIntegrationEnv(t)
s, cleanup := buildService(t, env)
defer cleanup()
if err := s.Start(env.User); err != nil {
t.Fatalf("Start: %v", err)
}
rooms, err := s.ListRooms()
if err != nil {
t.Fatalf("ListRooms: %v", err)
}
if len(rooms) == 0 {
t.Fatalf("ListRooms returned 0 rooms — user has no joined rooms? unexpected for an integration test account")
}
t.Logf("ListRooms OK: %d rooms (first=%s)", len(rooms), rooms[0].RoomID)
}
func TestIntegration_GetDiagnostics(t *testing.T) {
env := readIntegrationEnv(t)
s, cleanup := buildService(t, env)
defer cleanup()
if err := s.Start(env.User); err != nil {
t.Fatalf("Start: %v", err)
}
d := s.GetDiagnostics()
if !d.Started {
t.Errorf("Diagnostics.Started=false (want true)")
}
if !d.ClientReady {
t.Errorf("Diagnostics.ClientReady=false (want true)")
}
if !d.SyncActive {
t.Errorf("Diagnostics.SyncActive=false (want true)")
}
if d.HomeserverURL == "" {
t.Errorf("Diagnostics.HomeserverURL empty")
}
if d.RoomsCount < 1 {
t.Errorf("Diagnostics.RoomsCount=%d (want >=1)", d.RoomsCount)
}
if d.LastError != "" {
t.Errorf("Diagnostics.LastError=%q (want empty)", d.LastError)
}
t.Logf("Diagnostics OK: rooms=%d encrypted=%d dms=%d", d.RoomsCount, d.EncryptedRooms, d.DMsCount)
}
func TestIntegration_RoundtripMessage(t *testing.T) {
env := readIntegrationEnv(t)
if env.TestRoom == "" {
t.Skip("set MATRIX_INTEGRATION_TEST_ROOM=!abcd:server to enable the send/recv round-trip")
}
s, cleanup := buildService(t, env)
defer cleanup()
if err := s.Start(env.User); err != nil {
t.Fatalf("Start: %v", err)
}
body := fmt.Sprintf("e2e test %d", time.Now().UnixNano())
evID, err := s.SendText(env.TestRoom, body)
if err != nil {
t.Fatalf("SendText: %v", err)
}
t.Logf("SendText OK: event_id=%s body=%q", evID, body)
// Poll LoadTimeline up to 10s waiting for our event to appear.
deadline := time.Now().Add(10 * time.Second)
for {
msgs, err := s.LoadTimeline(env.TestRoom, 20)
if err != nil {
t.Fatalf("LoadTimeline: %v", err)
}
for _, m := range msgs {
if m.EventID == evID {
if m.Body != body {
// In an encrypted room the body may come back blank (EncryptedRaw=true)
// — still counts as round-trip success: the event landed.
if !m.EncryptedRaw {
t.Errorf("event body mismatch: got %q want %q", m.Body, body)
}
}
t.Logf("Round-trip OK: event_id=%s found after %v", evID, time.Since(deadline.Add(-10*time.Second)))
return
}
}
if time.Now().After(deadline) {
t.Fatalf("event %s never appeared in LoadTimeline after 10s", evID)
}
time.Sleep(500 * time.Millisecond)
}
}
// Sanity check that the homeserver is reachable from the test host before we
// burn the rest of the test budget on keyring + sync setup. Skips if env is
// missing so it doesn't run silently in CI either.
func TestIntegration_HomeserverReachable(t *testing.T) {
env := readIntegrationEnv(t)
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
uid, did, err := whoami(ctx, integrationHomeserverURL, env.Token)
if err != nil {
t.Fatalf("whoami(%s): %v", integrationHomeserverURL, err)
}
if uid != env.User {
t.Errorf("whoami user_id=%q want %q", uid, env.User)
}
// device_id may be empty when MAS-issued tokens lack device binding —
// don't fail on it, just log.
t.Logf("whoami OK: user=%s device=%s", uid, did)
_ = id.UserID(uid)
}