PetV3 — From a Desktop Pet to an Agent Factory

Ongoing project · August–September 2026

PetV3 began as an experiment in making work across ChatGPT Desktop and Claude Desktop visible and easier to coordinate. It was not an attempt to build an “AI factory”: the first question was whether a desktop character could show what the providers were doing, accept a reply from the same surface, and help me inspect several AI-assisted tasks.

As those capabilities accumulated, the project grew toward an agent-work coordination and control-plane form. That direction is still emerging, not a declaration of finished autonomy.

The answer so far is mixed but concrete. There are rendered desktop demos, a local overlay, and implemented pieces for provider activity, delivery confirmation, session/usage records, isolated work, and safe handoff. There are also failures, uneven behavior, and limits that matter just as much. This is not a finished control plane and cannot be left alone to assign, review, merge, and recover every task.

August 29 — start with the real desktop

The first Tauri/React plan produced a browser-like control window before the pet itself was reliably visible and interactive on Ubuntu GNOME Wayland with XWayland and two 6K displays. That was the first useful failure: the real operating environment, rather than a mock interface, had to be the acceptance gate.

Rendering and input moved to a native Rust boundary. The portable core owns coordinates, state, animation, and IPC; the Linux surface owns windows, input, monitors, and transparent composition through X11/XRender/XFixes/XRandR. An early 20-second release demo measured 28.3 presented fps and 38.7 MiB RSS; an ARGB always-on-top overlay was checked on the second display at (6500, 200). Those are historical demo measurements, not a claim about current performance.

This was also a learning project. I had a software and distributed-systems background, but not prior Rust, MCP, or prompt-operated multi-agent experience. The practical pattern emerged through conversations, small attempts, inspection, and correction: decide what should count as evidence, give a bounded instruction, inspect the result, then revise the plan when the real behavior disagreed. Fast implementation did not remove responsibility for scope, review, or the choice to retry.

The status language became deliberately modest. working? meant recent activity, not proof that a process was alive; no progress evidence meant missing data, not zero progress; and “implemented but not called” separated implementation from integration. If a Wayland limitation prevented visual confirmation, the honest state remained “human verification required.” Those phrases were not cosmetic caution—they preserved information another session or provider could use without converting uncertainty into success.

August 29–30 — turn parallel work into a verifiable landing path

Once tasks moved at different speeds, the problem became avoiding false success. A branch can have commits without passing its gate; a feature can exist without being called; a dashboard can run an old binary; files can change after a check but before a commit.

The workflow therefore became:

isolated worktree → visible overlap ledger → central landing gate

Each worker gets a closed task, branch, and worktree rather than a broad stash that could hide someone else's changes. The ledger records expected file overlap without acting as a lock, so the controller can inspect competing intentions instead of silently choosing one. land.sh checks the worker tip, rechecks the starting main state, merges, reruns the check, and verifies that main HEAD links independently. That final check came from a reproduced failure where a narrow package gate passed but another crate did not handle a new enum variant.

Isolation had to include build artifacts too: source archives, Cargo targets, TMPDIR, logs, and retained diagnostics. The same lesson applied to documentation: independently appended index lines conflicted, so history became one event file per record with a generated index.

The landing path was intentionally transactional rather than a convenience wrapper. A worker branch ran its scoped gate; the controller rechecked the original main HEAD and clean state; the change merged without unresolved conflicts; the same gate ran again on the merged result; finally main HEAD had to link on its own. This last step came from a reproduced incident: a narrow package gate passed while another crate could not handle a new enum variant. The branch and merged gates passed, the standalone link failed, and the safe starting point was restored. A gate only earns trust when it catches the failure it says it catches.

Source isolation alone was insufficient. Two archived trees once shared a Cargo target, and a reused binary retained an old CARGO_MANIFEST_DIR that pointed at an archive already removed. A landing therefore isolated branch and merged source archives, branch and merged targets, a Rust temporary directory, stage logs, and retained failure diagnostics. Successful runs removed only their own run root; failures retained evidence, and the next landing was refused at a retention limit rather than deleting diagnostics automatically.

The same concurrency rule applied to records. Workers appending a shared history index collided even when their event files did not, and “next sequence number” also collided. The replacement was one YYYY-MM-DD-slug.md event file per record, a generated index, and landing commits derived from Git rather than typed by hand. Documentation is subject to the same parallelism failures as code.

August 30–September 1 — make evidence and control visible

The terminal dashboard joined task, worker, file, backlog, ledger, and session evidence without adding another browser runtime. Its bounded view reveals detail progressively instead of expanding every file and conflict at once:

TASK R-3  enforce overlay tick order
 └─ WORKER work/r3-tick-order  committed · waiting to land
     └─ 3 files · 0 conflicts

It deliberately reports evidence rather than invented percentages: a fresh claim is working?, a branch commit is committed · waiting to land, and absent evidence stays absent. Animation refresh is separate from slower Git/session collection so visual activity does not require rescanning large records each frame. A stale release binary and an over-expanded first UI were both real failures that changed the launcher and dashboard design.

The useful parts did operate: work was distributed, separate work areas were preserved, progress could be inspected, and sessions could be stopped and continued. The behavior was not consistent. Some dashboard states required retries, some instructions drifted, and an agent-modifying feature did not work as expected. Human direction, review, and the decision to retry or stop remained essential.

The dashboard was a terminal TUI rather than another browser runtime because the controlled things were already local Git, processes, and terminal sessions. The first task graph expanded every file and conflict; it scrolled without a useful bound and moving targets made mouse interaction unreliable. The later view kept the task → worker → file relationship but revealed it progressively, with +N more for long file lists and a renderer that clipped an already-built graph to the screen height.

Progress remained evidence-based. A fresh file claim was working? · fresh claim; a WIP commit was still working?; a normal branch commit was committed · waiting to land; absent evidence remained absent. The question mark mattered because an mtime is not a liveness check. Animation ran from an injected 160ms frame phase while Git, backlog, ledger, and session candidates refreshed on a slower clock, keeping rendering deterministic and preventing multi-gigabyte session records from being reread for every frame.

There were concrete operating failures. Returning from another task could leave an I-beam cursor and misaligned clicks, so focus gain, resize, and editor-resume paths had to restore raw mode, mouse protocol, cursor state, and a full repaint through one re-entry path. A newer dashboard landed while the screen still showed an old UI because the launcher reused any release binary it found; it was changed to compare source and binary at startup and explain a rebuild. The dashboard displayed provider and concise verified model information, but deliberately omitted full prompts, working directories, and resume IDs.

One release benchmark for the progress-evidence work recorded 16 worktrees, 1,398 tracked files, and 1,684 session candidates totaling about 2.28 GB. Under those conditions the median read loop was 5.47 ms (maximum 10.38 ms), render loop 0.38 ms, input-to-rerender 0.45 ms, with zero idle render forks and 12,592 KiB RSS. These are conditioned measurements from that run, not a general performance promise.

September 1 — budgets, checkpoints, and handoff

The governor separates orchestration policy from product performance. Model intensity and worker admission depend on closed tasks, conflicts, supplied usage readings, and local resources; they are not a claim about official provider limits. Unknown usage is never treated as zero. A controller can record a usage_cap pause, and a later session can honor it, but neither provider is presented as autonomously reading every subscription gauge.

A current finding from the Claude Fable 5.1 resume context is that agent-run control scripts need an explicit project-local permission contract. Without a narrow allow-list for dashboard control, routing, the ledger, checkpoints, and refactor audits, the workflow can stop at an approval boundary instead of proceeding predictably. The project already has entries for those five scripts in .claude/settings.local.json; that boundary makes execution auditable, not autonomous or guaranteed. Any configuration describing it documents the intended contract rather than a newly applied permission.

The project-local Claude boundary has this deliberately narrow shape:

{
  "permissions": {
    "allow": [
      "Bash(scripts/dash-control.sh:*)",
      "Bash(scripts/agent-route.sh:*)",
      "Bash(scripts/ledger.sh:*)",
      "Bash(scripts/checkpoint.sh:*)",
      "Bash(scripts/refactor-audit.sh:*)"
    ]
  }
}

It permits these scoped control scripts; it does not install them, grant broad execution permission, or prove autonomous operation.

A pause is paired with append-only worklog and resume records. They preserve outcomes, command results, next actions, and stop conditions; a new ChatGPT/Codex or Claude session still has to remeasure Git and control state before accepting a checkpoint as current. The handoff procedure preserves unfinished work rather than forcing a commit, merge, cleanup, or build at the stopping point.

The dated September 1 policy used daily internal caps from 1%p (small) to 5%p (max) of a planning gauge, with five-hour values derived by multiplying by 5/24: 0.208333, 0.416667, 0.625000, 0.833333, and 1.041667%p. They were not observed provider limits or a claim about the current account. Small deterministic work could use Luna or Haiku; intermediate judgment Terra or Sonnet; Sol or Opus was reserved for control review such as state machines, security boundaries, or conflict mediation. The queue was a soft ceiling, reduced further by closed tasks, file conflicts, available usage, and local resources.

The controller obtained usage from /status or /usage before and after a wave; the repository did not independently read every subscription gauge. unknown was never converted to zero: it allowed one small calibration wave and then no next wave while still unknown. stop_on_cap=true was the default, with manual and usage_cap kept distinct. An observed pause at control revision 43 followed a user-supplied approximate gauge; unavailable daily-baseline and five-hour details were not fabricated. A later Claude session honoring the pause demonstrated handoff continuity, not unattended usage detection.

control.ini was untracked source of truth for pause, mode, brainstorming, and refactor state. A poll such as scripts/dash-control.sh poll --since 42 returned zero bytes when the revision did not change and one small JSON snapshot when it did. That reduced model-context cost separately from rendering cost. Every sixth landing required a diff-only refactor-necessity audit; a seventh was refused until the audit recorded none, inspect, or refactor. Even refactor did not authorize a code change—the need assessment and user authorization stayed separate.

Checkpoints used matching worklog and resume files. The first preserved user-visible outcomes, exact commands, decisions, and approvals; the second preserved an explicitly unverified snapshot, next actions, the usage ledger, and stop conditions. Publishing staged only that pair in a detached handoff worktree and rejected unfinished markers or key-like strings. A resuming controller read instructions, routing policy, checkpoint records, and backlog, then remeasured current Git and control state instead of treating checkpoint HEAD or dirty state as current fact.

The human role did not become “prompt and wait.” It was to define closed tasks, set evidence standards, inspect actual results, preserve foreign work, decide when a failure remained open, and retain authority over refactoring, pause, and resumption. The models produced a surprising amount once those boundaries were clear, but neither implementation speed nor a successful-looking report was a substitute for that judgment. The useful result is an operating experiment with real artifacts and real counterexamples—not a claim that an autonomous factory has been completed.