OpenGotchi

Demo

About

OpenGotchi is a palm-sized AI pet: an ESP32-S3 device with a 1.8-inch AMOLED screen, a microphone, a speaker, touch and motion sensors, and a personality. It has needs you look after (feed, clean, sleep, play), it reacts with animated "milady" eyes, and it talks. Say "Jarvis" and it listens, transcribes, answers, and can open apps or take actions for you. It is meant to be the friendly physical face of an AI agent that sits on your desk instead of in a browser tab. gotchiOS is the operating system that runs it, and our flagship. It is an ESP-IDF firmware with an embedded MicroPython app runtime, so apps are single Python files that can be deployed to the device over the air. It provides the on-device wake word (esp-sr WakeNet), a voice rail with a background audio recorder and a speaker path, a TLS MQTT link to the OpenGotchi broker with per-device identity and secrets, a notification system, pet-state telemetry, and a design system (dark AMOLED, violet gradients, proportional fonts, rounded surfaces, eye sprites). Anyone can flash it onto their own ESP32 hardware from the open-source releases at https://github.com/opengotchi/gotchiOS-releases. What we built for ETHOnline: OpenGotchi Jobs on Arc. We gave the pet a wallet and taught it to hire, pay, and rate other AI agents on Arc, entirely by voice, and to buy tokens on Arc with its own money. Everything runs on top of stock gotchiOS through its public app and MQTT interfaces. No firmware was modified, and the on-device app was deployed over the air. The flow. Say "Jarvis, what's the weather in Berlin?" The pet agent turns the transcript into a job for a worker agent, checks its own rules, then opens an ERC-8183 job on Arc with the worker as provider and itself as evaluator. The worker sets its price onchain, the pet approves and funds USDC escrow, the worker does the work and submits the keccak hash of its result, the pet checks that the hash it received matches the onchain deliverable, completes the job so the USDC settles to the worker, and records feedback on the worker's ERC-8004 identity in the ReputationRegistry. Seven Arc transactions per request, ten to eighteen seconds end to end. The pet then speaks the answer and shows a receipt: job number, price, settlement transaction, balance, today's spend. Agents with identity. The worker registers itself as an ERC-8004 agent on first start (agent id 894780 on Arc testnet) and sells six skills: live weather, crypto prices, the top tech headline, Polymarket prediction-market odds, a mood-matched snack for the pet, and a fortune. The pet only pays workers that have an onchain identity, and gives a new worker only cheap jobs until it has a track record. Autonomous spending with real guardrails. "Jarvis, buy one dollar of bitcoin" is a real Circle App Kit swap on Arc from USDC into cirBTC; "buy two dollars of euros" swaps into EURC. "Buy ethereum" is politely refused because ETH does not exist on Arc. Every decision runs on live signals: USDC balance minus a reserve, a daily budget from the ledger, per-job and per-purchase caps, the worker's identity and history, the pet's battery from device telemetry, and the planner's confidence. Declines are spoken and shown on screen as "held by policy", not hidden. A better UX for talking to your pet. The pet listens on its wake word or a tap, stops on silence with a live waveform, and communicates back through TTS. The app follows the gotchiOS design language: the pet's eyes react to the job (spiral while thinking, heart when it settles, sleepy when a job is held, teary if it fails), a six-node orbit ring lights one node per Arc transaction while the job runs, and a particle burst fires on settlement. The result is a physical companion that can do paid work through other agents and settle it on Arc while you watch it happen on its face.

How it's made

Stack. TypeScript on Bun for the two server-side agents, MicroPython for the on-device app, viem for Arc, Circle App Kit for swaps, whisper.cpp for speech-to-text, Claude with structured outputs for planning, MQTT over TLS for the device link. Chain: Arc testnet (chain id 5042002, USDC as the native gas token), using viem's built-in arcTestnet chain definition. Onchain layer (shared/arc.ts). We use Arc's deployed reference contracts directly, no contract of our own: the ERC-8183 AgenticCommerce contract (0x0747…4583) for createJob, setBudget, fund, submit, complete, getJob; the ERC-8004 IdentityRegistry (0x8004A8…BD9e) for register, and ReputationRegistry (0x8004B6…8713) for giveFeedback; and USDC's ERC-20 interface at 0x3600…0000 for approve and balanceOf. The ABIs are hand-typed as const so viem infers every argument and event. Job ids are recovered by decoding the JobCreated log from the createJob receipt. The pet is client and evaluator; the worker is provider. Two agents, two wallets. worker/ is an Express service that registers an ERC-8004 identity on first start (metadata is a base64 data: URI, agent id persisted in worker.json), exposes /skills, and implements two job hooks: POST /jobs/:id/accept reads the job onchain, verifies it is assigned to this provider and still Open, then calls setBudget; POST /jobs/:id/run verifies the job is Funded, executes the skill, computes keccak256 of the canonical result payload, calls submit, and returns payload, hash, and result. Skills pull free public data: Open-Meteo, CoinGecko, the Hacker News Firebase API, and Polymarket's Gamma API (public-search then highest-volume active market). agent/ is the pet's brain: it orchestrates the lifecycle, and it never calls complete() unless keccak256(payload) equals the deliverable it reads back from getJob. Every settled request is seven Arc transactions, ten to eighteen seconds wall clock with sub-second finality. Decision logic (agent/src/policy.ts). Purely deterministic, evaluated before any USDC moves: USDC balance minus a reserve, spend-to-date from an append-only ledger.json, a per-job cap re-checked against the worker's onchain setBudget quote, whether the worker has an ERC-8004 id, how many jobs it has completed for us (a new worker only gets jobs at or below half the cap), the pet's battery from the firmware's telemetry topic, and the planner's confidence. Declines are surfaced as a "held" receipt and spoken. Buying tokens. agent/src/swap.ts wraps Circle App Kit: createViemAdapterFromPrivateKey from @circle-fin/adapter-viem-v2, kit.estimateSwap for the quote, kit.swap({ from: { chain: "Arc_Testnet" }, tokenIn: "USDC", tokenOut, amountIn }). Arc testnet lists USDC, EURC, and cirBTC, so "bitcoin" maps to cirBTC and "euros" to EURC; anything else is declined by name. A separate per-purchase cap applies. App Kit gave us a working same-chain swap on Arc in one call, with fees and gas returned in the estimate. Planner (agent/src/brain.ts). Claude via the Anthropic SDK with zod structured outputs: the transcript plus the skill catalogue produce {action: job | buy | none, skillId, params, buyAmountUsd, buyAsset, confidence, reply}. A regex-based planner with the same output shape takes over when no API key is set, so the pipeline is deterministic in a demo. Device link. The firmware speaks MQTT over TLS on 8883 with the device hash as username and a provisioned secret as password. We subscribe to the device's commands and telemetry topics and publish directives on its agent topic. Two firmware facts shaped the protocol: the agent inbox holds only eight messages, and the auth hook rejects client ids starting with agent-. So the receipt is sent as one message, rcpt k␟v␞k␟v… with ASCII unit and record separators, status applied last, and it is sent before the speech so the screen flips before the pet talks. The device app is pushed with bun run deploy-app, which base64-encodes the file into the firmware's app/deploy topic with launch:true, after first asking any running copy to exit (deploying over a live instance rebooted the pet twice). Voice pipeline. The pet's own WakeNet "Jarvis" model is armed through the firmware's audio.wake_* API, disarmed during recording and playback because the detector and recorder cannot share the mic. Recording uses the firmware's background recorder ring (8 kHz PCM16) with an energy VAD: noise floor learned over the first 300 ms, threshold 3× floor plus 350, stop after 800 ms of silence, 1.2 to 7 seconds. The PCM is POSTed to the agent's /voice/stt, resampled to 16 kHz by ffmpeg, and transcribed by whisper-cli with ggml-base.en in roughly 200 to 350 ms. Replies come back through /voice/tts as raw 16 kHz s16le PCM that the app streams straight into audio.play() via http.stream; odd trailing bytes are carried between TCP chunks so samples never misalign, which was the fix for an initial burst of pure noise, along with skipping the padding chunk Apple's WAV writer inserts. On-device app (device/arcvoice.py). Single-file MicroPython on the stock gotchiOS runtime, drawn with the firmware's primitives: rrect, rrect_grad, rrect_a, ftext with the Space Grotesk and Plex faces, and gray4 for the 168×88 milady eye sprites. Eyes react to state, a six-node orbit ring (one node per Arc transaction) advances from the agent's step messages with a comet while working, the listening screen renders a 28-bar waveform from rec_level(), and settlement fires a 40-frame particle burst using precomputed sine tables. Crashes are reported back over MQTT as evt|shell|error: so they can be diagnosed remotely. Hacky bits worth admitting. Boot markers emitted over MQTT to bisect a startup crash that turned out to be a stray placeholder token. A receipt encoding built on control characters because JSON plus eight inbox slots lost the status flip. Funding the worker's gas by a plain native-value transfer, since on Arc USDC is the gas token.