PictoCode

Draw a rough app with your finger, then talk to it — the agent sees your sketch (and the running app), turns your voice commands into edits, and iterates until you say "export." No keyboard, ever. Built on Expo so the generated app is portable, runnable-on-device, and exportable.

The idea in one line

The agent never writes raw code — it emits a UISpec (JSON) that a native interpreter renders. That's what makes generation instant, safe to hot-swap, cross-platform, and trivially exportable. Voice + a screenshot of what's in focus drive the whole loop.

Run it — needs a dev build (voice = native modules)

On-device speech recognition, view screenshots, and file export are native modules, so this no longer runs in Expo Go. You build a custom dev client once and install it on your device:

cd pictocode
pnpm install
# install the native deps at SDK-correct versions:
pnpm expo install expo-speech-recognition react-native-view-shot expo-file-system expo-sharing expo-haptics

# build & install a dev client on your device (one time), then start the dev server:
pnpm dlx eas-cli@latest build --profile development --platform ios   # or: android
pnpm expo start --dev-client

Open the installed PictoCode dev build (not Expo Go) and scan the QR. Everything works on the offline engine with no API key — including on-device voice, which is local to the phone.

Turn on the real Claude agent

cp .env.example .env
# edit .env: set EXPO_PUBLIC_GENERATOR=claude and your EXPO_PUBLIC_ANTHROPIC_API_KEY
pnpm expo start --dev-client -c   # -c clears the cache so the new env is picked up

The badge in the top-right flips from ● Offline to ● Claude. If a Claude call ever fails, it transparently falls back to the offline engine so the demo never dead-ends. With a key, the agent sees your sketch and app screenshots; offline, it routes commands by keyword and tidies the stroke summary.

Security: EXPO_PUBLIC_* vars are baked into the client bundle. Fine for a local demo with a throwaway key. For anything real, stand up a tiny proxy that holds the key server-side and point EXPO_PUBLIC_ANTHROPIC_ENDPOINT at it.

The loop (all voice — hold the mic and talk)

1. Draw — sketch a rough layout with your finger (top = header/search, middle = content, bottom = tab bar/button).
2. Talk — hold the mic and speak. The command routes automatically:
   • "a coffee shop ordering app" → generate (sends your sketch as an image).
   • "clean up my sketch" → polish (redraws it as a clean wireframe on the canvas).
   • "make the header blue", "rename the button to Order now", "delete the search bar" → edit (the agent sees a screenshot of the running app and finds the element itself — no tapping).
   • "undo" / "clear" → canvas controls.
   • "export it" / "ship it" → export.
3. Focus — tap the Sketch or App card to choose which surface the agent screenshots for the next command.
4. Export — say "export" and the standalone App.tsx is written and handed to the OS share sheet (AirDrop / Files / Messages).

Architecture

App.tsx                      the shell: canvas + preview + the voice command loop
src/types.ts                 UISpec + Wireframe + Command (the contracts)
src/render/SpecRenderer.tsx   UISpec -> native components (the interpreter)
src/canvas/SketchCanvas.tsx   finger drawing, PNG capture, wireframe render-back
src/voice/useVoice.ts         on-device speech-to-text (expo-speech-recognition)
src/capture/capture.ts        screenshot the focused surface (react-native-view-shot)
src/export/share.ts           UISpec -> App.tsx -> OS share sheet
src/generator/
  index.ts                   dispatcher (claude | mock, always falls back)
  claude.ts                  Anthropic Messages API: generate / edit / polish / director
  mock.ts                    offline engine + keyword command router
  schema.ts                  system prompts (UISpec vocab, polish, command routing)
  export.ts                  UISpec -> standalone App.tsx (export ladder rung 1)

Export ladder

• Rung 1 (in the app now): your app as a standalone App.tsx — drop it into a fresh Expo project and it runs.
• Rung 2: eas build that project → an installable APK (Android, direct install) or IPA (iOS, needs signing). A real, separate app on the device.
• Rung 3: eas submit → the App Store / Play Store.

Roadmap to the native-grade version

Now that it's a dev build, the remaining rungs:

• Better inking — swap the SVG canvas for expo-pencilkit-ui (native PencilKit) on iOS.
• Live OTA export — publish generated apps via EAS Update and open them by link.
• Spoken responses — expo-speech TTS so the agent talks back, closing the fully hands-free loop.

Now shipped: the whole loop is keyboard-free. On-device speech-to-text (expo-speech-recognition) turns voice into commands; a director agent routes each command to generate / edit / polish / export; react-native-view-shot screenshots whatever surface is in focus so the agent sees what you see; edits are found visually (no tap-to-select); and export writes a standalone App.tsx straight to the OS share sheet. Multimodal generate + sketch polish (Claude redraws your scribble as a clean Wireframe on the canvas) are in too.