How it works
I saw Claude playing Pokemon and wanted to try the same idea on a phone game. So I built an agent that plays Clash of Clans using only screenshots and ADB commands.
Games drawn with OpenGL have no buttons an agent can read. So it looks at each screenshot, works out where to tap, taps, then takes another screenshot to check.
First run, goal: collect gold and dark elixir. It took 16 screenshots, with a few wrong taps on the way.
The video
What it does
- Sees the game through screenshots and taps through ADB.
- Checks every tap with a new screenshot before moving on.
- Saves a history of each run: numbered screenshots and a log of every action.
- Safety rules: no purchases, upgrades, attacks or gems. If a gem dialog appears, it stops and asks.
- Runs from plain markdown files for rules, workflow and feedback, not a framework.
What stuck
Gold worked right away. Dark elixir sits at the bottom center, so the agent kept tapping the wrong thing. That is why I built the feedback system.
Stack
OpenCode / ADB / Android / LLM vision / Markdown
