CONTROLAgents & automation
AI Assistant with UI Interaction
Ábel Énekes@enekes_abel
0:21This project gives an in-app AI assistant the ability to interact with the user interface by finding, highlighting, and acting on elements, such as logging out.
Gave @fynk's in-app AI assistant hands & eyes 👀🖐️ - my weekend hacking project
Ask "how do I log out?" → it finds the element on your current page, highlights it, and can act on it.
Built with playwright-lite (my Playwright-compatible API for the live DOM) + @TypesafeAI's Jev driving the goal logic 🧵
Unlike most Jev demos I've seen, which go hard on speed, I tried to optimize for usefulness. It was actually fast enough that I deliberately slowed it down so it reads as an assistant guiding you, not a script blitzing through the UI 😅
The bit I'm most proud of: it understood the hidden structure of our app because it's built on our Page Object Model from our e2e tests - it navigated to parts of the UI it can't even "see" yet!
Related builds
