CONTROLAgents & automation
JEV for Interactive Browser UI
BlackwellBoy@Blackwellboy
This project explores the effectiveness of JEV (likely a browser automation tool) for handling interactive browser elements like buttons, forms, and dynamic UI, distinguishing its utility from static web fact extraction.
Ive spent the last couple days properly fucking around with JEV and i think ive finally got a decent idea of what it is actually good at and more importantly what i can stop wasting time trying to make it do 😂
First thing, JEV itself wasnt really the problem.
My actual browser policy was already doing 30/30 on the first set and 29/30 zero shot on a completely different set. no fake DONEs, no random BLOCKEDs, no stale clicks...the one miss is still sitting there unfixed on purpose because i dont want to just patch the test until it says 30/30.
then i shoved it into Hermes and it looked like it completely fell apart.
0/30.
which obviously made me think okay cool ive just built a very fast browser agent that turns into a potato the second Hermes touches it.
except that wasnt what happened at all.
Hermes just wasnt fckin calling it 😂
the tool was there. the plugin was there. the schema was there. but give the 5090 controller a big pile of tools and tool_choice=auto and half the time it basically goes "nah i know better" and writes some text instead.
once i actually forced the call and looked at real tool receipts instead of trusting stdout, the interactive stuff went 22/22.
so that killed one idea straight away:
JEV should not be used for everything.
static web facts? normal search/extract is better.
buttons, forms, dropdowns, delayed UI, overlays, weird DOM shit? thats where JEV starts making sense.
then i tested whether we needed some clever learned router to decide between the two.
turns out no lol.
a stupid free deterministic rule got 60/60.
Laya got 73%.
and the bad mistakes were exactly the ones you dont want, sending interactive browser work to a static web path.
so another thing crossed off the list:
dont pay a model to make a decision an if statement can make perfectly.
Laya was interesting though.
its tiny, fast as hell, around 20ms in my testing and uses way less VRAM than SemIf.
for about 5 minutes i was thinking this might be the thing that lets us keep the browser policy permanently resident somewhere.
then we actually tested the quality.
SemIf: 0.85 on the small semantic set.
Laya english: 0.55.
typed version: 0.35.
and its confidence is way too optimistic as well.
so yeah, very fast at being wrong is still wrong...
Short context hurts it too. as we gave it more of the browser state it went 0.30 -> 0.35 -> 0.40, which pretty much confirmed one of the obvious problems. these browser decisions need more context than a tiny decision model wants to see.
So SemIf stays for now.
not because its prettier or newer or whatever. it just wins the actual job.
the architecture ive ended up liking is stupidly simple now:
free rules decide static vs interactive
static stuff goes to normal web search/extract.
interactive stuff goes to JEV.
JEV uses SemIf for the fast browser choices.
the 5090 only gets dragged in when we actually need language, like TYPE_TEXT.
and Hermes gets a tiny verified result back instead of the entire DOM and 40 pages of browser diarrhoea.
thats the next thing im testing properly now.
the first compact-vs-full test was basically useless because i used a one click job. the "compact" result was nearly the same size as the full one, so obviously it saved fck all.
now im doing it again on an actual multi-step browser job with the real action history included.
if compact packets save a meaningful amount of context while still giving Hermes enough information to continue the job, thats probably the bit i care about most.
because the whole point isnt to make Hermes watch every mouse movement.
its to give it a worker that disappears into the browser, does the annoying shit, proves it worked, and comes back with basically:
"done. verified. heres the proof."
the funniest part of all this is how many things ive ended up not using.
not Laya as the main browser policy.
not Laya as the router.
not JEV for static web facts.
not a learned router where free rules work.
not dumping browser traces into Hermes.
and definitely not changing the default Hermes setup just because one experiment looked cool.
so far JEV looks less like "replace the agent" and more like a really fast specialised nervous system for interactive UI.
which honestly is probably more useful anyway.
Related builds
