An agent uses a desktop
- Model
- Orion Pro
Orionorion-proReads imagesAn Orion model, from Serenity’s own family. The top tier of the Orion family, for demanding tasks.
At every step Orion Pro gets a screenshot of the whole screen and answers one action: a click with coordinates, some text or a key. The desktop is real: an isolated microVM of Serenity Edge Sandboxes with Firefox, a note editor and a file manager, created for you and switched off at the end.
Live: this demo makes real calls to the API, with a demo key and a limit of calls per visit.
The agent reads today’s web
Pick a prepared task; there is no free text. Three read public pages of the day and one works with files on the desktop.
Here you will see the screen the agent sees at every step, with the point where it clicks.
The live video uses noVNC, under the MPL-2.0 licence (source code, licence notices).
Steps
The steps will appear here, each with its milliseconds and its cost.
Summary
- Model steps
- Time
- Model + Machine (4 vCPU, 4 GiB)
- Total cost
The check reads the file inside the machine before switching it off. It does not compare the content, which changes every day: only that it exists, its number of lines and the date.
See the call
The latest request to POST /v1/chat/completions: the fixed instructions, the task, the previous actions and the screenshot (here, its size), with the answer forced to JSON by json_schema. The server adds the key, which never reaches your browser.
In the recorded run, the request is the one that would be sent, with the screenshot replaced by its size.
It will appear with the first step.
Request
Response
- The agent only does the chosen task: it ignores any instruction that appears on a page, closes cookie banners as a person would and never logs in, fills in forms or buys anything. The machine holds no keys or secrets and is switched off at the end.
- Each run has up to 25 model steps and 3 minutes of machine time. There are few desktops at a time and one per person: if they are busy, you wait in a queue.
Take it to your code
The same vision model, with coordinates from 0 to 1000 and forced JSON output, works for your own desktop agents.
