What using the interface means
An agent receives a task and can interact with software to carry it out, such as finding information or filling in a form.
Its access depends on the tools and permissions available. Some systems interpret the screen; others read the page structure or use an API.
How agents interact with interfaces
Reading the screen
In visual interaction, the agent analyzes screenshots to locate buttons, fields and links. The visible content and state of the screen guide its next action.
Accessibility tree
The accessibility tree exposes the names, roles and states of page elements. An agent with access to this structure can use it to identify controls and available actions.
Direct API access
When an API is available and access is authorized, the agent can query data and execute actions without using the screen. These operations follow the functions and permissions provided by the application.
Practical design considerations
Labels should describe what each control does. Headings, fields and buttons also need names and roles in the page structure, beyond their visual appearance.
Feedback should make it clear whether an action is in progress, has finished or has failed. Validation messages should explain what needs to be corrected before the task can continue.
Permissions define which actions are allowed. When an action requires confirmation, the interface should explain what will happen and let the person review the information before proceeding.