Automation
cmd/keel-mcp is an MCP server. After being connected, the Agent can start the Keel application, read out what is in the window, click, enter, press keys, scroll, take screenshots, and walk through the application like a human.
Two modes:
- Visible Mode: The window is displayed on the screen normally. You can watch the Agent operate it, or you can do it yourself and then let the Agent do it.
- Interfaceless mode: The window does not appear on the screen and is only rendered in memory. Suitable for running tests in batches without disturbing you.
Both modes do not use a real mouse and keyboard, nor do they require system permissions such as accessibility functions. A complete "input → click → open new window → save → close" process takes about 1.5 seconds (excluding compilation).
Access
go install github.com/dyike/keel/cmd/keel-mcp@latest
claude mcp add keel -- keel-mcp
When modifying keel-mcp itself, use go install ./cmd/keel-mcp to install the local version in the repository.
claude mcp add will write Claude Code's configuration. For other clients that support MCP, just configure keel-mcp as a stdio type server. keel-mcp does not require parameters; which application to test is specified in the launch tool.
What does a test look like?
Agent calls launch with parameter {"command": "go run ./examples/components -section multiwindow -lang zh-CN"} and gets:
Window w1 "Keel · 组件" 680×860
e1 text "Keel · Gio" @24,24 582×34
e2 text "你的名字" @42,88 546×21
e3 textbox "你的名字" value="" @42,115 546×40
e4 button "打招呼" @42,167 74×38
e5 button "打开设置窗口" @124,167 116×38
e6 text "主窗口和设置窗口拥有各自的状态。" @42,217 546×24
e7 text "⌘ + , 打开设置。" @42,253 546×21
Each line is an element: reference (e3), role, name, status, position (@x,y width×height, unit dp). Then:
type {"ref": "e3", "text": "小明"}
click {"text": "打招呼"} → e6 text "你好,小明!"
press_key {"key": "mod+,"} → Open windows: w1 "Keel · 组件" (shown above); w2 "设置" (new)
click {"window": "w2", "text": "保存"}
screenshot {"window": "w2"} → PNG
Each operation returns the element list after the operation, and the Agent does not need to call snapshot separately to confirm the result. When an action opens a new window, (new) is marked at the end of the list.
Operate applications you launch yourself
The application is displayed on the screen normally. You first manually operate it to a certain state, and then let the Agent continue to do it. You watch it operate:
KEEL_AUTOMATION=1 go run ./examples/components -section multiwindow -lang zh-CN
After the application starts, print the connection address, such as keel automation: listening on /var/folders/…/T/keel/components-4821.sock. Then have the Agent call attach (without parameters):
- When only one application is running, it connects directly and returns its current window and elements;
- If there are multiple addresses, list all addresses and let the Agent select one and pass the
socketparameter; - When not found, confirm that the application was started with
KEEL_AUTOMATION=1and did not exit.
Differences from launch:
launch |
attach |
|
|---|---|---|
| Who starts the app | keel-mcp |
You |
stop |
End the application | Just disconnect, the application continues to run |
| Connect again | Restart, the status is cleared | attach again, the status is retained |
logs |
Can see the output | Can't, the output is in the terminal you started it from |
An application can only accept one connection at a time. When another Agent session is already connected, attach will report "Another client is connected" after 3 seconds and will not stay stuck.
Each operation of the Agent will be immediately reflected in the window on the screen: the entered text, the result of the click, and the newly opened window. Your operations in the window can also be seen when the Agent reads it next time.
If you do not want to display the window, add KEEL_HEADLESS=1.
KEEL_AUTOMATION can also directly write the socket path (KEEL_AUTOMATION=/tmp/my.sock). In this case, attach will pass the same socket.
Tool
| Tools | Parameters | Function |
|---|---|---|
launch |
command, dir?, timeout_seconds?, visible? |
Start the application with a shell, wait for it to be ready, and return to the first window. The window is not displayed by default; visible: true is displayed on the screen. The previously launched application will be stopped first. Compilation time is included in the timeout, default is 120 seconds |
attach |
socket? |
Connect an app you launched with KEEL_AUTOMATION=1, preserving its current state |
snapshot |
window? |
Elements in the window |
click |
ref or text or x+y |
Left-click the center of the element or specify the coordinates |
type |
text, ref?, clear? |
Enter at the cursor. Give ref click to focus first; clear first select all and replace with new text |
press_key |
key |
Press a key combination: enter esc tab shift+tab space backspace up mod+a ctrl+shift+s. Window shortcut keys will trigger and Tab will move focus |
scroll |
dy, ref? or x+y |
Scroll wheel, positive number downward, unit dp. Defaults to window center |
wait_for |
text, timeout_ms? |
Wait until an element's name or value contains text. Used for background goroutine update interface scenarios, default 5 seconds |
screenshot |
window? |
Window screenshot, 1 pixel = 1 dp |
windows |
List open windows | |
close_window |
window? |
Close the window like clicking the close button. Close the last one and the application exits |
logs |
lines? |
Applies the latest standard output and error output: panic, log, compile error. Only valid for applications launched by launch |
stop |
Application of launch: End it; Application of attach: Disconnect only |
Except for launch, attach, logs and stop, window (such as w2) can be used. When omitted, it will act on the current window: the window that was last operated; if it has not been operated yet, it is the most recently opened window.
Element
| role | from | additional information |
|---|---|---|
text |
el.Text、kit.Kbd |
|
button |
kit.Button, el.Div with OnClick |
disabled; value: loading when the kit is loaded; the selected button (Selected(true)) has selected: true, and is not listed when it is not selected |
link |
kit.Link |
|
textbox |
kit.Input, kit.TextArea, el.Input |
value; the values of the password boxes are equal length • |
checkbox |
kit.Checkbox |
checked / unchecked; when half selected, value is mixed |
radio |
Every option for kit.RadioGroup |
checked / unchecked |
switch |
kit.Switch |
checked / unchecked |
select |
kit.Select |
value is the currently selected item; option will appear after clicking it |
option |
Expanded drop-down options | selected |
tab |
kit.Tabs's tags |
selected |
table |
kit.Table |
value is the total number of rows, such as 36 rows |
columnheader |
Header, click to sort | |
row |
The rows visible in the table are named with vertical lines connecting the columns | selected |
slider |
kit.Slider |
value is the current value; click the track or focus and press the direction keys to adjust |
accordion |
kit.Accordion |
Title and expanded content listed separately |
tag |
kit tag container | value is neutral/info/success/warning/danger, selected is the selected state; the select and remove buttons are listed separately |
group |
kit component grouping | The name is the group title, retaining the semantics of sub-components |
status |
kit status bar | The left and right content are listed separately as sub-elements |
alert |
kit inline prompt container | The name is the title, the value is neutral/info/success/warning/danger; the text and close button are listed separately |
badge |
Numbers, dots, icons (not clickable) | The name is the original count, value is the displayed value, dot or icon |
toggle |
Status button | selected means selected, supports disabled |
disclosure |
Collapse panel title | value is expanded / collapsed, supports disabled |
avatar |
kit avatar | The name is the person's name, the value is online/busy/offline, and the status is empty |
image |
kit.Image, Markdown image |
value is loading / loaded / error, the name is the alternative text |
footnotes |
Markdown footnotes | Quotes and backlinks listed separately |
progressbar |
kit.Progress |
value is a percentage or indeterminate |
dialog |
The open panel of kit.Dialog, kit.Sheet, kit.Popover, kit.HoverCard |
The text and buttons in it are listed separately |
alertdialog |
ConfirmDanger and Persistent() of kit.Dialog |
The point mask is not closed, Esc equals to cancel; the elements inside are listed separately |
radiogroup |
kit.RadioGroup |
Each option is radio, listed separately |
listbox |
Open list of kit.Select / kit.Combobox |
Option is option, listed separately |
combobox |
kit.Combobox |
value is the current selection; the text box and expand button inside are listed separately |
grid / gridcell |
kit.Calendar |
Every day is gridcell, the name is the date, selected means selected or within the range |
list / step |
kit.Stepper |
Each step of value is done / current / upcoming |
form |
kit.Form |
Row labels and controls are listed separately, and controls are named after the row label |
tree / treeitem |
kit.Tree |
The node value is expanded / collapsed, selected means selected |
tablist / tabpanel |
kit.Tabs |
The label is tab; the panel is named after the label title |
navigation |
kit.Pagination |
value is "current page/total number of pages" |
log / article |
kit.MessageScroller / kit.Message |
The message is named after the author and the content is listed separately |
attachment |
kit.Attachment |
The name is the file name, value is the upload progress or error |
separator |
Separator bars of kit.Resizable, kit.Dock, kit.Sidebar |
value is the size of the first panel (Resizable) |
Dock area for region |
kit.Dock |
Name is the current panel title; labels, menu buttons, and content are listed separately |
toolbar |
kit.Toolbar |
Buttons listed separately |
banner |
kit.TitleBar |
The name is the title; window buttons and application content are listed separately |
figure |
kit.LineChart, kit.BarChart, kit.Plot |
The name is the title, and the value of the chart is "number of items x number of series"; after switching to the data table, each row is listed with row |
tooltip |
Tip for kit.WithTooltip |
The name is the tip text, the rich content and the action keys are listed as child elements |
menu |
Open kit.Menu |
Menu items listed separately |
menuitem |
Menu item | When there is a submenu, value is submenu; supported disabled |
link |
Link in Markdown paragraph | value is the URL |
search |
Find panel for kit.CodeEditor |
Input boxes and buttons listed separately |
code |
Markdown code block | The name is the language; the code text and "Copy" button inside are listed separately |
Screen readers on macOS see the same elements; see Accessibility.
Other roles declared by the component through core.Role or el's Role will be listed as they are, and do not need to be registered in the automation code. You only need to supplement the above table. By default, an element will absorb the text inside it; if the child elements need to be listed separately (containers such as dialog boxes and menus), add the role of ui/window/automation.go to containerRoles.
In automation mode (KEEL_AUTOMATION is set), "reduced motion" is turned on by default when the application starts: overlays such as Sheet that slide in directly appear at the final position, and the coordinates read by the Agent are the clicked coordinates. The application can change it back by itself using theme.SetReducedMotion(false).
The element list contains only the visible part: table rows and page content that scroll out of view are not listed, and partially visible elements report their positions as visible parts. To see more lines, first scroll.
ref is only valid until the next operation because it is renumbered for each operation. Positioning by text (text) is more stable: complete matching takes precedence over partial matching, and controls take precedence over ordinary text; at the same time, the one drawn later is taken first, so buttons in dialog boxes and drop-down boxes take precedence over elements of the same name covered by them.
Coordinates are only used when there is no better way. Screenshots and coordinates use the same set of units: one pixel on the screenshot is one dp.
Principle
Agent ──MCP(stdio)──► keel-mcp ──JSON Line (unix socket)──► application process
│ │
launch: start the app and set ui/window automation mode:
KEEL_AUTOMATION=<socket> Window stays off-screen; render on request
attach: connect to an app you started Wait for another connection after disconnect
In the application process:
ui/windowreads theKEEL_AUTOMATIONenvironment variable when starting, and assigns a shadow window to each window: the same set of components, the same size, but has its own Gio input route. Agent operations are sent to the shadow window.- In visible mode, the real window is displayed as usual, and your mouse and keyboard follow the original path of the real window without being affected at all. Both sides share the same batch of component objects, so the Agent clicks the button through the shadow window, the callback is executed, the state changes, and the real window is immediately redrawn; the content you enter in the real window can also be seen the next time the shadow window is rendered. The size of the shadow window follows the real window. If you drag the window to change the size, the coordinates read by the Agent will also change.
- There is no real window in interfaceless mode (
KEEL_HEADLESS=1).window.Maindoes not enter the system event loop and only processes requests on the socket. - Components, callbacks, window shortcut keys,
core.Update, and frame locks all use the same code as the real window. - "What's in the page" comes from the semantic tree that Gio generates every frame: each component declares its role, name, and state, and the router calculates its absolute position in the window. The el element declares this information when drawing, and the Gio code written by yourself is declared with
core.Semantic. - Clicks, inputs, and scrolling are converted into Gio pointer and keyboard events, sent to the router of this window, and then rendered until the picture is stable: if the callback changes the state, you have to draw another frame to see it, and a maximum of 10 frames can be drawn.
The application side protocol is written in the file comments of ui/window/automation_server.go. keel-mcp does not reference any Keel package, only this protocol. If you want to write a test client in other languages, just send JSON according to the protocol.
Limitations
- The focus is on its own. After the Agent clicks into an input box, the focus is in the shadow window. The input box on the screen does not display the cursor and blue border, but the entered text will be displayed. On the other hand, if you click on the input box on the screen and the Agent calls
typewithoutref, it will report "no focus". You must bringref. - When you and the Agent operate at the same time, they will be executed in order without conflict. But when you hold down the mouse and drag, the Agent inserts and clicks, and the dragging state may be messed up.
- Closing the window will wait until it is actually closed before returning. In visible mode,
close_windowturns off the real window on the screen. It will be restored after the window is destroyed, so the window list returned is what it looks like on the screen. - Agent does not operate on the real window itself. Issues at the system window level cannot be detected: window position and size, system menu, input method candidate box, and main thread-related deadlocks between real windows. The latter is covered with a real window test of
KEEL_DESKTOP=1, see Testing. - **
native/*is not in range. ** Permissions, screenshots, and global shortcut keys adjust the system API, and the automation mode will not simulate them. - Time only advances when requested. App only renders when requested. The background goroutine's
core.Updatetakes effect on the next request; to wait for it, usewait_for. - Hover, drag, right-click, and double-click are not supported. Existing components are not used and will be added to
ui/window/automation.goif needed. - Does not report focus position.
typewithoutrefis entered into the input box that currently has focus; an error will be reported if there is no focus. - Layouts written with
core.Funcare not visible to Agent unless declared withcore.Semantic. It is not necessary to write in el, see Extension Guide.
Used in Go tests
cmd/keel-mcp/main_test.go Use the client of MCP official SDK to start keel-mcp, walk through the multiwindow example completely (TestEndToEnd), and test the connection, disconnection, reconnection and occupation prompts of attach (TestAttach), which is also a template for writing such tests. ui/window/automation_test.go Test automation modes (scrolling, tab focus, callback closing window) directly in the process. Both run with go test ./... without pop-up windows.
The floating content is listed after the main content in the order of declaration. When there is a modal overlay, the snapshot omits the main content and the overlay below it; it will be restored after closing. The bottom layer skips the entire subtree via the internal el-inert semantic tag, which is not output as a component role.