桌面控制：为 Openclaw Skills 提供的先进 UI 自动化

下载入口:https://github.com/openclaw/skills/tree/main/skills/matagul/desktop-control

安装与下载

1. ClawHub CLI

从源直接安装技能的最快方式。

npx clawhub@latest install desktop-control

2. 手动安装

将技能文件夹复制到以下位置之一

全局模式 ~/.openclaw/skills/ 工作区 <project>/skills/

优先级：工作区 > 本地 > 内置

3. 提示词安装

将此提示词复制到 OpenClaw 即可自动安装。

请帮我使用 Clawhub 安装 desktop-control。如果尚未安装 Clawhub，请先安装（npm i -g clawhub）。

什么是 桌面控制技能？

桌面控制技能代表了 Openclaw Skills 生态系统中可用的最先进的自动化工具包。它允许 AI 代理以类人的灵活性和机器般的精度与桌面环境进行交互。通过缩小高层逻辑与底层系统交互之间的差距，该技能使开发者能够自动化任何应用程序，无论其是否提供原生 API。

它基于稳健的 Python 库构建，处理从平滑的贝塞尔曲线鼠标移动到复杂的组合键热键等一切事务。无论您是在自动化过时的企业应用程序，还是构建复杂的 UI 测试机器人，此技能都提供了与屏幕交互、管理窗口以及实时处理视觉数据所需的基础原语。

桌面控制技能 应用场景

• 在非 Web 应用程序中自动执行重复的数据录入和表单填写。
• 为桌面软件套件创建自动化的操作演示和 UI 测试。
• 通过程序化方式管理复杂的多窗口工作流和显示器配置。
• 构建基于屏幕颜色变化或图像出现来触发动作的视觉监控器。
• 在文件资源管理器中执行涉及拖放交互的批量文件操作。

桌面控制技能 工作原理

1. Openclaw Skills 环境初始化 DesktopController，设置安全边界和故障安全触发器。
2. 该技能通过绝对像素坐标或使用 OpenCV 的视觉模式匹配来识别目标元素。
3. 指令被发送到系统输入缓冲区，以模拟鼠标点击、滚动或按键。
4. 通过高分辨率截图捕获屏幕状态，以验证交互是否成功。
5. 窗口管理功能确保目标应用程序在整个生命周期中处于焦点并正确定位。

桌面控制技能 配置指南

要开始使用此技能，请通过 pip 安装必要的系统依赖项：

pip install pyautogui pillow opencv-python pygetwindow

安装完成后，您可以在自动化脚本中初始化控制器：

from skills.desktop_control import DesktopController
dc = DesktopController(failsafe=True)

桌面控制技能 数据架构与分类体系

桌面控制技能使用以下结构组织交互数据：

数据类型	描述	格式
坐标	来自左上角的 X 和 Y 像素位置	(int, int)
RGB 颜色	红、绿、蓝颜色值	(r, g, b)
屏幕区域	用于局部屏幕截图的定义框	(左, 上, 宽, 高)
窗口元数据	应用程序窗口的标题、位置和状态	字符串/对象
键盘映射	特殊键和修饰键的标准名称	字符串 (例如 'enter', 'ctrl')

---

description: Advanced desktop automation with mouse, keyboard, and screen control

Desktop Control Skill

The most advanced desktop automation skill for OpenClaw. Provides pixel-perfect mouse control, lightning-fast keyboard input, screen capture, window management, and clipboard operations.

?? Features

Mouse Control

• ? Absolute positioning - Move to exact coordinates
• ? Relative movement - Move from current position
• ? Smooth movement - Natural, human-like mouse paths
• ? Click types - Left, right, middle, double, triple clicks
• ? Drag & drop - Drag from point A to point B
• ? Scroll - Vertical and horizontal scrolling
• ? Position tracking - Get current mouse coordinates

Keyboard Control

• ? Text typing - Fast, accurate text input
• ? Hotkeys - Execute keyboard shortcuts (Ctrl+C, Win+R, etc.)
• ? Special keys - Enter, Tab, Escape, Arrow keys, F-keys
• ? Key combinations - Multi-key press combinations
• ? Hold & release - Manual key state control
• ? Typing speed - Configurable WPM (instant to human-like)

Screen Operations

• ? Screenshot - Capture entire screen or regions
• ? Image recognition - Find elements on screen (via OpenCV)
• ? Color detection - Get pixel colors at coordinates
• ? Multi-monitor - Support for multiple displays

Window Management

• ? Window list - Get all open windows
• ? Activate window - Bring window to front
• ? Window info - Get position, size, title
• ? Minimize/Maximize - Control window states

Safety Features

• ? Failsafe - Move mouse to corner to abort
• ? Pause control - Emergency stop mechanism
• ? Approval mode - Require confirmation for actions
• ? Bounds checking - Prevent out-of-screen operations
• ? Logging - Track all automation actions

---

?? Quick Start

Installation

First, install required dependencies:

pip install pyautogui pillow opencv-python pygetwindow

Basic Usage

from skills.desktop_control import DesktopController

# Initialize controller
dc = DesktopController(failsafe=True)

# Mouse operations
dc.move_mouse(500, 300)  # Move to coordinates
dc.click()  # Left click at current position
dc.click(100, 200, button="right")  # Right click at position

# Keyboard operations
dc.type_text("Hello from OpenClaw!")
dc.hotkey("ctrl", "c")  # Copy
dc.press("enter")

# Screen operations
screenshot = dc.screenshot()
position = dc.get_mouse_position()

---

?? Complete API Reference

Mouse Functions

move_mouse(x, y, duration=0, smooth=True)

Move mouse to absolute screen coordinates.

Parameters:

• x (int): X coordinate (pixels from left)
• y (int): Y coordinate (pixels from top)
• duration (float): Movement time in seconds (0 = instant, 0.5 = smooth)
• smooth (bool): Use bezier curve for natural movement

Example:

# Instant movement
dc.move_mouse(1000, 500)

# Smooth 1-second movement
dc.move_mouse(1000, 500, duration=1.0)

move_relative(x_offset, y_offset, duration=0)

Move mouse relative to current position.

Parameters:

• x_offset (int): Pixels to move horizontally (positive = right)
• y_offset (int): Pixels to move vertically (positive = down)
• duration (float): Movement time in seconds

Example:

# Move 100px right, 50px down
dc.move_relative(100, 50, duration=0.3)

click(x=None, y=None, button='left', clicks=1, interval=0.1)

Perform mouse click.

Parameters:

• x, y (int, optional): Coordinates to click (None = current position)
• button (str): 'left', 'right', 'middle'
• clicks (int): Number of clicks (1 = single, 2 = double)
• interval (float): Delay between multiple clicks

Example:

# Simple left click
dc.click()

# Double-click at specific position
dc.click(500, 300, clicks=2)

# Right-click
dc.click(button='right')

drag(start_x, start_y, end_x, end_y, duration=0.5, button='left')

Drag and drop operation.

Parameters:

• start_x, start_y (int): Starting coordinates
• end_x, end_y (int): Ending coordinates
• duration (float): Drag duration
• button (str): Mouse button to use

Example:

# Drag file from desktop to folder
dc.drag(100, 100, 500, 500, duration=1.0)

scroll(clicks, direction='vertical', x=None, y=None)

Scroll mouse wheel.

Parameters:

• clicks (int): Scroll amount (positive = up/left, negative = down/right)
• direction (str): 'vertical' or 'horizontal'
• x, y (int, optional): Position to scroll at

Example:

# Scroll down 5 clicks
dc.scroll(-5)

# Scroll up 10 clicks
dc.scroll(10)

# Horizontal scroll
dc.scroll(5, direction='horizontal')

get_mouse_position()

Get current mouse coordinates.

Returns: (x, y) tuple

Example:

x, y = dc.get_mouse_position()
print(f"Mouse is at: {x}, {y}")

---

Keyboard Functions

type_text(text, interval=0, wpm=None)

Type text with configurable speed.

Parameters:

• text (str): Text to type
• interval (float): Delay between keystrokes (0 = instant)
• wpm (int, optional): Words per minute (overrides interval)

Example:

# Instant typing
dc.type_text("Hello World")

# Human-like typing at 60 WPM
dc.type_text("Hello World", wpm=60)

# Slow typing with 0.1s between keys
dc.type_text("Hello World", interval=0.1)

press(key, presses=1, interval=0.1)

Press and release a key.

Parameters:

• key (str): Key name (see Key Names section)
• presses (int): Number of times to press
• interval (float): Delay between presses

Example:

# Press Enter
dc.press('enter')

# Press Space 3 times
dc.press('space', presses=3)

# Press Down arrow
dc.press('down')

hotkey(*keys, interval=0.05)

Execute keyboard shortcut.

Parameters:

• *keys (str): Keys to press together
• interval (float): Delay between key presses

Example:

# Copy (Ctrl+C)
dc.hotkey('ctrl', 'c')

# Paste (Ctrl+V)
dc.hotkey('ctrl', 'v')

# Open Run dialog (Win+R)
dc.hotkey('win', 'r')

# Save (Ctrl+S)
dc.hotkey('ctrl', 's')

# Select All (Ctrl+A)
dc.hotkey('ctrl', 'a')

key_down(key) / key_up(key)

Manually control key state.

Example:

# Hold Shift
dc.key_down('shift')
dc.type_text("hello")  # Types "HELLO"
dc.key_up('shift')

# Hold Ctrl and click (for multi-select)
dc.key_down('ctrl')
dc.click(100, 100)
dc.click(200, 100)
dc.key_up('ctrl')

---

Screen Functions

screenshot(region=None, filename=None)

Capture screen or region.

Parameters:

• region (tuple, optional): (left, top, width, height) for partial capture
• filename (str, optional): Path to save image

Returns: PIL Image object

Example:

# Full screen
img = dc.screenshot()

# Save to file
dc.screenshot(filename="screenshot.png")

# Capture specific region
img = dc.screenshot(region=(100, 100, 500, 300))

get_pixel_color(x, y)

Get color of pixel at coordinates.

Returns: RGB tuple (r, g, b)

Example:

r, g, b = dc.get_pixel_color(500, 300)
print(f"Color at (500, 300): RGB({r}, {g}, {b})")

find_on_screen(image_path, confidence=0.8)

Find image on screen (requires OpenCV).

Parameters:

• image_path (str): Path to template image
• confidence (float): Match threshold (0-1)

Returns: (x, y, width, height) or None

Example:

# Find button on screen
location = dc.find_on_screen("button.png")
if location:
    x, y, w, h = location
    # Click center of found image
    dc.click(x + w//2, y + h//2)

get_screen_size()

Get screen resolution.

Returns: (width, height) tuple

Example:

width, height = dc.get_screen_size()
print(f"Screen: {width}x{height}")

---

Window Functions

get_all_windows()

List all open windows.

Returns: List of window titles

Example:

windows = dc.get_all_windows()
for title in windows:
    print(f"Window: {title}")

activate_window(title_substring)

Bring window to front by title.

Parameters:

• title_substring (str): Part of window title to match

Example:

# Activate Chrome
dc.activate_window("Chrome")

# Activate VS Code
dc.activate_window("Visual Studio Code")

get_active_window()

Get currently focused window.

Returns: Window title (str)

Example:

active = dc.get_active_window()
print(f"Active window: {active}")

---

Clipboard Functions

copy_to_clipboard(text)

Copy text to clipboard.

Example:

dc.copy_to_clipboard("Hello from OpenClaw!")

get_from_clipboard()

Get text from clipboard.

Returns: str

Example:

text = dc.get_from_clipboard()
print(f"Clipboard: {text}")

---

?? Key Names Reference

Alphabet Keys

'a' through 'z'

Number Keys

'0' through '9'

Function Keys

'f1' through 'f24'

Special Keys

• 'enter' / 'return'
• 'esc' / 'escape'
• 'space' / 'spacebar'
• 'tab'
• 'backspace'
• 'delete' / 'del'
• 'insert'
• 'home'
• 'end'
• 'pageup' / 'pgup'
• 'pagedown' / 'pgdn'

Arrow Keys

• 'up' / 'down' / 'left' / 'right'

Modifier Keys

• 'ctrl' / 'control'
• 'shift'
• 'alt'
• 'win' / 'winleft' / 'winright'
• 'cmd' / 'command' (Mac)

Lock Keys

• 'capslock'
• 'numlock'
• 'scrolllock'

Punctuation

• '.' / ',' / '?' / '!' / ';' / ':'
• '[' / ']' / '{' / '}'
• '(' / ')'
• '+' / '-' / '*' / '/' / '='

---

??? Safety Features

Failsafe Mode

Move mouse to any corner of the screen to abort all automation.

# Enable failsafe (enabled by default)
dc = DesktopController(failsafe=True)

Pause Control

# Pause all automation for 2 seconds
dc.pause(2.0)

# Check if automation is safe to proceed
if dc.is_safe():
    dc.click(500, 500)

Approval Mode

Require user confirmation before actions:

dc = DesktopController(require_approval=True)

# This will ask for confirmation
dc.click(500, 500)  # Prompt: "Allow click at (500, 500)? [y/n]"

---

?? Advanced Examples

Example 1: Automated Form Filling

dc = DesktopController()

# Click name field
dc.click(300, 200)
dc.type_text("John Doe", wpm=80)

# Tab to next field
dc.press('tab')
dc.type_text("[email protected]", wpm=80)

# Tab to password
dc.press('tab')
dc.type_text("SecurePassword123", wpm=60)

# Submit form
dc.press('enter')

Example 2: Screenshot Region and Save

# Capture specific area
region = (100, 100, 800, 600)  # left, top, width, height
img = dc.screenshot(region=region)

# Save with timestamp
import datetime
timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
img.save(f"capture_{timestamp}.png")

Example 3: Multi-File Selection

# Hold Ctrl and click multiple files
dc.key_down('ctrl')
dc.click(100, 200)  # First file
dc.click(100, 250)  # Second file
dc.click(100, 300)  # Third file
dc.key_up('ctrl')

# Copy selected files
dc.hotkey('ctrl', 'c')

Example 4: Window Automation

# Activate Calculator
dc.activate_window("Calculator")
time.sleep(0.5)

# Type calculation
dc.type_text("5+3=", interval=0.2)
time.sleep(0.5)

# Take screenshot of result
dc.screenshot(filename="calculation_result.png")

Example 5: Drag & Drop File

# Drag file from source to destination
dc.drag(
    start_x=200, start_y=300,  # File location
    end_x=800, end_y=500,       # Folder location
    duration=1.0                 # Smooth 1-second drag
)

---

? Performance Tips

1. Use instant movements for speed: duration=0
2. Batch operations instead of individual calls
3. Cache screen positions instead of recalculating
4. Disable failsafe for maximum performance (use with caution)
5. Use hotkeys instead of menu navigation

---

?? Important Notes

• Screen coordinates start at (0, 0) in top-left corner
• Multi-monitor setups may have negative coordinates for secondary displays
• Windows DPI scaling may affect coordinate accuracy
• Failsafe corners are: (0,0), (width-1, 0), (0, height-1), (width-1, height-1)
• Some applications may block simulated input (games, secure apps)

---

?? Troubleshooting

Mouse not moving to correct position

• Check DPI scaling settings
• Verify screen resolution matches expectations
• Use get_screen_size() to confirm dimensions

Keyboard input not working

• Ensure target application has focus
• Some apps require admin privileges
• Try increasing interval for reliability

Failsafe triggering accidentally

• Increase screen border tolerance
• Move mouse away from corners during normal use
• Disable if needed: DesktopController(failsafe=False)

Permission errors

• Run Python with administrator privileges for some operations
• Some secure applications block automation

---

?? Dependencies

• PyAutoGUI - Core automation engine
• Pillow - Image processing
• OpenCV (optional) - Image recognition
• PyGetWindow - Window management

Install all:

pip install pyautogui pillow opencv-python pygetwindow

---

Built for OpenClaw - The ultimate desktop automation companion ??