overlord

AI Overlord, managing your disparate agents through local computer use. This project allows AI to control macOS natively, providing direct system control through native macOS commands and utilities.

[!CAUTION] This comes with obvious risks. Overlord can control everything on your Mac. Please be careful.

Features

• Native macOS GUI interaction (no Docker required)
• Screen capture using native macOS commands
• Keyboard and mouse control through cliclick
• Multiple LLM provider support (Anthropic, Bedrock, Vertex)
• Streamlit-based interface
• Automatic screen resolution scaling
• File system interaction and editing capabilities

Prerequisites

• macOS Sonoma 15.7 or later
• Python 3.12+
• Homebrew (for installing additional dependencies)
• cliclick (brew install cliclick) - Required for mouse and keyboard control

Setup Instructions

1. Clone the repository and navigate to it:

git clone https://github.com/hanzoai/overlord.git
cd overlord

2. Create and activate a virtual environment:

python3.12 -m venv venv
source venv/bin/activate

3. Run the setup script:

chmod +x setup.sh
./setup.sh

4. Install Python requirements:

pip install -r requirements.txt

Running the Demo

Set up your environment and Anthropic API key

1. In a .env file add:

API_PROVIDER=anthropic
ANTHROPIC_API_KEY=<key>
WIDTH=800
HEIGHT=600
DISPLAY_NUM=1

Set the screen dimensions (recommended: stay within XGA/WXGA resolution), and put in your key from Anthropic Console.

2. Start the Streamlit app:

streamlit run streamlit.py

The interface will be available at http://localhost:8501

Screen Size Considerations

We recommend using one of these resolutions for optimal performance:

• XGA: 1024x768 (4:3)
• WXGA: 1280x800 (16:10)
• FWXGA: 1366x768 (~16:9)

Higher resolutions will be automatically scaled down to these targets to optimize model performance. You can set the resolution using environment variables:

export WIDTH=1024
export HEIGHT=768
streamlit run streamlit.py

[!IMPORTANT] The Beta API used in this reference implementation is subject to change. Please refer to the API release notes for the most up-to-date information.