Origin
Some of this works today, some is still being built. Sharing for feedback and collaboration.
OPERATOR evolved from daily AI development work:
- Started with Cursor IDE - powerful but limited to single-context work
- Expanded to multiple agents and terminals running in parallel
- Needed a way to orchestrate, remember, and communicate across all of them
- OPERATOR emerged as the solution - still evolving daily
This is a living system. I'm sharing to see how others approach agent orchestration.
Core Capabilities
What It Does
- Multi-agent orchestration with unified voice control: One voice interface commanding multiple AI agents across projects. Speak and it coordinates.
- Context compaction detection and rule re-injection: When long-running agents "forget" rules as context grows, Operator detects the compaction and restores the full instruction set.
- Automated agent confirmation handling (90% target): Most agent pop-ups follow patterns. Operator handles them automatically - only novel decisions reach you.
- Persistent cross-session memory: Context survives sessions, not just conversations.
See Architecture section for implementation details.
How It Works
The dashboard is VIEW-ONLY. Operator controls what's displayed. You tell Operator what you want via voice, and it executes - spawning agents, switching projects, displaying files.
Workflow Evolution
The progression that led to OPERATOR:
- Phase 1: Manual coding with AI autocomplete
- Phase 2: Voice input to single agent - faster but limited context
- Phase 3: Multiple agents in parallel with git worktrees - powerful but constant context-switching
- Phase 4: OPERATOR - one voice interface orchestrating all agents with unified memory
Architecture
OPERATOR ARCHITECTURE
+-------------+
Voice In | YOU | Voice Out
+------------>| (F9 Key) |<------------+
| +-------------+ |
| |
+------+------+ +-------+-------+
| STT | | TTS |
| Whisper | | Piper+Effects |
+------+------+ +-------+-------+
| ^
| Transcription |
v |
+-------------+-------------------------------------+ |
| OPERATOR CORE | |
| | |
| +-------------------------------------------+ | |
| | TRIAGE (Haiku) | | |
| | SIMPLE -> Execute | COMPLEX -> Opus | | |
| +-------------------------------------------+ | |
| +---+
| +-------------------------------------------+ |
| | CLAUDE-BOOTSTRAP FRAMEWORK | |
| | 40+ Skills | CLAUDE.md Rules | |
| | TDD-first | Self-verification | |
| +-------------------------------------------+ |
| |
| - Agent orchestration - Rule injection |
| - Status aggregation - Context refresh |
| |
+-------+------------------+------------------+-----+
| | |
v v v
+-------+------+ +------+-------+ +------+-------+
| RAG DATABASE | | AGENTS | | DASHBOARD |
| | | (Claude Code)| | xterm.js |
| Conversations| | | | Socket.IO |
| History | | [R] Agent 1 | +--------------+
| Memory | | [I] Agent 2 |
| Preferences | | [Q] Agent 3 |
+--------------+ +--------------+
Two-Layer Intelligence
- Fast Triage (Haiku): Quick commands get instant response. "Show me spec.md" - no deep reasoning needed.
- Deep Thinking (Opus): Complex tasks spawn Opus workers for dedicated reasoning.
Persistent Memory (In Development)
RAG database storing conversations, decisions, preferences, and project context. Query past sessions naturally.
Automated Agent Management
Target: handle 90% of agent confirmations automatically. Only novel decisions reach the user.
Context Persistence & Rule Refresh
Problem: Long-running agents "forget" CLAUDE.md rules as context grows. Solution: detect context compaction and re-inject the full ruleset automatically. May become unnecessary as models improve context handling.
Features
Some implemented, others in development.
Voice Control
Hotkey-triggered voice input for all commands.
Mainframe Display
Voice-controlled view switching - files, terminals, diffs.
Status Updates
Spoken notifications: build status, task completion, blockers.
Context Persistence
Cross-session memory via RAG database.
Silent Handling
Automatic handling of routine agent confirmations.
Full Transparency
All automated decisions logged and inspectable.
Example Interactions
"Start implementing the payment system we discussed"
"Starting payment implementation. Spawning an agent on the payments branch.
I'll handle the standard confirmations. I'll let you know when it's ready
for review, or if anything needs your input."
"Payment system implementation complete. 12 files changed, all tests passing.
Ready for your review on the payments branch."
"I'm back. What's the status?"
"Welcome back. While you were away:
- Payment branch: Implementation complete, awaiting review
- Auth refactor: Agent hit a blocker on token refresh. I tried two
approaches but both failed the tests. Need your input on this one.
- Main branch: CI passed, no new issues
Want me to show you what the auth agent tried?"
Design Principles
- Single interface: One voice conversation replaces terminal juggling
- Human review: Operator handles routine; you make important decisions
- Graduated autonomy: Starts cautious, increases as it learns your patterns
- Full logging: Every automated action is inspectable
If Anthropic or OpenAI ships native orchestration that does this, great. This is exploration, not product defense.
The Interface
+------------------------------------------------------------------+
| [ OPERATOR ] |
| // AGENT CONTROL SYSTEM |
+------------------------------------------------------------------+
| | |
| PROJECTS | MAINFRAME |
| +--------------+ | +------------------------------------+ |
| | Project A | | | | |
| | Project B | | | (Operator controls this view) | |
| | Project C | | | | |
| +--------------+ | | - Code display | |
| | | - Agent terminals | |
| AGENTS | | - Spec documents | |
| +--------------+ | | - Diff views | |
| | [R] Agent 1 | | | - Whatever you ask to see | |
| | [Q] Agent 2 | | | | |
| | [I] Agent 3 | | +------------------------------------+ |
| +--------------+ | |
| | +------------------------------------+ |
| OPERATOR WORKERS | | SPEECH LOG | |
| +--------------+ | | > "Show me the spec" | |
| | [*] Think-1 | | | < "Displaying spec.md" | |
| | [*] Think-2 | | | > "Start payment feature" | |
| +--------------+ | | < "Spawning agent..." | |
| (blinking) | +------------------------------------+ |
+------------------------------------------------------------------+
Visual Design
- Matrix/CRT Aesthetic: Dark background, phosphor green text, subtle scanlines, gentle glow effects
- Sidebar: Projects and agents at top, Operator's thinking workers (yellow blinking) at bottom
- Mainframe: Central display controlled entirely by Operator
- Speech Log: Bottom-right, shows the voice conversation, expandable
Note
The Matrix theme is cosmetic. Core system is CLI/voice-first. Dashboard is visualization only. Themes and voices are configurable.
Why I'm Sharing This
Looking to connect with others building similar systems. Interested in approaches to agent orchestration, persistent memory, and automated decision-making.
Considering open sourcing if there's interest. Looking for feedback on the approach.
About / Contact
The Person
44 years old
20+ years of professional IT experience
Led teams of up to 18 people
AI journey started 2018 with Andrew Ng's machine learning course
Early adopter mentality - always pushing into new territory
Current Situation
Currently unemployed - laid off end of 2024 from the tech industry. Open to AI-focused opportunities - building systems like OPERATOR rather than traditional web products.
I have a SaaS role offer on the table, but AI is where my passion lies.
100% remote only due to a medical condition.
Email: info@operator-ai.info