Backend Engineer for AI Agent Development

About the Role

We’re hiring a Backend Engineer to work on HolmesGPT, a widely popular CNCF open source project for AI-driven cloud troubleshooting. You’ll build the infrastructure that connects AI agents to external tools, optimize agent performance, and create integrations that help engineers investigate and solve cloud incidents faster.

This is a hands-on technical role pushing the boundaries of what AI can do autonomously. You’ll architect systems that give AI agents the ability to reason about complex cloud infrastructure, execute actions safely, and solve real production incidents.

What You’ll Do

• Build and maintain MCP (Model Context Protocol) servers to connect AI agents with external tools and data sources
• Integrate OpenAI and Anthropic (Claude) Completion APIs into production agent workflows
• Optimize AI agent accuracy, latency, and reliability through context engineering, subagents, and more
• Design and implement observability for AI agent systems (tracing requests, monitoring token usage, debugging agent behavior)
• Build integrations with cloud platforms (AWS, GCP, Azure) and monitoring tools
• Work closely with CEO, CTO, product, and customers to deliver end-to-end solutions
• Push the limits of autonomous AI capabilities - making agents smarter, faster, and more reliable

Requirements - Must Have:

• 8+ years backend development experience with strong Python skills (our codebase is primarily Python)
• Hands-on experience building AI agents using OpenAI or Anthropic APIs
• Strong debugging skills and systems thinking
• Experience with large and complex systems and a strong system-level view
• Comfortable working in fast-moving startup environment with high autonomy
• Strong English communication skills

Requirements - Nice to Have:

• Built MCP servers or similar tool integration systems
• Kubernetes experience - deploying, debugging, and managing production workloads
• Observability/monitoring tools - Prometheus, Grafana, Datadog, New Relic, Loki, or similar
• Experience building developer tools or CLI applications
• Open source contributions (especially CNCF projects)
• Built internal tools that other engineers use
• DevOps/CI/CD experience - automating releases and improving deployment workflows
• Deep care for developer experience and code quality