Principal BackEnd & Data Engineer — Deep Tech / GenAI Platform (Python)

About SparkBeyond

SparkBeyond is building the future of AI in the enterprise. Our AI technology allows a company to optimize operational performance using GenAI, 24x7. Our customers are already using AI to predict and explain product demand, decide which loans should be given, check transactions for fraud, give maintenance instructions to engineers and much more. Operating at the intersection of Generative AI, Data Science, and Machine Learning, our platform pushes the boundaries of what machines and LLM’s can infer from large volumes of structured data and reason what to do based on that knowledge - to the point it’s been described as “science fiction made real.”

We serve large enterprises looking to jump to the future where GenAI deeply understands what drives the performance of a business and so can help decision-making and innovation through data-driven discovery.

The Role

We’re looking for a Principal Backend & Data Engineer to own and evolve the core data-intensive backend architecture of our AI platform. This is a senior individual contributor role with broad architectural responsibility and deep hands-on impact.

You will be a technical authority in building scalable, high-performance backend and data systems that power our AI research engine - working closely with AI scientists, platform engineers, and product leaders. Beyond implementation, you will define technical direction, set engineering standards, and influence how we build and scale data-driven AI systems across the organization.

What You’ll Do

Architecture & Technical Ownership

• Own the architecture of core backend and data systems that support large-scale, AI-driven computation and inference.
• Design and evolve high-performance analytical processing layers handling complex data workloads at scale.
• Make and drive long-term architectural decisions across backend, data, and AI integration domains.
• Define standards and best practices for performance, scalability, reliability, and data correctness.

Backend & Data Engineering

• Design, implement, and optimize data-intensive backend services using Python.
• Build and evolve analytical compute pipelines using technologies such as Polars, DuckDB, AirFlow, Iceberg, Spark and other ingestion and processing frameworks.
• Collaborate closely with AI scientists to productionize algorithms and embed advanced intelligence into the platform.

Platform & Infrastructure

• Architect and maintain containerized systems using Docker and Kubernetes.
• Ensure systems are observable, debuggable, and operable in production environments.
• Drive performance tuning and cost-efficiency across compute-heavy workloads.

Technical Leadership

• Act as a technical authority and thought leader within the engineering organization.
• Mentor senior engineers through design reviews, architectural guidance, and hands-on collaboration.
• Serve as a force multiplier by unblocking complex technical challenges and raising the overall engineering bar.

What We’re Looking For

• A passion for building cutting-edge AI and data platforms that solve real, high-impact business problems.
• Significant experience (8+ years) designing, building, and operating large-scale backend and data-intensive systems in production.
• Deep Hands-on expertise in Python and strong understanding of analytical data processing and distributed systems.
• Proven track record of owning and evolving complex system architectures over time.
• Strong experience with containerization and orchestration technologies (Docker, Kubernetes).
• Experience with Infrastructure-as-Code and modern delivery pipelines (e.g., Terraform, ArgoCD).
• Demonstrated technical leadership through architectural ownership, mentorship, and influence across teams.
• Comfort working at the intersection of backend engineering, data engineering, and AI systems.

Why Join SparkBeyond

• Work on deeply technical problems at the frontier of Generative AI and data-driven reasoning.
• Own and shape core platform architecture with real influence and autonomy.
• Collaborate with top-tier AI researchers and engineers in a fast-moving deep-tech environment.
• Build systems that operate at enterprise scale and drive meaningful business impact.