DevJobs

Senior Data Engineer

Overview
Skills
  • Python Python
  • SQL SQL
  • Kafka Kafka
  • Snowflake Snowflake
  • Airflow Airflow
  • Dagster
  • Databricks
  • Debezium
  • Delta
  • Iceberg
  • S3
  • BigQuery
  • GCS
  • Hudi

You'll own our data lake - the system of record for millions of financial events every day, including bets, wallet movements, and live odds, across 12M+ active users.

You'll be responsible for how data is ingested, stored, retained, and governed across our data infrastructure, built on S3 and Snowflake/Databricks. Your work will ensure that analytics, finance, and regulatory teams have access to accurate, reliable, and fully reconciled data.


What You'll Do:

  • Lakehouse Architecture: Own the lakehouse architecture, including bronze/silver/gold layers, Iceberg/Delta tables, and schema evolution.
  • Data Ingestion: Land operational data via CDC streaming (Kafka, Debezium), handling late and duplicate events.
  • Performance & Cost Optimization: Design data layouts for speed and cost efficiency, including partitioning, compaction, file sizing, and query performance on Trino/Athena/Snowflake.
  • Data Retention & Archival: Own storage tiering, regulatory retention, immutability, and GDPR deletion.
  • Data Quality & Reliability: Ensure data correctness through freshness SLAs, drift detection, and reconciliation against source wallet and ledger systems.
  • Data Governance: Own catalog and lineage, row/column access control, PII masking, encryption, and audit trails.
  • Monitoring: Monitor ingestion health, data anomalies, and cloud storage/compute costs.


Requirements:

  • 5+ years of experience in Data Engineering, with proven ownership of large-scale data lake or lakehouse architectures.
  • Production experience with cloud object storage (S3/GCS) and Snowflake, Databricks, or BigQuery.
  • Hands-on experience with an open table format (Iceberg, Delta, or Hudi).
  • Deep understanding of partitioning, file layout, and query optimization at terabyte-plus scale.
  • Experience with CDC and streaming ingestion (Kafka + Debezium or equivalent).
  • Strong SQL and data modeling skills, with a solid understanding of relational databases and OLTP systems, particularly in the context of financial ledgers.
  • Experience with data lake governance, including catalogs, lineage, access control, PII, and retention.
  • Python and an orchestration tool (Airflow/Dagster) as working tools. This is an architecture and ownership role, not a heads-down coding position.
  • Proven hands-on experience designing, building, and deploying AI agents and LLM-powered workflows in production. This goes beyond using existing AI tools or APIs and requires actual development and implementation experience.


Nice to have:

  • Experience in fintech, iGaming, or another regulated, audit-heavy environment.


AlfaBet