DevJobs

Data Engineer

Overview
Skills
  • Python Python
  • Pandas Pandas
  • Oracle Oracle
  • PostgreSQL PostgreSQL
  • GitHub Actions GitHub Actions
  • Docker Docker
  • Kubernetes Kubernetes
  • Airflow Airflow
  • SQL Server
  • Jupyter
  • Parquet
  • PySpark
  • S3-compatible storage

About the Job

We’re looking for a Data Engineer to build and operate ETL pipelines using Python, Apache Airflow, and Kubernetes. You’ll work across the full data pipeline, from ingesting data from source systems to transforming it into curated datasets for analytics. You’ll also monitor pipeline reliability and data freshness in production.


Responsibilities

  • Develop Python-based ETL pipelines across source, staging (silver), and curated (gold) layers, including configurable ingestion, data standardization, aggregation, and lineage tracking.
  • Design and operate Airflow DAGs for scheduled workflows, incremental loads, and backfills.
  • Run and scale containerized data workloads using Docker and Kubernetes, configure resources, and troubleshoot pod and workflow failures.
  • Optimize queries and indexes for large analytical datasets in PostgreSQL, SQL Server, and Oracle.
  • Monitor pipeline performance and data freshness in production, automate testing and deployment through CI/CD, and validate releases.


Requirements

  • Advanced Python skills, including object-oriented development, packaging, type hints, automated testing, logging, and efficient data transformations.
  • Hands-on experience with Apache Airflow, including DAG design, scheduling, task dependencies, retries, backfills, pools and queues, incremental loads, and operational troubleshooting.
  • Experience with Kubernetes and Docker, including containerized workloads, resource requests and limits, pod diagnostics, scaling, and environment configuration.
  • Experience designing ETL pipelines across source, staging (silver), and curated (gold) layers, including parameterized ingestion, data standardization, aggregation, and lineage.
  • Strong database engineering skills across PostgreSQL, SQL Server, and Oracle, including query tuning, indexing, and working with large analytical datasets.
  • Experience with production observability, including monitoring DAG and task duration, failures, queue wait times, pod health, resource usage, data volumes, throughput, data freshness and latency, and business KPIs.
  • Experience with CI/CD using GitHub Actions or an equivalent tool, including writing Dockerfiles, running automated tests, validating deployments, and planning for safe rollbacks.


Nice to Have

  • Experience with Parquet, object storage patterns, S3-compatible storage, pandas or PySpark, and data validation in Jupyter.
  • Familiarity with medallion architecture, metadata management, data catalogs, tenant isolation, and role-based data access.
  • Experience migrating data workflows from Windows Task Scheduler or VM-based environments to services running on Airflow and Kubernetes.


Sela