Senior Data DevOps Engineer

As a Data DevOps Engineer, you will design, build, and operate the core platforms, tooling, and automation that power our data engineering ecosystem across Central Data Operations (CDOPS). Your mission is to ensure data engineers can move fast without sacrificing reliability, security, or observability. 

You will own the runtime platforms (especially Apache Airflow and Power BI/Microsoft Fabric), container images, CI/CD pipelines, and developer workflows that support data ingestion, transformation, and analytics. 

This role sits at the intersection of data engineering and DevOps, with a strong focus on production excellence, operational rigor, and developer experience (DX). 

Your work will directly enable BI, AI, and Analytics teams by providing stable, well‑instrumented, and easy‑to‑use data platforms. 

Day‑to‑Day Responsibilities 

Data Platform Architecture & Engineering 

• Build, maintain, and evolve Apache Airflow platform used by the CDOPS data team. 

• Develop and version custom Airflow Docker images, including dependency management, plugins, and secure base images. 

• Define best practices for DAG structure, retries, SLAs, backfills, and failure handling. 

• Define best practices for AIRFLOW and our Data Warehouse development process 

• Improve orchestration scalability, resilience, and upgrade strategies. 

CI/CD & Automation for Data Platforms 

• Design and operate CI/CD pipelines for:  

• Airflow DAG validation, testing, and deployment 

• Container image builds and releases 

• Configuration promotion across environments 

• Enforce Git‑based workflows, automated checks, and policy‑as‑code for data platforms. 

• Reduce deployment friction while improving safety, traceability, and rollback capabilities. 

Platform Reliability, Operations & DevEx 

• Architect and run containerized data systems using Docker and Docker Compose. 

• Debug and operate Linux‑based systems (CPU, memory, I/O, networking, DNS). 

• Improve developer experience through:  

• Local development tooling 

• Platform templates and automation 

• Clear documentation and onboarding paths 

• Eliminate toil and manual operational work through automation. 

Monitoring, Observability & Incident Management 

• Design and operate platform‑level observability using Prometheus and Grafana. 

• Define and maintain metrics, dashboards, and alerting standards for data platforms. 

• Lead incident response, root‑cause analysis, and post‑incident improvements. 

• Maintain runbooks, system diagrams, and operational documentation. 

Collaboration & Cross‑Functional Support 

• Partner with Data Engineering, BI, and AI teams to design robust data flows and operational solutions. 

• Translate developers’ needs into scalable platform solutions. 

• Provide technical leadership through code reviews, design reviews, and architectural discussions. 

• Mentor engineers in production‑grade data systems, DevOps practices, and operational excellence. 

Preferred Skills & Knowledge 

• Python skills for automation and platform tooling. 

• Expert knowledge of Docker (multi‑stage builds, debugging, Compose) and Linux systems. 

• Hands‑on experience with Apache Airflow architecture and operations. 

• Proven experience building and maintaining CI/CD pipelines. 

• Expertise in monitoring and observability tooling (Prometheus, Grafana). 

• Strong Git‑based development practices and familiarity with Infrastructure‑as‑Code concepts. 

• Exposure to Snowflake, dbt, Microsoft Fabric, Terraform, AIRBYTE, Apache Hop, Hashicorp Vault is a plus. 

• Clear communicator with strong system‑level thinking. 

Preferred Experience 

• Running Airflow in production at scale (Celery or Kubernetes executors). 

• Designing monitoring architectures with Prometheus and grafana 

• Experience in SaaS environments. 

• Managing multi‑container applications, secure secrets handling, and production troubleshooting. 

• Willingness and motivation to progressively take on Data Engineering responsibilities as part of career growth. 

Basic Qualifications 

• Bachelor’s degree in Computer Science, Engineering, Information Systems, or equivalent experience. 

• 3+ years hands‑on experience in Data Engineering, Platform Engineering or DevOps roles. 

• English fluency 

• Proven experience operating production‑grade data platforms. 

• Strong understanding of cloud environments, Linux systems, and data pipeline orchestration. 

• Demonstrated success in supporting cross‑functional data teams.