Senior Data & Software Engineer

Job Description:

The Senior Data & Software Engineer works with a small team to build complex data flows for a custom application. Successful candidate will have advanced Python programming skills, familiarity with Java, an understanding of data security, privacy, governance and compliance principles, and a demonstrated history of building production data pipelines and ETL workflows at scale.

Work you'll do:

• Work with stakeholders to understand data requirements, assess feasibility, and design appropriate solutions with minimal oversight.
• Leverage strong problem-solving and debugging skills for data quality issues, pipeline failures, and performance bottlenecks.
• Leverage a background in large-scale data migration or platform modernization efforts.
• Contribute to data engineering documentation, best practices, and design patterns.

What you'll need:

• Minimum of 5 years' experience with the following:
  • Apache Spark & PySpark
  • Using orchestration tools to deploy data pipelines, including configuring and updating Spark Jobs
  • Advanced Python skills (including Pandas & NumPy)
  • Building end-to-end data pipelines leveraging Python
  • Docker, Podman
  • AWS S3, Lambda & Step functions
  • Apache Iceberg, Airflow, etc.
  • SQL (with Trino)
  • NoSQL, DynamoDB
  • Containerizing and deploying applications in cloud environments like AWS
  • Working with MySQL and PostgreSQL including performance tuning, schema design, and query optimization for complex, analytical workloads
  • Leveraging industry standard tools for code control (Git, IaaC control, etc.)
  • Working with data catalogs, tracking data lineage, and handling a variety of data formats, including Geospatial
  • Using Bash scripting for automation and data processing tasks
  • Integrating Al/ML services and models

Bonus points if you have:

• Unity Catalog OSS, Apache Polaris
• Apache Superset
• Terraform or CloudFormation
• OpenLineage
• H3, PostGIS

Security clearance:

• Active TS/SCI with Poly security clearance is required