About the role

Aplyr's Quick Take

This role is for a Lead Data Platform Engineer focused on building and optimizing data pipelines for large-scale IoT data using AWS, Kafka, and OpenSearch. You'll be responsible for designing high-availability systems, performance tuning, and leading deployment activities. It's primarily an individual contributor role with some leadership responsibilities.

Good fit

Ideal candidates have 8-12 years of experience in data engineering or backend development, particularly in AWS environments. Strong communication skills and a collaborative working style are essential, especially for global teamwork.

Worth noting

The role emphasizes performance tuning and optimization, which may appeal to those who enjoy problem-solving at scale. The requirement for OpenSearch experience is quite specific and could limit the candidate pool.

Position:

Lead Data Platform Engineer (OpenSearch)

Job Description:

Role: Lead Data Platform Engineer (AWS | Kafka | OpenSearch)

Preferred Location: Ahmedabad, Pune, Indore

Years of Experience: 08 – 12 years

Notice Period – 0-30 Days (Preferred)

Introduction
Our team builds a data platform that reliably collects, processes, and retrieves large-scale IoT data based on AWS. We are seeking senior engineer who can design unobstructed flow even among billions of data records, push data storage performance to the limit, and technically lead service growth.

Key Responsibilities

High-availability data pipeline design: AWS MSK (Kafka), end-to-end data streaming architecture design and operation using Logstash
Storage optimization and sharding strategies: Optimizing large-scale search performance through OpenSearch's index design, sharding, and lifecycle (ISM) management
Data Modeling and Partitioning: Table partitioning and data modeling in PostgreSQL/MySQL, architecture guide for efficient querying
Performance Tuning and Guidance: Leading SQL optimization across services and spreading technical best practices to backend development teams
Release Management: Be the gatekeeper during key database / data pipeline release activities, create review checklist, review deployment details and validates details before deployment to ensure zero-error deployments without roll back needs.
Infrastructure automation: DB schema versioning (such as Flyway) and deployment process automation systems

Requirements

Over 7 years of experience in data engineering or related backend development
Extensive experience in operating infrastructure handling large-scale traffic in AWS cloud environments
Strong capabilities in building real-time data pipelines and troubleshooting using Kafka
Experience in OpenSearch (Elasticsearch) cluster design and sharding optimization is essential.
Deep understanding of the internal structure of RDBMS (PostgreSQL/MySQL) and ability to tune complex queries

Preferred Qualifications

Seamless technical collaboration capabilities: Communication skills to discuss infrastructure optimization with DevOps engineers and provide efficient data modeling and query guides to backend developers
Global collaboration skills: Able to share technical documents with engineers across the globe and conduct work via video conferences during collaborative work sessions
IoT Domain Experience: Experience in handling large-scale device messages and building data pipelines
Ensuring data stability: Experience in designing logic to prevent data loss during failures and ensure consistency verification
Nice to have è IaC Proficiency: Nice to have Experience coding and automated operations of data infrastructure using Terraform, Ansible, and others
4 years of bachelor’s degree or higher in Computer Science or equivalent discipline

What We Offer:

Competitive Salary
Work Life Balance
Professional Development
Innovative Work Environment.
Recognition and Rewards

Location:

IN-GJ-Ahmedabad, India-Ognaj (eInfochips)

Time Type:

Full time

Job Category:

Engineering Services

Skills & Tags

data

Aplyr's read

Arrow Electronics is a key player in technology solutions, employing a diverse workforce from engineers to sales professionals to drive innovation in electronics distribution.
Synthesized from recent postings & public sources

What's promising

•Strong global presence with operations in over 80 countries, offering vast career opportunities.
•Diverse role offerings from engineering to sales, catering to various career paths.
•Focus on cutting-edge technology solutions, keeping employees at the forefront of industry advancements.

What to watch

•Highly competitive industry may lead to pressure on margins and job security.
•Complex organizational structure could lead to bureaucratic challenges.
•Rapid technological changes require constant skill updates, potentially increasing employee stress.

Why Arrow Electronics

•Specializes in both electronic components and enterprise computing solutions, offering a broad industry scope.
•Strong emphasis on innovation, reflected in diverse engineering and technical roles.
•Global distribution network provides employees with international exposure and opportunities.

Aplyr’s read is generated by AI from public sources. Was it useful?