Systems Engineer, Kernel (Performance)

Senior Systems Engineer, Kernel Performance & Observability

CoreWeave is seeking a Systems Kernel Engineer focused on Performance and Observability to join our HAVOCK Team. In this role, you will build the next generation of automated introspection tools that detect issues before our customers do. You will leverage eBPF and programmable debuggers to create "OOB (Out of Band) Kernel Introspection" capabilities, allowing us to analyze live systems and core dumps with unprecedented depth. You will turn manual debugging steps into automated software solutions.

Hardware - Acceleration - Virtualization - Operating Systems - Containerization - Kubelet

Our Team’s Stack:

• Observability: eBPF (BCC/libbpf), drgn, bpftrace, perf
• Python, Go, C
• Custom Linux Kernel, Ubuntu
• Prometheus, Victoria Metrics, Grafana
• Docker, kubernetes (k8s)

Focus Areas:

• Advanced Observability – Designing eBPF-based tracing tools to analyze latency, resource contention, and locking in real-time.
• Introspection Tooling – Advancing our usage of programmable debuggers (like drgn) to analyze live kernels and core dumps.
• Automated Triage – Building systems that automatically categorize kernel crashes (RIP registers, stack traces) and reporting.
• Out-of-Band (OOB) Debugging – Implementing stealth forensics to debug hosts via DPUs without impacting the main CPU.
• Performance Analytics – Correlating kernel metrics with fleet-wide telemetry in Grafana.

Responsibilities:

• Develop eBPF programs to trace complex performance regressions in production.
• Build and maintain Debug images and associated tooling.
• Automate the analysis of soft lockups and crash dumps (automating crash/drgn workflows).
• Work on "OOB Kernel Introspection," enabling memory traces and debugging.
• Collaborate with the fleet team to integrate kernel metrics into high-level dashboards.

Requirements:

• 5+ years of experience in Systems Engineering or Kernel Development.
• eBPF Mastery: Strong experience writing eBPF programs (using bcc, bpftrace, or libbpf) for production environments.
• Scripting for Debugging: Proficiency in Python (specifically for drgn or GDB scripting) to automate analysis.
• Deep Linux Internals: Understanding of how to trace the scheduler, memory subsystem, and block layer.
• Experience connecting system metrics to visualization tools (Prometheus, Grafana).

Nice-to-haves:

• Experience with OOB (Out of Band) management
• Experience with Fuzzing (Syzkaller) or Sanitizers (KASAN, UBSAN)
• Previous wo