Kafka Developer

Job Title: Kafka Developer

Job Summary

We are seeking a highly skilled Java – Kafka Integration Engineer with strong expertise in Apache Kafka, Kubernetes, and distributed systems. The ideal candidate will be responsible for designing, developing, deploying, and maintaining scalable Kafka-based integration solutions while ensuring high availability, performance, and security across distributed environments.

Key Responsibilities

- Design, implement, and manage Kafka-based messaging and streaming solutions.
- Administer and maintain Kafka clusters, ensuring reliability, scalability, and fault tolerance.
- Develop Kafka producers, consumers, and stream processing applications using Java (or Python where applicable).
- Deploy and manage Kafka and related components in Kubernetes environments.
- Design and implement Custom Resource Definitions (CRDs) and develop controllers to manage Kubernetes resources.
- Integrate Kafka with API Gateway solutions to securely expose and manage APIs.
- Automate infrastructure provisioning, deployment, and operational tasks using scripts and CI/CD pipelines.
- Monitor Kafka clusters and applications, proactively identifying and resolving performance or reliability issues.
- Collaborate with cross-functional teams to support integration, DevOps, and platform engineering initiatives.

Required Skills & Qualifications

1. Kafka Expertise (Administration & Development)
- In-depth knowledge of Apache Kafka architecture, including topics, partitions, brokers, producers, and consumers.
- Strong experience with Kafka APIs:
- Producer API
- Consumer API
- Kafka Streams API
- Kafka Connect API

2. Programming Languages
- Strong proficiency in Java (preferred) or Python for Kafka application development.
- Familiarity with scripting languages such as Bash or PowerShell for automation and operational tasks.

3. Kubernetes Expertise
- In-depth understanding of Kubernetes, including:
- Operators and providers
- Container management
- Component upgrades and lifecycle management
- Debugging Kubernetes resources and managed objects
- Proficiency with Custom Resource Definitions (CRDs).
- Ability to design and implement new CRDs and develop controllers to manage them effectively.

4. API Gateway Knowledge
- Hands-on experience with at least one API Gateway technology such as:
- Kong
- Apigee
- Gravitee

5. DevOps & CI/CD
- Solid understanding of DevOps practices.
- Experience with CI/CD pipelines to automate build, test, and deployment processes.

6. Distributed Systems
- Strong understanding of distributed systems concepts, including:
- Replication
- Partitioning
- Fault tolerance
- High availability

7. Monitoring & Logging
- Experience with monitoring and observability tools such as Grafana, Kafka Manager, or similar.
- Familiarity with logging frameworks to ensure system health, performance, and reliability.

8. Automation & Scripting
- Strong automation and scripting skills for managing Kafka infrastructure, deployments, and operational workflows.