The Kafka Developer is responsible for designing, developing, and maintaining event-driven data pipelines and streaming solutions within the project ecosystem. The role focuses on enabling scalable, reliable, and loosely coupled microservices using Apache Kafka, tightly integrated with Spring Boot applications, CI/CD pipelines, and cloud platforms.
Day-to-Day Responsibilities
- Design and develop Kafka producers and consumers using Java and Spring Kafka to support event-driven workflows.
- Collaborate with Full Stack and backend teams to define event models, message contracts, and topic strategies.
- Design and manage Kafka topics, partitions, replication factors, and retention policies based on business and performance needs.
- Implement event-driven integration patterns, enabling asynchronous communication between microservices.
- Develop stream processing applications using Kafka Streams, including filtering, transformations, aggregations, and joins.
- Implement schema management using Avro, JSON, or Protobuf, ensuring backward and forward compatibility.
- Apply Kafka security best practices, including SSL/TLS, SASL authentication, ACLs, and secure data transmission.
- Design robust error-handling mechanisms, including retries, dead-letter topics, and idempotent message processing.
- Optimize Kafka performance by tuning producer/consumer configurations, partition strategies, and batching.
- Integrate Kafka-based services with Spring Boot microservices deployed on AWS, VMware TAS, or container platforms.
- Contribute to CI/CD pipelines for automated build, test, and deployment of Kafka applications.
- Monitor Kafka clusters and applications using Prometheus, Grafana, ELK, or native Kafka metrics.
- Perform production support and BAU activities, including troubleshooting message lag, consumer failures, and cluster issues.
- Conduct root cause analysis for incidents and implement preventive improvements.
- Participate in Agile ceremonies, including sprint planning, daily stand-ups, reviews, and retrospectives.
- Maintain technical documentation, including topic catalogs, event schemas, and integration guidelines.
- Continuously improve system reliability, scalability, and maintainability by adopting industry best practices.
- Stay updated on Kafka ecosystem enhancements and streaming technologies and proactively suggest improvements.
