Kafka Engineer, Middleware & Messaging Operations

In this role, you will:

• Support the continuous operation, optimization, and management of Apache Kafka and other messaging systems (e.g., RabbitMQ, Redis) for a wide range of software applications.
• Work closely with development and operations teams to ensure smooth integration of Kafka into application workflows, providing support and guidance.
• Monitor and maintain the performance and health of Kafka clusters, implementing proactive solutions to maintain reliability.
• Collaborate on capacity planning, scaling Kafka infrastructure to meet current and future demand without sacrificing performance.
• Implement and maintain self-service configuration tools, allowing teams to manage Kafka configurations through systems like Git.
• Develop and maintain infrastructure-as-code (IaC) solutions to manage messaging systems, ensuring that deployments and changes are consistent and repeatable.
• Contribute to the automation of operational tasks, including Kafka deployments, monitoring, and failure recovery processes.
• Ensure Kafka systems meet security and compliance standards, managing access controls and securing data in transit.
• Document configurations, best practices, and operational procedures to promote knowledge sharing and ensure team alignment.
• Participate in on-call rotations to provide 24x7 support, responding to incidents and troubleshooting issues to minimize downtime.

• Educational background: Bachelor’s or Master’s degree in Computer Science or a related field
• At least 2 years of experience in managing Apache Kafka cluster
• Strong hands-on experience with Apache Kafka in production environments, including Kafka Streams, Kraft, and Kafka Connect.
• In-depth knowledge of messaging system architecture, with an emphasis on designing, deploying, and optimizing Kafka clusters.
• Experience with automation tools (e.g., Terraform, Jikkou, Ansible) and scripting (e.g., Python, Bash) to streamline Kafka operations.
• Familiarity with containerization technologies (e.g., Docker, Kubernetes) and their integration with Kafka.
• Knowledge of monitoring tools (e.g., Prometheus, Grafana, Kafka-UI) and experience setting up proactive alerting systems.
• Experience with version control systems like Git for managing Kafka configuration.
• Understanding of security best practices for messaging systems, including access management and encryption.

Nice-to-Haves:

• Experience with other messaging platforms such as RabbitMQ or Redis.
• Experience with disaster recovery and business continuity planning for Kafka infrastructure.
• Experience with Solr and other middleware is a plus.

Who you are:

• A technical expert passionate about Kafka and messaging technologies.
• A problem solver who thrives in fast-paced, production-critical environments.
• Eager to collaborate with others, sharing your expertise to strengthen the team.
• Driven to continuously improve processes and systems through automation and innovation.
• A forward thinker, constantly seeking opportunities to automate and improve processes.