Data Engineer - South Africa

Data Engineer 

We are seeking a Data Integrations / Big Data Engineer with experience in Apache NiFi, Kafka, and enterprise systems, to design and implement secure, scalable, and real-time data pipelines for a high-profile client implementation project. The ideal candidate will have expertise in managing structured and semi-structured data, integrating legacy and modern systems, and automating data flows across cloud and on-prem environments.

Key Responsibilities:

• Design, develop, and maintain data pipelines for ingesting and transforming data from various sources (including APIs, flat files, DBs).
• Integrate messaging systems like Kafka and RabbitMQ into the pipeline for event streaming and system decoupling.
• Work closely with Business Analysts and BPEs to translate client requirements into technical data flows.
• Implement and manage connectors for REST APIs, and file-based batch processes.
• Ensure data lineage, auditability, and performance tuning of NiFi flows.
• Deploy and monitor data flows in distributed environments using Zookeeper, Kafka, and ElasticSearch for logging and observability.
  
• Strong experience in designing and implementing data integration solutions for real-time streaming data.

Required Skills:

• 3–5+ years of experience in data integration/engineering roles.
• Experience with Apache NiFi (flow design, templates, version control, custom processors).
• Proficient in Kafka, RabbitMQ, and event-driven architectures.
• Experience integrating with legacy systems ((BAPIs, IDocs, OData).
• Scripting and transformation in Python, Groovy, or Java.
• Familiarity with distributed systems (Zookeeper, ElasticSearch, etc.).
• Experience with CI/CD for NiFi or container-based deployments (Docker, Kubernetes).
  
  

Nice to Have:

• Knowledge of security practices (TLS, role-based access in NiFi).
• Hands-on with monitoring tools (Prometheus, Grafana).
• Familiarity with cloud platforms (AWS/GCP/Azure) for data storage or processing.

Other Requirements:

• Eligibility for Top-Level Security Clearance:
  • Candidates must be eligible to obtain and maintain security clearance at the highest level, in accordance with applicable national security regulations.
• On-Site Work Requirement:
  • This role requires full-time, on-site presence at the client’s premises located in Pretoria. Remote or hybrid work arrangements are not applicable.