We are looking for a skilled Engineer with focus on AWS Cloud, IaC, Python and proven experience processing data with Athena to join our team and contribute to the development and maintenance of a Datalake platform.
The ideal candidate will be responsible for amplifying the platform itself, including provisioning new infrastructure following IaC practices using Terraform, writing supplemental ingestion Lambda functions using Python, developing new Ingestion workflows and verifying their performance on AWS infrastructure.
Key Responsibilities:
- Participate in team's ceremonies understanding SAFe framework and Scrum.
- Develop new functions of the platform including the infrastructure of it.
- Support and maintain existing platform features.
- Learn new SaaS technologies like Metabase or Upsolver in a short period of time. pipelines - Excellent communication skills, attention to detail, and an analytical mind.
- Experience with AWS Lambda and AWS Cloud Watch
- Experience in Python development for simple ingestion (SDK integration and dump to S3 datalake) using AWS Lambda functions.
- Experience in writing Terraform HCL for IaC on AWS.
- Experience with AWS cloud monitoring (Cloudwatch, Cloudtrail).
- Hands on experience using AWS Glue Catalog, and Athena processing data stored on S3. Understanding of AWS RDS (Postgres).
- Experience with managing AWS infra including EC2 provisioning/management, autoscaling and serverless processing.
- Good understanding of AWS IAM security model.
- Working knowledge on Lambda.
- Strong teamwork, motivation, and communication skills.
- Proficiency in communication in English.
- Proven experience in developer or engineering or system engineering roles, with a focus on data retrieval and enrichment.
- Knowledge of AWS services and SQL.
- Ability to document and troubleshoot errors.
- Working knowledge of test management software, understanding of SDLC
- Python (Capable), Terraform (Strong),
Good to have:
- Athena schema layout optimization and Athena Query Tuning.
- Experience using different communication technologies and protocols.
- Experience with Azure, OCI (specifically data retrieval from its blob storages).
- Experience with Upsolver or Metabase - CI tools experience.
- Experience Linux Operating system management.
