A cloud data engineer is responsible for designing, building, and managing data processing systems and solutions within cloud environments. This typically involves working with a variety of cloud-based technologies, such as Amazon Web Services (AWS) to ensure that the organization's data is stored securely and efficiently.
We looking for a Cloud Data Engineer that will have a strong background in computer science, software engineering, or a related field, along with experience in database management, programming (e.g., Python, Java), and cloud computing platforms.
Responsibilities
- Designing, implementing, and managing the organization's data processing & transformation systems and pipelines on the cloud.
- Performing data migration to move and migrate data across cloud regions and cloud providers.
- Monitoring and optimizing the performance of the organization's data processing & transformation systems.
- Ensure the availability and security of databases.
- Troubleshooting complex issues with the database infrastructure.
- Line up all database schemata across all different deployments (Schema merge)
- Implementing backups and disaster recovery plans.
- Working with other IT teams to integrate the organization's databases with other systems.
- Keeping up-to-date with the latest developments in cloud-based database technologies and best practices.
- Collaborating with other IT teams to integrate the organization's databases with other systems.
- Converting/migrating databases to different database types.
Requirements
- A bachelor's or master's degree in computer science, software engineering, or a related field
- +8 year experience in DB admin/SQL developer position
- Experience with Data engineering technologies in Amazon AWS (Kafka, Clickhouse) (must have )
- Experience working with cloud-based database technologies, and Amazon Web Services (AWS) is required .
- Excellent Experience of AWS RDS Aurora setup and migration MySQL
- Expertise in SQL and NoSQL databases, as well as experience with data processing & transformation tools and techniques, such as but not limited to: Apache Kafka, Apache Spark, airflow, dbt.
- Knowledge of security best practices and experience implementing them in a cloud-based environment.
- Familiarity with agile software development methodologies and experience working in a fast-paced, dynamic environment
- Good Experience with ElasticSearch (must have).
- Good Experience with Database in Memory "Redis, Memcache" (must have)
- Proficient in writing and optimizing SQL statements.
- Ability to plan resource requirements from high-level specifications.
- Experience in NDB cluster/big data
- Experience with other DBMS, such as but not limited to: MangoDB, PostgreSQL, Clickhouse, Risingwave – both IaaS and DBaaS.
- Strong problem-solving skills and the ability to troubleshoot complex issues with the database infrastructure.
- Exceptional reporting skills, maintaining well-documented and current organizational data processing and transformation pipelines, ensuring their alignment with business objectives are clear and up-to-date.
