At 3Pillar Global, we specialize in building impactful software products that power digital businesses. We are looking for a talented Software Engineer who will not only write code but also deliver exceptional product capabilities for our clients. Your contributions will play a pivotal role in achieving our passion for client success, solidifying our position as a leading product development services company.
We are hiring a Senior Data Engineer coming with a strong experience in building data pipeline using PySpark, Python, ETL and Hadoop. Should have experience in Shell scripting and Banking domain.
The incumbent has to work from Office location - Vikroli, Mumbai for all weekdays.
Responsibilities:
- Design, develop, and maintain data pipelines and workflows using Hadoop and PySpark to process large volumes of data efficiently.
-
Implement data ingestion, transformation, and storage solutions to support business requirements.
-
Develop and optimize Python scripts and algorithms for data manipulation, analysis, and modeling tasks.
-
Design and manage database systems, including schema design, indexing, and performance optimization.
-
Collaborate with cross-functional teams to understand data requirements and deliver solutions that meet business needs.
-
Perform data quality assessment, validation, and cleansing to ensure accuracy and integrity of data.
-
Monitor and troubleshoot data pipeline issues, performance bottlenecks, and system failures.
-
Stay up to date with emerging technologies and best practices in data engineering and analytics.
Qualification:
- Bachelor's degree in Computer Science, Engineering, or related field.
-
Should have a minimum of 5 years of experience in the field of Database or Data Analytics
- Minimum of 3 years of experience in data engineering, with expertise in Hadoop, PySpark, Python, and database technologies.
-
Strong proficiency in Hadoop ecosystem components such as HDFS, MapReduce, Hive, and Spark.
-
Experience with PySpark for data processing and analysis tasks.
-
Proficiency in Python programming for scripting and automation.
-
Solid understanding of database concepts and hands-on experience with SQL and NoSQL databases.
-
Experience with data modeling, schema design, and ETL processes.
-
Strong problem-solving skills and ability to work independently and collaboratively in a fast-paced environment.
-
Excellent communication and interpersonal skills, with the ability to effectively communicate technical concepts to non-technical stakeholders.
