PySpark Developer
Description
We are looking for a skilled Data Engineer with expertise in Python, PySpark, and Cloudera to join our team. The ideal candidate will be responsible for developing and optimizing big data pipelines while ensuring efficiency and scalability. Experience with Databricks is a plus. Additionally, familiarity with Git, GitHub, Jira, and Confluence is highly valued for effective collaboration and version control.
Key Responsibilities
- Design, develop, and maintain ETL pipelines using Python and PySpark.
- Work with Cloudera Hadoop ecosystem to manage and process large-scale datasets.
- Ensure data integrity, performance, and reliability across distributed systems.
- Collaborate with data scientists, analysts, and business stakeholders to deliver data-driven solutions.
- Implement best practices for data governance, security, and performance tuning.
- Use Git and GitHub for version control and efficient code collaboration.
- Track and manage tasks using Jira, and document processes in Confluence.
- (Optional) Work with Databricks for cloud-based big data processing.
Required Skills & Experience
- Strong programming skills in Python.
- Hands-on experience with PySpark for distributed data processing.
- Expertise in Cloudera Hadoop ecosystem (HDFS, Hive, Impala).
- Experience with SQL and working with large datasets.
- Knowledge of Git and GitHub for source code management.
- Experience with Jira for task tracking and Confluence for documentation.
- Strong problem-solving and analytical skills.
Preferred Qualifications
- Basic knowledge of Databricks for cloud-based big data solutions.
- Experience with workflow orchestration tools (e.g., Airflow, Oozie).
- Understanding of cloud platforms (AWS, Azure, or GCP).
- Exposure to Kafka or other real-time streaming technologies.
Axiom is a global information technology, consulting and outsourcing company and services provider. Our IT solutions empower organizations and individuals throughout the world to maximize value and quality to succeed in today's challenging business environment. As a fast-growing new economy company, we focus our strengths to offer world-class solutions and services through the convergence of technology, innovation, expertise and experience. We provide software consulting, development and IT-enabled services to clients across the globe. We work towards delivering sustained value creation for customers, employees, industries and society at large. Core offerings include data warehousing, middleware development, product development and web-enablement of legacy applications in verticals like telecom, finance, healthcare, manufacturing, energy & utilities, retail & distribution, enablement of legacy Relentless exploration of technology horizons and a Global Delivery Model that is a judicious combination of onsite, offsite and offshore development, offer a complete range of high-ROI business solutions spanning the consulting, technology, operations and process outsourcing value chain.
Please mention you found this job on AI Jobs. It helps us get more startups to hire on our site. Thanks and good luck!
Be the first to apply. Receive an email whenever similar jobs are posted.
Understand the required skills and qualifications, anticipate the questions you may be asked, and study well-prepared answers using our sample responses.
Developer Q&A's