The Data Engineer Lead will be responsible for building, expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection for cross functional teams.
The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up.
The Data Engineer Lead will support our data analysts, data scientists on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects.
They must be self-directed and comfortable supporting the data needs of multiple teams, systems and products.
Responsibilities
- Build and maintain the Data Warehouse (DW) to store both structured and non- structured data; and make it optimized for reading, aggregating, and querying large volumes of data.
- Create and maintain optimal data pipeline architecture.
- Build the data pipeline infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS ‘big data’ technologies.
- Develop different components of the data management framework, including the Data Definitions for all data and the Metadata for data transformation/processing description.
- Assemble large, complex data sets that meet functional/non-functional business requirements.
- Work with stakeholders including the Executive, Product and Data Science teams to assist with data-related technical issues and support their data infrastructure needs.
- Prepare data for prescriptive and predictive modeling by the Data Science team to build and optimize our products and services.
- Create data tools for data analysts that assist them in build reports, dashboards and extract useful business insights.
- Assist in building analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics.
- Work with data and analytics experts to strive for greater functionality in our data systems.
- Keep our data separated and secure through multiple data centers.
- Work closely with quality assurance team to enhance data quality and reliability.
- Keep up to date with latest trends and technologies in data engineering and processing field.
Requirements
- A minimum of 4-6 years of work experience in a similar role
- Strong project management and organizational skills.
- Bachelor’s degree in Computer Science/Engineering or any other relevant field.
- Advanced working SQL knowledge and experience working with relational and NoSQL databases.
- Experience in building and optimizing ‘big data’ data pipelines, architectures and data sets.
- Working knowledge of message queuing, stream processing, and highly scalable ‘big data’ data stores.
- Experience with big data tools: Hadoop, Spark, Kafka, etc.
- Experience with AWS cloud services: EC2, EMR, RDS, Redshift, etc.
- Experience with stream-processing systems: Storm, Spark-Streaming, etc.
- Experience with object-oriented/object function scripting languages: Python, Java, C++, Scala, etc.
- Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
- A successful history of manipulating, processing and extracting value from large-disconnected datasets.
- Experience supporting and working with cross-functional teams in a dynamic environment.
- Strong analytical and problem-solving skills.
Benefits
- Training & Development
- performance bounce
- Work from Home
