What You'll Do

This Data Engineer will join our growing Data Science Team.  Our data processing pipeline is implemented using AWS services and Apache Spark to post-process file-based and streaming inputs.  The data science utilities development platform is Python-based, using boto3 to access data, with numpy and Pandas-based data manipulation tools to access and leverage the data for R&D purposes.  You will collaborate with multiple engineering teams to ensure viability of data solutions with data science, software engineers, devops, QA, and systems engineers.  You will participate in data processing architecture design discussions and documentation.  You'll have the opportunity to develop and implement scalable data processing capabilities, ingress and egress data storage and retrieval interfaces.  You'll develop and maintain observability features that allow real-time insights into data processing status and automated unit and integration tests to minimize regressions and accelerate the software release process 

This role is a great opportunity for a growing data engineer to get hands-on experience with scaling from basic data warehousing problems to building and maintaining large-scale datalake ecosystems with complex system requirements. 

Role Responsibilities - How You Will Make an Impact

Design & Optimize Data Pipelines: 

• Develop, enhance, and refactor scalable ETL pipelines to handle large-scale IoT data. 
• Optimize existing Apache Spark workflows for increased performance and reliability. 

AWS Infrastructure Management: 

• Manage and expand our AWS-based infrastructure (S3, EC2, Lambda, EMR, etc.) to support data storage, processing, and orchestration. 
• Ensure cost-efficient resource utilization and scalability in the cloud environment.

Collaboration & Deployment: 

• Act as primary liaison between software architects and data science team, refining solution designs to be “necessary and sufficient” with respect to both needs and constraints.  
• Work closely with data scientists and ML engineers to integrate and deploy machine learning solutions into production. 
• Implement CI/CD best practices for data pipelines and support rapid iteration cycles. 

Performance Monitoring & Troubleshooting: 

• Establish robust monitoring systems for data pipelines and cloud services.
• Diagnose and resolve performance bottlenecks and system issues proactively. 

Data Quality & Security: 

• Enforce data governance, quality checks, and security protocols across the data lifecycle.
• Ensure compliance with industry standards and regulatory requirements. 

Innovation & Process Improvement: 

• Stay abreast of emerging technologies and recommend improvements to scale and streamline our data architecture. 
• Participate in code reviews and contribute to the development of best practices for data engineering. 
• Deploy software using contemporary DevOps practices including multi-cloud, multi-tenant, and hybrid strategies.
• Safely operate firearms platforms under supervision from trained and licensed range officers and qualified ARC personnel.  Prior experience with firearms is not required.
• This position may require travel up to 10% of the time in support of in-person events including system testing. 

Relevant Skills and Experience

• Bachelor's degree in computer science, computer engineering or equivalent practical experience
• 3+ years of experience as a data engineer 
• Proven experience with Apache Spark and the Python data science ecosystem
• Demonstrated expertise in building and scaling data pipelines in a cloud environment, preferably AWS
• Strong understanding of distributed systems and big data processing
• Excellent problem-solving skills and the ability to work collaboratively in a fast-paced, agile environment
• Strong communication skills and a proactive attitude toward learning and innovation
• Experience working in a start-up environment
• Experience working for DoD or government contractor
• Ability to obtain a DoD security clearance
• Familiarity with IoT data ingestion and machine learning workflows is a plus