Staff Software Engineer - Data Engineering

Lookout, Inc. is the endpoint to cloud security company purpose-built for the intersection of enterprise and personal data. We safeguard data across devices, apps, networks and clouds through our unified, cloud-native security platform — a solution that's as fluid and flexible as the modern digital world. By giving organizations and individuals greater control over their data, we enable them to unleash its value and thrive. Lookout is trusted by enterprises of all sizes, government agencies and millions of consumers to protect sensitive data, enabling them to live, work and connect — freely and safely. To learn more about the Lookout Cloud Security Platform, visit www.lookout.com and follow Lookout on our blog, LinkedIn and Twitter.

As a Staff Software Engineer on the Data Engineering team, you will contribute broadly to the data engines, ETL pipelines, analysis and aggregation services, and other core intellectual property related services at Lookout. This position is an opportunity to take on some of the most interesting challenges in the company and join the team that provides and supports the fundamental building blocks of the Security Platform that underlies Lookout’s category-defining personal and enterprise products. Lookout’s Security Platform is capable of analyzing millions of apps on a daily basis. Our analysis is trusted by Google to help identify potentially harmful applications before they enter the Play Store and by the world’s leading telecommunications providers to protect their customers. These are challenging problems and you must be open-minded to tackling new exploratory areas as well as evolving and continually improving key existing systems. You’ll have a strong sense of ownership of product and quality, and you’ll work closely with our super-talented development team. At Lookout, we do believe quality is every engineer’s responsibility.

We're looking for motivated engineers who have experience building, monitoring, and maintaining high-volume, low-latency distributed SaaS solutions, with emphasis on the Data management and performance. You will be a member of the team that is responsible for a massively scalable platform that uses the data generated from a variety of core services of Lookout, including static analysis of iOS and Android applications, and phishing, smishing and social engineering focused detections.

At Lookout, we believe in agile software development in every sense of the word, small and focused teams, metrics that work, short feedback loops, well-designed APIs, easy to use UIs, test driven development, automation wherever possible, and making the right decisions.

Requirements for the position:

• Strong software engineering fundamentals - Object Oriented design, data structures, and algorithms

• Experience in building frameworks and solutions for Data platforms, focusing on the BI pipelines

• Broad understanding of cloud architecture tools and services, such as S3, EMR, Glue, Athena, Kafka, Kubernetes, Lambda functions. Experience in AWS is highly desirable

• Rich experience and deep expertise in the areas of Big Data and large scale data platforms, especially in Data Lake

• Stream processing engines – Spark Structured Streaming/Kafka

• Analytical processing on Big Data using Spark

• Hands-on administration, configuration management, monitoring, performance tuning of Spark Batch and Streaming workloads, Distributed platforms, and JVM based systems

• Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases

• Excellent written and verbal communication skills

• Experience in ETL testing (Billing or Data Warehouse) is a plus

• Knowledge of automation, CI/CD tools like Jenkins, CloudBees, Spinnaker, others

• 5-8+ years of overall experience in developing / maintaining large-scale, distributed production class software on public cloud platforms such as AWS

• BS degree in Computer Science or similar Engineering discipline, or equivalent work experience

Responsibilities:

• Responsible for data pipeline orchestration

• Create and maintain code and repositories

• Create large data sets for functional and performance tests

• Create different ETL jobs

• Develop and contribute to test framework

• Debug test failures and triaging production issues

• Validation and certification of the release candidate

• Functional, System, Smoke testing on various data pipelines

• Building tools for validation, debugging of data pipelines

• Work with peer Data Engineers, engineers, developers and other members of the team to understand the complex systems and develop solutions for the same

• Understands the business requirements and author designs and PoC’s for the Data Engineering projects

• Build automation tests that fit in CI/CD pipeline

• Participate in code reviews

Tools that you will work with:

• Scala, Java, Python, and others as needed

• Build Tools: Gradle, Maven, Sbt, and others

• Linux (SSH, Bash, scripting)

• AWS primitives and distributed technologies: SWF, EMR, Kafka, EC2, IAM roles, and others

• Understanding of Docker is nice to have