We are looking for people with experience gathering data from the web at scale. You’ll be part of Luma’s infrastructure team, working cross-functionally with Applied AI and Research on mission critical work-streams.
Responsibilities
- Own / lead engineering projects for data acquisition.
- Work with Research and Applied AI to ensure their data and technical needs are met.
- Develop and deploy highly scalable distributed systems capable of handling petabytes of data.
- Build and gather high quality datasets by collaborating closely with the research team.
- Build and maintain backend services for data storage.
- Architect and implement algorithms for data management including indexing & search.
- Conduct and analyze system performance experiments.
Experience
- Proficiency in Python.
- Experience of 5+ years in web-scraping at-scale.
- Experience building and working efficiently in very large distributed systems.
- Experience working with Kubernetes, proxies and other relevant tools.
- Experience working with research datasets.
- Please note this role is not meant for recent grads.
All your applications are reviewed by real people.
