Role: Java Developer with Web Crawler Experience
Location: Austin TX(Hybrid)
Responsibilities:
1. Web Crawler Development: Design and implement efficient and scalable web crawlers in Java to collect data from various online sources.
2. Data Extraction: Develop and maintain systems for structured data extraction, handling various data formats (HTML, JSON, XML, etc.).
3. Data Storage and Processing: Design data storage and processing pipelines, ensuring extracted data is clean, structured, and easily accessible.
4. Performance Optimization: Optimize web crawling processes for speed, efficiency, and accuracy, while ensuring minimal impact on source websites.
5. Error Handling and Logging: Implement error-handling mechanisms and logging systems to detect and resolve issues during crawling operations.
6. Data Integrity and Compliance: Ensure data collection practices are ethical, legal, and compliant with relevant regulations (e.g., robots.txt, copyright laws).
Requirements:
Proficiency in Java and experience with Java-based web scraping libraries (e.g., Jsoup, Apache HttpClient).
Knowledge of web crawling frameworks and tools, such as Scrapy, Selenium, or Puppeteer.
Strong understanding of HTML, CSS, JavaScript, and web data structures.
Familiarity with data parsing and handling techniques for JSON, XML, and other common formats.
Experience with database technologies (SQL, NoSQL) to store and manage scraped data.
Knowledge of HTTP protocols, headers, proxies, and load handling.
Axiom is a global information technology, consulting and outsourcing company and services provider. Our IT solutions empower organizations and individuals throughout the world to maximize value and quality to succeed in today's challenging business environment. As a fast-growing new economy company, we focus our strengths to offer world-class solutions and services through the convergence of technology, innovation, expertise and experience. We provide software consulting, development and IT-enabled services to clients across the globe. We work towards delivering sustained value creation for customers, employees, industries and society at large. Core offerings include data warehousing, middleware development, product development and web-enablement of legacy applications in verticals like telecom, finance, healthcare, manufacturing, energy & utilities, retail & distribution, enablement of legacy Relentless exploration of technology horizons and a Global Delivery Model that is a judicious combination of onsite, offsite and offshore development, offer a complete range of high-ROI business solutions spanning the consulting, technology, operations and process outsourcing value chain.
Please mention you found this job on AI Jobs. It helps us get more startups to hire on our site. Thanks and good luck!
Understand the required skills and qualifications, anticipate the questions you may be asked, and study well-prepared answers using our sample responses.
Java Developer Q&A's