Scraper Maintenance workflowΒΆ

  1. Look at the job outputs and/or job logs
  2. If there is an error, pause or cancel the affected job on the AnswersEngine Fetch scraper
  3. Go to the scraper directory locally
  4. Modify the seeder script or parser scripts locally
  5. Run the seeder script or parser script locally against some fetched job pages
  6. Deploy the scraper
  7. Resume the scrape job, or run another scrape by creating another scrape job.
  8. Check the job outputs on AnswersEngine