Answers Engine Fetch documentation

Table of Contents

• Answers Engine Fetch
  • Getting Started
    • Install AnswersEngine Command Line Interface using rubygems
    • Get your access token
    • Set environment variable of your access token
    • Create the scraper
    • Deploying the scraper
    • Run the scraper
    • Viewing the Job Stats
    • Viewing the Job Pages
    • Viewing a Global Page Content
    • View the scraper output
    • View the scraper logs
• High Level Concepts
  • Scrapers
    • Available commands
  • Global Pages
    • Available Commands
  • Jobs
    • Available Commands
    • Paused Jobs
  • Job Workers
  • Job Pages
    • Available Commands
  • Job Outputs
    • Available Commands
  • Job Error Logs
    • Available Commands
  • Parsers
    • Reserved words or methods in parser scripts:
    • Available Commands
  • Seeder
    • Reserved words or methods in seeder scripts:
    • Available Commands
  • Finisher
    • Reserved words or methods in finisher scripts:
    • Available Commands
  • Exporters
    • Available Exporter Commands
    • Available Export Commands
    • Automatically Start Exporters
    • JSON Exporter
      • JSON Write Modes
    • CSV Exporter
      • CSV Fields
    • Content Exporter
      • Exporting Failed Contents
      • Customizing the File Names
  • Schemas
    • 1. Create the json schema file
    • 2. Create the schema config file and list the schema file that will be used to validate the collection(s)
    • 3. Update your config.yaml file to include the schema config file.
• User Access
• Scraper Development workflow
• Scraper Maintenance workflow
• Coding Tutorials
  • Simple Ebay scraper (Ruby)
    • Exercise 1: Create a base scraper locally
    • Exercise 2: Run the scraper on Fetch
    • Exercise 3. Create parser script
    • Exercise 4: Enqueue more pages, and pass variables to the next pages
    • Exercise 5: Exporting outputs
    • Exercise 6: Create a finisher script
• Advanced Tutorials
  • How to write a QA script to ingest and parse outputs from multiple scrapers
    • Step 1: Setup
    • Step 2: Seeding
    • Step 3: QA Validation Rules
    • Additional Validation Rules
    • Group Validations
    • Thresholds
    • External sources
• How-Tos
  • Setting a scraper’s scheduler
  • Changing a Scraper’s or a Job’s Proxy Type
  • Setting a specific ruby version
  • Setting a specific Ruby Gem
  • Changing a Scraper’s Standard worker count
  • Changing a Scraper’s Browser worker count
  • Changing an existing scrape job’s worker count
  • Enqueueing a page to Browser Fetcher’s queue
  • Setting fetch priority to a Job Page
  • Setting a user-agent-type of a Job Page
  • Setting the request method of a Job Page
  • Setting the request headers of a Job Page
  • Setting the request body of a Job Page
  • Setting the page_type of a Job Page
  • Reset a Job Page
  • Handling cookies
    • Low level cookie handling using Request/Response Headers
    • High level cookie handling using the Cookie Jar
  • Force Fetching a specific unfresh page
  • Handling JavaScript
  • Doing dry-run of your script locally
  • Executing your script locally, and uploading to Fetch
  • Querying scraper outputs
  • Restart a scraping job
  • Setting Environment Variables and Secrets on your account.
    • 1. Set the environment variable or secrets on your account.
    • 2. Change your config.yaml to use the variables or secrets.
    • 3. Access the environment variables and secrets in your script.
  • Setting Input Variables and Secrets on your scraper and scrape job.
    • 1. Set the input variable or secrets on your scraper.
    • 2. Change your config.yaml to use the variables or secrets.
    • 3. Access the input variables and secrets in your script.
  • Using a custom docker image for the scraper
  • How to use shared code libraries from other Git repositories using Git Submodule
  • How to debug page fetch
    • no_url_decode: true
    • http2: true
    • response headers and request headers are different
    • bzip compression headers
  • Advanced Usage
    • Parsing Failed Responses