Answers Engine Fetch documentation¶
Table of Contents¶
- Answers Engine Fetch
- Getting Started
- Install AnswersEngine Command Line Interface using rubygems
- Get your access token
- Set environment variable of your access token
- Create the scraper
- Deploying the scraper
- Run the scraper
- Viewing the Job Stats
- Viewing the Job Pages
- Viewing a Global Page Content
- View the scraper output
- View the scraper logs
- Getting Started
- High Level Concepts
- User Access
- Scraper Development workflow
- Scraper Maintenance workflow
- Coding Tutorials
- Advanced Tutorials
- How-Tos
- Setting a scraper’s scheduler
- Changing a Scraper’s or a Job’s Proxy Type
- Setting a specific ruby version
- Setting a specific Ruby Gem
- Changing a Scraper’s Standard worker count
- Changing a Scraper’s Browser worker count
- Changing an existing scrape job’s worker count
- Enqueueing a page to Browser Fetcher’s queue
- Setting fetch priority to a Job Page
- Setting a user-agent-type of a Job Page
- Setting the request method of a Job Page
- Setting the request headers of a Job Page
- Setting the request body of a Job Page
- Setting the page_type of a Job Page
- Reset a Job Page
- Handling cookies
- Force Fetching a specific unfresh page
- Handling JavaScript
- Doing dry-run of your script locally
- Executing your script locally, and uploading to Fetch
- Querying scraper outputs
- Restart a scraping job
- Setting Environment Variables and Secrets on your account.
- Setting Input Variables and Secrets on your scraper and scrape job.
- Using a custom docker image for the scraper
- How to use shared code libraries from other Git repositories using Git Submodule
- How to debug page fetch
- Advanced Usage