Apify for developers
Apify is a web scraping and automation platform. It provides a number of services to help you turn any website into an API.
Crawler
Recursively crawl large websites and extract data from them using a few simple lines of JavaScript.
-
Full JavaScript power
Tell the crawler what to do using a few simple lines of JavaScript with jQuery, Underscore.js or any other client-side library. Your code runs in a browser hosted on our servers.
-
Navigates arbitrary websites
Crawl and extract data from websites that employ AJAX, complex pagination or infinite scroll using the same tools you already use for your front-end development.
-
Entirely in the cloud
No need to install browser plugins, download any tools, set up cron jobs, databases or proxy servers. You already have all you need - your current web browser.
settings Actor
Actors run on our serverless computing platform to enable the creation of custom web scraping and automation jobs.
-
Use headless Chrome and Puppeteer
Apify provides first-class support for running headless Chrome jobs using Puppeteer. You only need to select the appropriate base image for your act and then call the
launchPuppeteer()function provided by theapifyNPM package. -
Host your code on GitHub
The source code of your Apify actors can be hosted directly on Apify, as well as on GitHub, Docker Hub or in fact anywhere on the internet. Your actor can be automatically built and published on every push to Git. It only takes a few clicks to move your existing project to Apify and publish it in the library for anyone to use it!
-
Go beyond traditional serverless platforms
Unlike traditional serverless computing platforms, the run of an Apify actor is not time-limited. Actors can run for as long as necessary, even forever. An actor can perform anything from a simple action, such as sending an email or cropping an image, to complex operations such as crawling an entire website and removing duplicates from a large dataset.
fast_forward Proxy
Fully-fledged proxy solution offering proxy rotation, automatic health monitoring and high grade of customizability.
-
Inteligent IP Rotation
Proxy servers are rotated automatically for every request using a semi-random algorithm which selects the least used proxy server for the requested hostname. This method ensures that traffic is directed through an unpredictable sequence of proxy servers while at the same time each server is not used too many times. Sessions with fixed proxy servers are also available.
-
Health monitoring
We continuously monitor the health of our proxy servers, discard any servers with issues from our pool and automatically replace them for our users. This method drastically increases the stability of our service when compared to traditional manual solutions. We also offer custom host monitoring where we periodically check if target host blocks a proxy server which is then immediately discarded to prevent failures.
-
Customizable
We work with multiple providers and offer custom solutions for your use cases. We can provide different locations from around the world, select the best proxies for your project, and you can also use your own proxy servers with our proxy rotation and health checking.
Get started
flare Additional services
There's much more and we're adding new features and services every day! View change log.
-
Scheduler
Schedule your crawlers or actors to run periodically every day, hour, minute or even every second using a cron-like expression.
-
Storage
Apify provides a simple key-value store to help you quickly store and retrieve data from your actors.
-
Integrations
Apify integrates seamlessly with Zapier, Keboola, Integromat or in fact any other service thanks to the REST API.
Become
an Apify expert
Every day we get dozens of requests from people all over the world to set up web automation solutions for them. Would you like to help us with that and make some money? Please let us know at jobs@apify.com