Run Scrapy spiders on Apify
Reliable cloud infrastructure for your Scrapy project. Run, monitor, schedule, and scale your spiders in the cloud.
Easy start
From Scrapy spider to Apify Actor
Deploy your Scrapy code in the cloud with just a few commands.
1# Install Apify CLI
2brew install apify-cli
3
4# Create Apify Actor from your Scrapy spider
5cd your-scrapy-project
6apify init
7
8# Deploy it to Apify
9apify login
10apify pushFull support for Scrapy
We fully support Scrapy spiders as they are. Take advantage of the Apify platform's features without having to modify your Scrapy spider.
Total package
Comprehensive tooling for your scrapers
Scheduling, monitoring, API, proxies, storage — you name it, we've got it. Explore our powerful features designed for reliable scraping at scale.
Run your spiders in the cloud
Use our CLI to quickly deploy your code to the cloud.
API and scheduler
Run your spider on schedule or call it via API.
Integration at your command
Connect your Scrapy projects with your favorite web apps.
Performance and quality monitoring
Inspect all spider runs, their logs, and runtime costs. Listen to events and get custom automated alerts.
Various formats and storage
Scalable and reliable cloud data storage designed for web scraping and automation workloads.
Proxies
Never get blocked with our carefully curated pool of datacenter and residential proxies and human-like browser fingerprints.
Monetize your spiders
Join hundreds of developers who publish their web scrapers as paid Actors on Apify Store, find users and get paid.
Other ways to create an Actor
Learn more
Looking for more Python tools? Our code templates, docs and library are a good place to start.
Platform Docs
Find more details and make your first steps on the Apify platform.
Scrapy & Python code templates
Build scrapers quickly with Python templates for Requests, Beautiful Soup, Scrapy, Playwright, and Selenium.
Python SDK
Use our library built for Actor lifecycle management, local storage emulation, and event handling.
Actorization
Actorized spiders
See the Scrapy spiders that were already transformed into Actors and published in Apify Store.
Sephora.com Scraper
scraping_empire/sephora-com-scraper
Scrape everything on Sephora.com from category products, product urls, and even search results.
scraping_empire
9
YoutubeToAudio
astounding_anise/youtubeToAudio
Introducing the ultimate YouTube Video to Audio Converter - your solution to effortlessly transform YouTube videos into high-quality audio files! With this powerful actor, you can easily extract audio tracks from any YouTube video with just a few clicks.
astounding_anise
10
Social & Email Scraper
kirada/social-email-scraper
Empower your data quest! The Social & Email Scraper Apify Actor: your precision-driven solution for effortlessly extracting vital contact details from any webpage. Elevate outreach strategies with ease and conquer web data intricacies.
kirada
187
Woocommerce Scraper by Bumble
bumble/woocommerce
Our WooCommerce Scraper seamlessly integrates with any WooCommerce website, effortlessly gathering detailed product and pricing data. Download results in HTML, JSON, CSV, Excel, XML, or RSS for versatile use.
bumble
52
Github Trending Repositories / Developers
saswave/github-trending-repositories-developers
From a Github Trending category, extract all related informations about repositories or developers trending date range in Daily / Weekly / Monthly. With filters based on language spoken, code language, sponsorable status and date range
saswave
4
Kijiji.ca Scraper
smartspidering/kijiji-ca-scraper
Scrape and extract kijiji.ca products data.
smartspidering
8
Yes, to access all the platform features you would need to get an account using your email. You can explore our platform's capabilities for free with our Free plan or choose a more extensive plan for your spiders.
Many developers earn passive income by sharing their Actors on the Apify Store, but the exact earnings depend on usage and user subscriptions. You can make anything from a few bucks to a hundreds of dollars per month. The amount of money you can make from hosting spiders on the Apify platform varies based on factors like the complexity of your Actors and the demand for the data they provide.
To give you a more specific answer, last month we paid out more than $100K in total to community Actors' developers. You can read about the journeys from two of our them - Caleb David or Tuğkan Cengiz.
The datasets in the output are available in multiple formats, including JSON, XML, CSV, HTML, and RSS. You can download your data, export it using an API or plug it into other apps.
Yes. Your Scrapy spider can be connected with almost any cloud service or web app thanks to integrations on the Apify platform. You can integrate with Make, Zapier, Slack, LangChain, Airbyte, GitHub, Google Sheets, Google Drive, and more. Or you can use webhooks to carry out an action whenever an event occurs, e.g. get a notification whenever your new Actor successfully finishes a run.
Yes. You can start the scraper, start, create, delete tasks, run them synchronously, start builds, get logs, create webhooks, get datasets, monitor your Scrapy spider using an API. You can also access key-value stores, request queues and user data. API can be accessed with Python, Node.js, and directly with Endpoints.
Yes. You can do so by either republishing your Scrapy spider from Zyte to the Apify Store (See monetization process here) or by picking custom web scraping projects from our Discord.
From Discord, you can provide scraping services for requests coming to the specialized channel. You will be the one communicating with the customer, setting deadlines and deliverables for their case. From Store, your published Actor acts as a web scraping tool rather than web scraping service. You will be only responsible for its maintenance and improvements. However, even in this case you can provide web scraping as a service if potential users contact you with requests to customize your existing Actor or to build them a new one.
Learn more about Scrapy
Learn how to scrape using Scrapy, the open-source data extraction framework developed by Zyte.
Please feel free to contact us at python@apify.com.