Proxy

Apify Proxy provides access to Apify's proxy services that can be used in crawlers, actors or any other application that support HTTP proxies. Currently, Apify Proxy provides access to datacenter proxy servers, residential proxy, and Google SERP proxy. It supports HTTP as well as other protocols like HTTPS and FTP.

You can view your Apify Proxy settings on the Proxy page in the app.

Connection settings

The following table shows HTTP proxy connection settings for the Apify Proxy.

Proxy type HTTP
Hostname proxy.apify.com
Port 8000
Username Specifies parameters of the proxy connection. Please refer to the username parameters of the proxy solution you want to use in the sections below.
Password Proxy password. Your password is displayed on the Proxy page in the app. Also, in Apify actors, it is passed as the APIFY_PROXY_PASSWORD environment variable. See actor documentation for more details.
URL http://<username>:<password>@proxy.apify.com:8000

WARNING: All usage of Apify Proxy with your password is charged towards your account. Do not share the password with untrusted parties or use it from insecure networks, because the password is sent unencrypted due to the limitations of the HTTP protocol.

Datacenter proxy servers

Apify Proxy provides access to Apify's pool of datacenter IP addresses to crawlers, actors or any other application that support HTTP proxies. The proxy enables intelligent rotation of IP addresses during web scraping to avoid being blocked by target websites.

Overview

Datacenter proxy automatically rotates IP addresses. For each HTTP or HTTPS request, the proxy takes the list of all IP addresses available to the user and selects the one that has been used the longest time ago for the specific hostname. This behavior minimizes the chance of the proxy being blocked.

Note that by default each proxied HTTP request is potentially sent via a different target proxy server, which adds overhead and could be potentially problematic for websites which save cookies based on IP address. If you want to force the proxy to pick an IP address and then pass all subsequent connections via the same IP address, you can use the session parameter. See Username parameters more details.

Prices for dedicated proxy servers are mainly based on the number of proxy servers, their type, and location. Please contact us for more information.

Features

  • Periodic health checks of proxies in the pool to ensure requests are not forwarded via dead proxies.
  • Intelligent rotation of IP addresses to ensure target hosts are accessed via proxies that have accessed them the longest time ago, to reduce the chance of blocking.
  • Periodically checks whether proxies are banned by selected target websites, and if they are, stops forwarding traffic to them to get the proxies unbanned as soon as possible.
  • Ensures proxies are located in specific countries using IP geolocation.
  • Allows selection of groups of proxy servers with specific characteristics.
  • Supports persistent sessions that enable you to keep the same IP address for certain parts of your crawls.
  • Measures statistics of traffic for specific users and hostnames.

Shared proxy groups

Each user has access to a selected number of proxy servers from a shared pool of Apify Proxy servers. These proxy servers are spread into groups (called Proxy Groups) on the Apify platform where each group shares a common feature (location, provider, speed and so on).

The number of proxy servers available depends on the user's subscription plan. When a user first signs up to the Apify platform, a 30-day trial of the "Freelancer" plan is started, and proxy servers are allocated accordingly. After the trial ends, the user has to subscribe to a paid plan to continue using Apify Proxy.

For a full list of plans and number of allocated proxy servers for each plan, please take a look at our pricing.

Please contact us if you need more proxy servers then the allocated numbers, or you wish to use the proxy by itself without access to other features of the Apify platform.

Dedicated proxy groups

Apify Proxy allows for the creation of special dedicated proxy groups. These are assigned to a single user and only the user can use them.

This feature is useful if you have your own pool of proxy servers and still want to benefit from the features of Apify Proxy (like IP rotation, persistent sessions, and health checking).

Also, if you do not have your own pool, the Apify customer support team can set up a dedicated group for you based on your needs and requirements.

Please contact us for more details or if you have any questions.

Username parameters

HTTP proxy username is used to pass various parameters for the proxy connection. For example, the username can look as follows:

groups-SHADER,session-rand123456

The following table describes the available parameters:

groups If specified, all proxied requests will use proxy servers from selected proxy groups. For example groups-SHADER+BUYPROXIES94952.
This parameter is optional, by default, the proxy uses all available proxy servers from all groups the user has access to.
session If specified, all proxied requests with the same session identifier are routed through the same IP address. For example session-rand123456.
This parameter is optional, by default, each proxied request is assigned a randomly picked least used IP address.
The session string can only contain numbers (0-9), letters (a-z or A-Z), dot (.), underscore (_), a tilde (~) and the maximum length is 50 characters!

If you do not want to specify both `groups` and `session` parameters and therefore use default behavior for both, use the following username:

auto

Session persistence

When using Apify Proxy with session parameter set in the username (see Username parameters) a single IP is assigned to the session ID provided after the first request is made. This IP/session_id combination is persited, and its expiration is set to 24 hours later. Each additional request extends the expiration back to 24 hours, so if you use the session at least once a day it will never expire, with two possible exceptions:

  • Proxy server stops responding and is marked as dead during a health check
  • If the Proxy Server is part of a Proxy Group that is refreshed monthly and is rotated out.

If the session is discarded due to the reasons above, then a new IP is assigned to the session.

NodeJS Examples

The following sections contain several examples of how to use Apify Proxy in NodeJS (used as the default language in actors).

Usage in PuppeteerCrawler

Use default functionality of Apify Proxy (randomly choose a proxy server from all available)

const Apify = require('apify');

Apify.main(async () => {
    const requestList = new Apify.RequestList({ sources: [{ url: 'http://www.example.com' });
    await requestList.initialize();

    const crawler = new Apify.PuppeteerCrawler({
        requestList,
        launchPuppeteerOptions: { useApifyProxy: true },
        handlePageFunction: ({ page, request }) => {
            return Apify.pushData({ title: await page.title() });
        },
        handleFailedRequestFunction: ({ request }) => {
            console.error('Request failed', request.url, request.errorMessages);
        },
    });

    await crawler.run();
});

Only use proxy servers from proxy group SHADER during the PuppeteerCrawler run

const Apify = require('apify');

Apify.main(async () => {
    const requestList = new Apify.RequestList({ sources: [{ url: 'http://www.example.com' });
    await requestList.initialize();

    const crawler = new Apify.PuppeteerCrawler({
        requestList,
        launchPuppeteerOptions: {
            useApifyProxy: true,
            // if you need to use more then one group
            // simply add the additional ID's to the array below
            apifyProxyGroups: ['SHADER'],
        },
        handlePageFunction: ({ page, request }) => {
            return Apify.pushData({ title: await page.title() });
        },
        handleFailedRequestFunction: ({ request }) => {
            console.error('Request failed', request.url, request.errorMessages);
        },
    });

    await crawler.run();
});

Keep a single IP selected from SHADER proxy group during the whole PuppeteerCrawler run

const Apify = require('apify');

Apify.main(async () => {
    const requestList = new Apify.RequestList({ sources: [{ url: 'http://www.example.com' });
    await requestList.initialize();

    const crawler = new Apify.PuppeteerCrawler({
        requestList,
        launchPuppeteerOptions: {
            useApifyProxy: true,
            // if you need to use more then one group
            // simply add the additional ID's to the array below
            apifyProxyGroups: ['SHADER'],
            apifyProxySession: 'my_session',
        },
        handlePageFunction: ({ page, request }) => {
            return Apify.pushData({ title: await page.title() });
        },
        handleFailedRequestFunction: ({ request }) => {
            console.error('Request failed', request.url, request.errorMessages);
        },
    });

    await crawler.run();
});

Get a new IP selected from SHADER proxy group for each browser opened during the whole PuppeteerCrawler run

const Apify = require('apify');

Apify.main(async () => {
    const requestList = new Apify.RequestList({ sources: [{ url: 'http://www.example.com' });
    await requestList.initialize();

    const crawler = new Apify.PuppeteerCrawler({
        requestList,
        // Notice that now we are setting options
        // in launchPuppeteerFunction instead of launchPuppeteerOptions
        launchPuppeteerFunction: () => Apify.launchPuppeteer({
            useApifyProxy: true,
            // if you need to use more then one group
            // simply add the additional ID's to the array below
            apifyProxyGroups: ['SHADER'],
            apifyProxySession: `my_sess_${Math.floor(Math.random() * 100000)}`,
        }),
        handlePageFunction: ({ page, request }) => {
            return Apify.pushData({ title: await page.title() });
        },
        handleFailedRequestFunction: ({ request }) => {
            console.error('Request failed', request.url, request.errorMessages);
        },
    });

    await crawler.run();
});

Usage in Apify.launchPuppeteer()

Use one IP chosen from SHADER and BUYPROXIES94952 proxy groups for the browser instance

const Apify = require('apify');

Apify.main(async () => {
    const password = process.env.APIFY_PROXY_PASSWORD;
    const username = 'groups-SHADER+BUYPROXIES94952,session-my_session';
    const browser = await Apify.launchPuppeteer({
        proxyUrl: `http://${username}:${password}@proxy.apify.com:8000`,
    });

    const page = await browser.newPage();

    await page.goto('http://www.example.com');

    const html = await page.content();

    console.log('HTML:');
    console.log(html);
});

Usage with request and request-promise NPM packages

Use one randomly selected IP from all available proxy servers.

const request = require('request');

const password = process.env.APIFY_PROXY_PASSWORD;
request(
    {
        url: 'http://www.example.com',
        proxyUrl: `http://auto:${password}@proxy.apify.com:8000`
    },
    (err, response, HTML) => {
        if (err) {
            console.error(error);
            return;
        }
        console.log(HTML);
    }
);
        

Use one IP selected from SHADER proxy group for two requests.

const request = require('request-promise');

async function main() {
    const password = process.env.APIFY_PROXY_PASSWORD;
    const proxyUrl = `http://groups-SHADER,session-my_session:${password}@proxy.apify.com:8000`;
    try {
        const response1 = await request({
            url: 'https://api.apify.com/v2/browser-info',
            proxyUrl,
        });
        const response2 = await request({
            url: 'https://api.apify.com/v2/browser-info',
            proxyUrl,
        });
        console.log(response1.clientIp);
        console.log('should be the same as');
        console.log(response2.clientIp);
    } catch (e) {
        console.error(e);
    }
}
main();

Use randomly choose IP selected from SHADER and BUYPROXIES94952 proxy groups for two requests.

const request = require('request-promise');

async function main() {
    const password = process.env.APIFY_PROXY_PASSWORD;
    const proxyUrl = `http://groups-SHADER+BUYPROXIES94952:${password}@proxy.apify.com:8000`;
    try {
        const response1 = await request({
            url: 'https://api.apify.com/v2/browser-info',
            proxyUrl,
        });
        const response2 = await request({
            url: 'https://api.apify.com/v2/browser-info',
            proxyUrl,
        });
        console.log(response1.clientIp);
        console.log('should be different than');
        console.log(response2.clientIp);
    } catch (e) {
        console.error(e);
    }
}
main();

PHP Examples

The following section contain several examples of how to use Apify Proxy in PHP.

Usage with CURL

IMPORTANT: For all examples in this section you need to have cURL extension enabled in your PHP installation. See installation instructions for more information.

Single request with random IP chosen from all available proxy groups.

<?php
$curl = curl_init('https://api.apify.com/v2/browser-info');
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_PROXY, 'http://proxy.apify.com:8000');
// Replace <YOUR_PROXY_PASSWORD> below with your password
// found at https://my.apify.com/proxy
curl_setopt($curl, CURLOPT_PROXYUSERPWD, 'auto:<YOUR_PROXY_PASSWORD>');
$response = curl_exec($curl);
curl_close($curl);
if ($response) echo $response;
?>

Two requests with the same session chosen from all available proxy groups.

<?php
function doRequest() {
    $curl = curl_init('https://api.apify.com/v2/browser-info');
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($curl, CURLOPT_PROXY, 'http://proxy.apify.com:8000');
    // Replace <YOUR_PROXY_PASSWORD> below with your password
    // found at https://my.apify.com/proxy
    curl_setopt($curl, CURLOPT_PROXYUSERPWD, 'session-my_session:<YOUR_PROXY_PASSWORD>');
    $response = curl_exec($curl);
    curl_close($curl);
    return $response;
}
$response1 = doRequest();
$response2 = doRequest();
echo $response1;
echo "\nShould be contain same clientIp as\n";
echo $response2;
?>

Two requests with the different IPs chosen from SHADER and BUYPROXIES94952 proxy groups.

<?php
function doRequest() {
    $curl = curl_init('https://api.apify.com/v2/browser-info');
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($curl, CURLOPT_PROXY, 'http://proxy.apify.com:8000');
    // Replace <YOUR_PROXY_PASSWORD> below with your password
    // found at https://my.apify.com/proxy
    curl_setopt($curl, CURLOPT_PROXYUSERPWD, 'groups-SHADER+BUYPROXIES94952:<YOUR_PROXY_PASSWORD>');
    $response = curl_exec($curl);
    curl_close($curl);
    return $response;
}
$response1 = doRequest();
$response2 = doRequest();
echo $response1;
echo "\nShould have different clientIp than\n";
echo $response2;
?>

Python Examples

The following section contain several examples of how to use Apify Proxy in python.

Usage with Python 3+

Single request with random IP chosen from all available proxy groups.

import urllib.request as request
import ssl
# Replace <YOUR_PROXY_PASSWORD> below with your password
# found at https://my.apify.com/proxy
password = '<YOUR_PROXY_PASSWORD>'
proxy_url = f"http://auto:{password}@proxy.apify.com:8000"
proxy_handler = request.ProxyHandler({
    'http': proxy_url,
    'https': proxy_url,
})

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
httpHandler = request.HTTPSHandler(context=ctx)

opener = request.build_opener(httpHandler,proxy_handler)
print(opener.open('https://api.apify.com/v2/browser-info').read())

Two requests with the same session chosen from all available proxy groups.

import urllib.request as request
import ssl

def do_request():
    # Replace <YOUR_PROXY_PASSWORD> below with your password
    # found at https://my.apify.com/proxy
    password = '<YOUR_PROXY_PASSWORD>'
    proxy_url = f"http://session-my_session:{password}@proxy.apify.com:8000"
    proxy_handler = request.ProxyHandler({
        'http': proxy_url,
        'https': proxy_url,
    })

    ctx = ssl.create_default_context()
    ctx.check_hostname = False
    ctx.verify_mode = ssl.CERT_NONE
    httpHandler = request.HTTPSHandler(context=ctx)

    opener = request.build_opener(httpHandler,proxy_handler)
    return opener.open('https://api.apify.com/v2/browser-info').read()

print(do_request())
print('Should return the contain the same clientIp as ')
print(do_request())

Two requests with the different IPs chosen from SHADER and BUYPROXIES94952 proxy groups.

import urllib.request as request
import ssl

def do_request():
    # Replace <YOUR_PROXY_PASSWORD> below with your password
    # found at https://my.apify.com/proxy
    password = '<YOUR_PROXY_PASSWORD>'
    proxy_url = f"http://groups-SHADER+BUYPROXIES94952:{password}@proxy.apify.com:8000"
    proxy_handler = request.ProxyHandler({
        'http': proxy_url,
        'https': proxy_url,
    })

    ctx = ssl.create_default_context()
    ctx.check_hostname = False
    ctx.verify_mode = ssl.CERT_NONE
    httpHandler = request.HTTPSHandler(context=ctx)

    opener = request.build_opener(httpHandler,proxy_handler)
    return opener.open('https://api.apify.com/v2/browser-info').read()

print(do_request())
print('Should return the contain different clientIp than ')
print(do_request())

Usage with Python 2+

IMPORTANT: For all examples in this section you need to have six enabled in your Python installation. Run pip install six to enable it.

Single request with random IP chosen from all available proxy groups.

import six
from six.moves.urllib import request
# Replace <YOUR_PROXY_PASSWORD> below with your password
# found at https://my.apify.com/proxy
password = '<YOUR_PROXY_PASSWORD>'
proxy_url = (
    'http://auto:%s@proxy.apify.com:8000' %
    (password)
)
proxy_handler = request.ProxyHandler({
    'http': proxy_url,
    'https': proxy_url,
})
opener = request.build_opener(proxy_handler)
print(opener.open('https://api.apify.com/v2/browser-info').read())

Two requests with the same session chosen from all available proxy groups.

import six
from six.moves.urllib import request
import ssl

def do_request():
    # Replace <YOUR_PROXY_PASSWORD> below with your password
    # found at https://my.apify.com/proxy
    password = '<YOUR_PROXY_PASSWORD>'
    proxy_url = (
        'http://session-my_session:%s@proxy.apify.com:8000' %
        (password)
    )
    proxy_handler = request.ProxyHandler({
        'http': proxy_url,
        'https': proxy_url,
    })

    ctx = ssl.create_default_context()
    ctx.check_hostname = False
    ctx.verify_mode = ssl.CERT_NONE
    httpHandler = request.HTTPSHandler(context=ctx)

    opener = request.build_opener(httpHandler,proxy_handler)
    return opener.open('https://api.apify.com/v2/browser-info').read()

print(do_request())
print('Should return the contain the same clientIp as ')
print(do_request())

Two requests with the different IPs chosen from SHADER and BUYPROXIES94952 proxy groups.

import six
from six.moves.urllib import request
import ssl

def do_request():
    # Replace <YOUR_PROXY_PASSWORD> below with your password
    # found at https://my.apify.com/proxy
    password = '<YOUR_PROXY_PASSWORD>'
    proxy_url = (
        'http://groups-SHADER+BUYPROXIES94952:%s@proxy.apify.com:8000' %
        (password)
    )

    proxy_handler = request.ProxyHandler({
        'http': proxy_url,
        'https': proxy_url,
    })

    ctx = ssl.create_default_context()
    ctx.check_hostname = False
    ctx.verify_mode = ssl.CERT_NONE
    httpHandler = request.HTTPSHandler(context=ctx)

    opener = request.build_opener(httpHandler,proxy_handler)
    return opener.open('https://api.apify.com/v2/browser-info').read()

print(do_request())
print('Should return the contain different clientIp than ')
print(do_request())

Residential proxy

Sometimes datacenter proxy servers are not a viable option for certain solutions and for these cases Apify Proxy includes an option to use Residential Proxy. This proxy solution allows the user access to a much larger pool of proxy servers than with datacenter proxy servers and therefore it is usually a better option for situations where a large number of proxy servers is required.

On the Apify platform, users can use Residential Proxy after they are given access to it by the Apify Support Team. Pricing is based on data traffic, which is measured for each connection made through the proxy and displayed on the platform's dashboard.

Please contact us if you want to use Apify Residential Proxy or if you need more information.

Username parameters

HTTP proxy username is used to pass various parameters for the proxy connection. For example, the simplest way to use residential proxy is with the username below:

groups-RESIDENTIAL

The following table describes the available parameters:

groups Required to be set to RESIDENTIAL
session If specified, all proxied requests with the same session identifier are routed through the same IP address. For example session-rand123456.
This parameter is optional, by default, each proxied request is assigned a randomly picked least used IP address.
The session string can only contain numbers (0-9), letters (a-z or A-Z), dot (.), underscore (_), a tilde (~) and the maximum length is 50 characters!
country If specified, all proxied requests will use IP addresses that geolocated to the specified country. For example country-GB for IP's from Great Britain.
This parameter is optional, by default, each proxied request is assigned an IP address from a random country.
The country code needs to be a two letter ISO country code - see the full list of available country codes

This is how the username would look for the most complex variation: Session set and IP selected from the United States

groups-RESIDENTIAL,session-my_session_1,country-US

And here is how it would look if you need a random proxy from the US

groups-RESIDENTIAL,country-US

Session persistence

When using Apify Proxy with session parameter set in the username (see Username parameters) a single IP is assigned to the session ID provided after the first request is made. This IP is persisted for one minute and its expiration is refreshed with each request. If the proxy server becomes unresponsive or the session expires a new IP is selected for the next request.

NodeJS Examples

The following sections contain several examples of how to use Apify Proxy in NodeJS (used as the default language in actors).

Usage in PuppeteerCrawler

Use a single session with IP from the US for the whole PuppeteerCrawler run (for as long as the session lasts)

const Apify = require('apify');

Apify.main(async () => {
    const requestList = new Apify.RequestList({ sources: [{ url: 'http://www.example.com' });
    await requestList.initialize();

    const password = process.env.APIFY_PROXY_PASSWORD;
    const username = 'groups-RESIDENTIAL,session-my_session,country-US';
    const proxyUrl = `http://${username}:${password}@proxy.apify.com:8000`;

    const crawler = new Apify.PuppeteerCrawler({
        requestList,
        launchPuppeteerOptions: { proxyUrl },
        handlePageFunction: ({ page, request }) => {
            return Apify.pushData({ title: await page.title() });
        },
        handleFailedRequestFunction: ({ request }) => {
            console.error('Request failed', request.url, request.errorMessages);
        },
    });

    await crawler.run();
});

Create a new session with IP from GB for each browser launched during the Crawler run.

const Apify = require('apify');

Apify.main(async () => {
    const requestList = new Apify.RequestList({ sources: [{ url: 'http://www.example.com' });
    await requestList.initialize();

    const password = process.env.APIFY_PROXY_PASSWORD;

    const crawler = new Apify.PuppeteerCrawler({
        requestList,
        // The main difference from the example above
        // is that we are creating new session id for each
        // browser launched.
        launchPuppeteerFunction: () => {
            const sessionNumber = Math.floor(Math.round() * 100000);
            const username = 'groups-RESIDENTIAL,session-s_${sessionNumber},country-US';
            const proxyUrl = `http://${username}:${password}@proxy.apify.com:8000`;
            return Apify.launchPuppeteer({ proxyUrl });
        },
        handlePageFunction: ({ page, request }) => {
            return Apify.pushData({ title: await page.title() });
        },
        handleFailedRequestFunction: ({ request }) => {
            console.error('Request failed', request.url, request.errorMessages);
        },
    });

    await crawler.run();
});

Usage in Apify.launchPuppeteer()

Use a single IP from Germany for all requests done in the launched browser

const Apify = require('apify');

Apify.main(async () => {
    const password = process.env.APIFY_PROXY_PASSWORD;
    const username = 'groups-RESIDENTIAL,session-my_session1,country-DE';
    const browser = await Apify.launchPuppeteer({
        proxyUrl: `http://${username}:${password}@proxy.apify.com:8000`,
    });

    const page = await browser.newPage();

    await page.goto('http://www.example.com');

    const html = await page.content();

    console.log('HTML:');
    console.log(html);
});

Usage with request and request-promise NPM packages

Make a request with Residential Proxy using the request NPM package

const request = require('request');

const password = process.env.APIFY_PROXY_PASSWORD;
const username = 'groups-RESIDENTIAL';
request(
    {
        url: 'http://www.example.com',
        proxyUrl: `http://${username}:${password}@proxy.apify.com:8000`
    },
    (err, response, HTML) => {
        if (err) {
            console.error(error);
            return;
        }
        console.log(HTML);
    }
);
        

Use the same IP for two requests with proxy geolocated in France using request-promise NPM package.

const request = require('request-promise');

async function main() {
    const password = process.env.APIFY_PROXY_PASSWORD;
    const username = 'groups-RESIDENTIAL,session-my_session_3,country-FR';
    const proxyUrl = `http://${username}:${password}@proxy.apify.com:8000`;
    try {
        const response1 = await request({
            url: 'https://api.apify.com/v2/browser-info',
            proxyUrl,
        });
        const response2 = await request({
            url: 'https://api.apify.com/v2/browser-info',
            proxyUrl,
        });
        console.log(response1.clientIp);
        console.log('should be the same as');
        console.log(response2.clientIp);
    } catch (e) {
        console.error(e);
    }
}
main();

PHP Examples

The following section contain several examples of how to use Apify Proxy in PHP.

Usage with CURL

IMPORTANT: For all examples in this section you need to have cURL extension enabled in your PHP installation. See installation instructions for more information.

Single request with random IP chosen from all available countries.

<?php
$curl = curl_init('https://api.apify.com/v2/browser-info');
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_PROXY, 'http://proxy.apify.com:8000');
// Replace <YOUR_PROXY_PASSWORD> below with your password
// found at https://my.apify.com/proxy
curl_setopt($curl, CURLOPT_PROXYUSERPWD, 'groups-RESIDENTIAL:<YOUR_PROXY_PASSWORD>');
$response = curl_exec($curl);
curl_close($curl);
if ($response) echo $response;
?>

Two requests with the same session with IP geolocated in Australia.

<?php
function doRequest() {
    $curl = curl_init('https://api.apify.com/v2/browser-info');
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($curl, CURLOPT_PROXY, 'http://proxy.apify.com:8000');
    // Replace <YOUR_PROXY_PASSWORD> below with your password
    // found at https://my.apify.com/proxy
    curl_setopt(
        $curl,
        CURLOPT_PROXYUSERPWD,
        'groups-RESIDENTIAL,session-my_session_4,country-AU:<YOUR_PROXY_PASSWORD>'
    );
    $response = curl_exec($curl);
    curl_close($curl);
    return $response;
}
$response1 = doRequest();
$response2 = doRequest();
echo $response1;
echo "\nShould be contain same clientIp as\n";
echo $response2;
?>

Two requests with the different IPs both from the US.

<?php
function doRequest() {
    $curl = curl_init('https://api.apify.com/v2/browser-info');
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($curl, CURLOPT_PROXY, 'http://proxy.apify.com:8000');
    // Replace <YOUR_PROXY_PASSWORD> below with your password
    // found at https://my.apify.com/proxy
    curl_setopt($curl, CURLOPT_PROXYUSERPWD, 'groups-RESIDENTIAL,country-USA:<YOUR_PROXY_PASSWORD>');
    $response = curl_exec($curl);
    curl_close($curl);
    return $response;
}
$response1 = doRequest();
$response2 = doRequest();
echo $response1;
echo "\nShould have different clientIp than\n";
echo $response2;
?>

Python Examples

The following section contain several examples of how to use Apify Proxy in python.

Usage with Python 3+

Single request with random IP chosen from all available countries.

import urllib.request as request
import ssl
# Replace <YOUR_PROXY_PASSWORD> below with your password
# found at https://my.apify.com/proxy
password = '<YOUR_PROXY_PASSWORD>'
proxy_url = f"http://groups-RESIDENTIAL:{password}@proxy.apify.com:8000"
proxy_handler = request.ProxyHandler({
    'http': proxy_url,
    'https': proxy_url,
})

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
httpHandler = request.HTTPSHandler(context=ctx)

opener = request.build_opener(httpHandler,proxy_handler)
print(opener.open('https://api.apify.com/v2/browser-info').read())

Two requests with the same IP geolocated in Japan.

import urllib.request as request
import ssl

def do_request():
    # Replace <YOUR_PROXY_PASSWORD> below with your password
    # found at https://my.apify.com/proxy
    password = '<YOUR_PROXY_PASSWORD>'
    username = f"groups-RESIDENTIAL,session-my_session_6,country-JP"
    proxy_url = f"http://{username}:{password}@proxy.apify.com:8000"
    proxy_handler = request.ProxyHandler({
        'http': proxy_url,
        'https': proxy_url,
    })

    ctx = ssl.create_default_context()
    ctx.check_hostname = False
    ctx.verify_mode = ssl.CERT_NONE
    httpHandler = request.HTTPSHandler(context=ctx)

    opener = request.build_opener(httpHandler,proxy_handler)
    return opener.open('https://api.apify.com/v2/browser-info').read()

print(do_request())
print('Should contain the same clientIp as')
print(do_request())

Two requests with the different IPs from the Czech Republic.

import urllib.request as request
import ssl

def do_request():
    # Replace <YOUR_PROXY_PASSWORD> below with your password
    # found at https://my.apify.com/proxy
    password = '<YOUR_PROXY_PASSWORD>'
    username = f"groups-RESIDENTIAL,country-CZ"
    proxy_url = f"http://{username}:{password}@proxy.apify.com:8000"
    proxy_handler = request.ProxyHandler({
        'http': proxy_url,
        'https': proxy_url,
    })

    ctx = ssl.create_default_context()
    ctx.check_hostname = False
    ctx.verify_mode = ssl.CERT_NONE
    httpHandler = request.HTTPSHandler(context=ctx)

    opener = request.build_opener(httpHandler,proxy_handler)
    return opener.open('https://api.apify.com/v2/browser-info').read()

print(do_request())
print('Should contain different clientIp than ')
print(do_request())

Usage with Python 2+

IMPORTANT: For all examples in this section you need to have six enabled in your Python installation. Run pip install six to enable it.

Single request with random IP chosen from all available proxy groups.

import six
from six.moves.urllib import request

# Replace <YOUR_PROXY_PASSWORD> below with your password
# found at https://my.apify.com/proxy
password = '<YOUR_PROXY_PASSWORD>'
proxy_url = (
    'http://groups-RESIDENTIAL:%s@proxy.apify.com:8000' %
    (password)
)
proxy_handler = request.ProxyHandler({
    'http': proxy_url,
    'https': proxy_url,
})
opener = request.build_opener(proxy_handler)
print(opener.open('https://api.apify.com/v2/browser-info').read())

Two requests with the same session geolocated in the US.

import six
from six.moves.urllib import request
import ssl

def do_request():
    # Replace <YOUR_PROXY_PASSWORD> below with your password
    # found at https://my.apify.com/proxy
    password = '<YOUR_PROXY_PASSWORD>'
    username = 'groups-RESIDENTIAL,session-my_session_7,country-us'
    proxy_url = (
        'http://%s:%s@proxy.apify.com:8000' %
        (username, password)
    )
    proxy_handler = request.ProxyHandler({
        'http': proxy_url,
        'https': proxy_url,
    })

    ctx = ssl.create_default_context()
    ctx.check_hostname = False
    ctx.verify_mode = ssl.CERT_NONE
    httpHandler = request.HTTPSHandler(context=ctx)

    opener = request.build_opener(httpHandler,proxy_handler)
    return opener.open('https://api.apify.com/v2/browser-info').read()

print(do_request())
print('Should contain the same clientIp as ')
print(do_request())

Two requests with different IPs geolocated in the Czech Republic.

import six
from six.moves.urllib import request
import ssl

def do_request():
    # Replace <YOUR_PROXY_PASSWORD> below with your password
    # found at https://my.apify.com/proxy
    password = '<YOUR_PROXY_PASSWORD>'
    username = 'groups-RESIDENTIAL,country-CZ'
    proxy_url = (
        'http://%s:%s@proxy.apify.com:8000' %
        (username, password)
    )

    proxy_handler = request.ProxyHandler({
        'http': proxy_url,
        'https': proxy_url,
    })

    ctx = ssl.create_default_context()
    ctx.check_hostname = False
    ctx.verify_mode = ssl.CERT_NONE
    httpHandler = request.HTTPSHandler(context=ctx)

    opener = request.build_opener(httpHandler,proxy_handler)
    return opener.open('https://api.apify.com/v2/browser-info').read()

print(do_request())
print('Should contain different clientIp than ')
print(do_request())

Tips and tricks

Since Residential proxy is less predictable than datacenter proxy and is priced differently (by number of IPs vs traffic used), there are some important things that need to be considered before you use the Residential proxy in your solutions.

The traffic used by automated browsers needs to be controlled

Since Residential proxy is priced by data traffic used, it's very easy to realy quickly use up all prepaid traffic, especialy when accessing websites with large files loaded on every page.

There are two main ways to reduce the traffic use of your solutions:

When both of these options are combined, the data traffic of your solution can be reduced dramatically.

The speed of connected proxy can vary greatly

Each host on the Residential proxy network uses a different device, has different network speed and different latencies. This means that request made with one session can be extremely fast, while another request with different session can be extremely slow. The difference can be from a few milliseconds to a few seconds.

If your solution requires quickly loaded content, the best option is to set session, try a small request and see if the response time is acceptable. If it is, you can use this session for other requests. Otherwise, repeat the attempt with a different session.
Just remember that sessions are not persistent, so from time to time, you will have to reevaluate the speed of the session and switch to a different one if needed.

Connections can be interrupted

For this problem there is no easy solution, so either do not use the Residential proxy for larger requests or if you have no choice, expect that interruptions might happen and write your solution with this situation in mind.

Google SERP proxy

If you need to get search results from Google Search or other google search powered services from multiple countries with an option to dynamically switch between countries then you can use Google SERP proxy.

Requests made through the proxy are automatically routed through a proxy server from the selected country and pure HTML code of the search result page is returned.

Currently supported google search services are:

  • Google Search (http://www.google.<country domain>/search)
  • Google Shopping (http://www.google.<country domain>/search?tbm=shop)

Important: Only HTTP requests are allowed, and the Google hostname needs to start with www. subdomain.

Pricing is based on the number of requests made. Please contact us if you want to use Google SERP Proxy or if you need more information.

Username parameters

HTTP proxy username is used to pass various parameters for the proxy connection.

In the case of Google SERP proxy, the username should always look like this

groups-GOOGLE_SERP

Unlike datacenter or residential proxies, there is no session parameter.

Country selection

A correct google domain needs to be used to get results for the desired country code.
For example:

Search results from the US: http://www.google.com/search?q=<query>

Shopping results from Great Britain: http://www.google.co.uk/seach?tbm=shop&q=<query>

Search results from Germany: http://www.google.de/search?q=<query>

See full list of available domain names for specific countries. When used, always remember to prepend the domain name with www. prefix.

NodeJS Examples

The following sections contain several examples of how to use Google SERP proxy in actors.

Usage in PuppeteerCrawler

Get a list of results from the US for keyword wikipedia and parse them through cheerio

const Apify = require('apify');

Apify.main(async() => {
    const password = process.env.APIFY_PROXY_PASSWORD;
    const proxyUrl = `http://groups-GOOGLE_SERP:${password}@localhost:8000`;
    const url = 'http://www.google.com/search?q=wikipedia';

    const requestList = new Apify.RequestList({
        sources: [{ url }],
    });
    await requestList.initialize(); // Load requests.

    const crawler = new Apify.PuppeteerCrawler({
        requestList,
        launchPuppeteerOptions: {
            headless: true,
            proxyUrl,
        },
        gotoFunction: async({ page, request }) => {
            await page.setRequestInterception(true);
            page.on('request', request => {
                if (request.resourceType() !== 'document') return request.abort();
                return request.continue();
            });
            return page.goto(url, { waitUntil: 'domcontentloaded' });
        },
        handlePageFunction: async({ page, request }) => {
            await Apify.utils.puppeteer.injectJQuery(page);
            const searchResults = await page.evaluate(() => {
                return $('[class$="list-result"] div div:nth-child(2) ').map(function() {
                    const title = $(this).find('a[jsaction="spop.c"]')[0];
                    return {
                        title: $(title).text(),
                        url: $(title).attr('href'),
                        price: $(this).find('div:nth-child(2)').text(),
                    }
                }).toArray();
            });
            console.log(searchResults);
        },
        handleFailedRequestFunction: async({ request }) => {
            console.error(request.errorMessages);
        },
    });

    await crawler.run();
});

Get a list of shopping results from the Czech Republic for keyword Apple iPhone XS 64GB and parse them through cheerio

const Apify = require('apify');

Apify.main(async() => {
    const password = process.env.APIFY_PROXY_PASSWORD;
    const proxyUrl = `http://groups-GOOGLE_SERP:${password}@localhost:8000`;
    const url = 'http://www.google.cz/search?q=Apple+iPhone+XS+64GB&tbm=shop';

    const requestList = new Apify.RequestList({
        sources: [{ url }],
    });
    await requestList.initialize(); // Load requests.

    const crawler = new Apify.PuppeteerCrawler({
        requestList,
        launchPuppeteerOptions: {
            headless: true,
            proxyUrl,
        },
        gotoFunction: async({ page, request }) => {
            await page.setRequestInterception(true);
            page.on('request', request => {
                if (request.resourceType() !== 'document') return request.abort();
                return request.continue();
            });
            return page.goto(url, { waitUntil: 'domcontentloaded' });
        },
        handlePageFunction: async({ page, request }) => {
            await Apify.utils.puppeteer.injectJQuery(page);
            const searchResults = await page.evaluate(() => {
                return $('[class$="list-result"] > div > div:nth-child(2)').map(function() {
                    const title = $(this).find('a[jsaction="spop.c"]')[0];
                    return {
                        title: $(title).text(),
                        price: $(this).find('div:nth-child(2) > div:nth-child(1)').text(),
                    }
                }).toArray();
            });
            console.log(searchResults);
        },
        handleFailedRequestFunction: async({ request }) => {
            console.error(request.errorMessages);
        },
    });

    await crawler.run();
});

Usage in Apify.launchPuppeteer()

Get a list of results from the US for keyword wikipedia and parse them through cheerio

const Apify = require('apify');

Apify.main(async() => {
    const password = process.env.APIFY_PROXY_PASSWORD;
    const proxyUrl = `http://groups-GOOGLE_SERP:${password}@localhost:8000`;
    const url = 'http://www.google.com/search?q=wikipedia';

    try {
        const browser = await Apify.launchPuppeteer({
            headless: true,
            proxyUrl,
        });
        const page = await browser.newPage();
        await page.setRequestInterception(true);
        page.on('request', request => {
            if (request.resourceType() !== 'document') return request.abort();
            return request.continue();
        });
        await page.goto(url, { waitUntil: 'domcontentloaded' });
        await Apify.utils.puppeteer.injectJQuery(page);
        const searchResults = await page.evaluate(() => {
            return $('[class$="list-result"] div div:nth-child(2) ').map(function() {
                const title = $(this).find('a[jsaction="spop.c"]')[0];
                return {
                    title: $(title).text(),
                    url: $(title).attr('href'),
                    price: $(this).find('div:nth-child(2)').text(),
                }
            }).toArray();
        });
        console.log(searchResults);
        await page.close();
        await browser.close();
    } catch (error) {
        console.error(error.message());
    }
});

Get a list of shopping results from the Czech Republic for keyword Apple iPhone XS 64GB and parse them through cheerio

const Apify = require('apify');

Apify.main(async() => {
    const password = process.env.APIFY_PROXY_PASSWORD;
    const proxyUrl = `http://groups-GOOGLE_SERP:${password}@localhost:8000`;
    const url = 'http://www.google.cz/search?q=Apple+iPhone+XS+64GB&tbm=shop';

    try {
        const browser = await Apify.launchPuppeteer({
            headless: true,
            proxyUrl,
        });
        const page = await browser.newPage();
        await page.setRequestInterception(true);
        page.on('request', request => {
            if (request.resourceType() !== 'document') return request.abort();
            return request.continue();
        });
        await page.goto(url, { waitUntil: 'domcontentloaded' });
        await Apify.utils.puppeteer.injectJQuery(page);
        const searchResults = await page.evaluate(() => {
            return $('[class$="list-result"] > div > div:nth-child(2)').map(function() {
                const title = $(this).find('a[jsaction="spop.c"]')[0];
                return {
                    title: $(title).text(),
                    price: $(this).find('div:nth-child(2) > div:nth-child(1)').text(),
                }
            }).toArray();
        });
        console.log(searchResults);
        await page.close();
        await browser.close();
    } catch (error) {
        console.error(error.message());
    }
});

Usage with request-promise and cheerio NPM packages

Get a list of results from the US for keyword wikipedia and parse them through cheerio

const request = require('request-promise');
const cheerio = require('cheerio');

async function main() {
    const password = process.env.APIFY_PROXY_PASSWORD;
    const proxyUrl = `http://groups-GOOGLE_SERP:${password}>@proxy.apify.com:8000`;
    const html = await request({ url: 'http://www.google.com/search?q=wikipedia', proxy: proxyUrl });
    const $ = cheerio.load(html);
    const searchResults = $('#search div.g').map(function() {
        return {
            title: $($(this).find('h3')[0]).text(),
            url: $(this).find('div cite').text(),
            description: $(this).find('.s div .st').text(),
        }
    }).toArray();
    console.log(searchResults);
}

main();

Get a list of shopping results from the Czech Republic for query Apple iPhone XS 64GB and parse them through cheerio

const request = require('request-promise');
const cheerio = require('cheerio');

async function main() {
    const password = process.env.APIFY_PROXY_PASSWORD;
    const proxyUrl = `http://groups-GOOGLE_SERP:${password}>@proxy.apify.com:8000`;
    const query = encodeURI('Apple iPhone XS 64GB');

    try {
        const html = await request({
            url: `http://www.google.cz/search?tbm=shop&q=${query}`,
            proxy: proxyUrl,
        });
        const $ = cheerio.load(html);
        const searchResults = $('[class$="list-result"] > div > div:nth-child(2) ').map(function() {
            const title = $(this).find('a[jsaction="spop.c"]')[0];
            return {
                title: $(title).text(),
                price: $(this).find('div:nth-child(2)').text(),
            }
        }).toArray();
        console.log(searchResults);
    } catch (error) {
        console.error(error.message)
    }
}

main();

PHP Examples

The following section contain several examples of how to use Google SERP proxy in PHP.

Usage with CURL

IMPORTANT: For all examples in this section you need to have cURL extension enabled in your PHP installation. See installation instructions for more information.

Get HTML of search results from the US for keyword wikipedia

<?php
$query = urlencode('wikipedia');
$curl = curl_init('http://www.google.com/search?q=' . $query);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_PROXY, 'http://proxy.apify.com:8000');
// Replace <YOUR_PROXY_PASSWORD> below with your password
// found at https://my.apify.com/proxy
curl_setopt($curl, CURLOPT_PROXYUSERPWD, 'groups-GOOGLE_SERP:<YOUR_PROXY_PASSWORD>');
$response = curl_exec($curl);
curl_close($curl);
echo $response;
?>

Get HTML of shopping results from the Czech Republic for query Apple iPhone XS 64GB

<?php
$query = urlencode('Apple iPhone XS 64GB');
$curl = curl_init('http://www.google.com/search?tbm=shop&q=' . $query);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_PROXY, 'http://proxy.apify.com:8000');
// Replace <YOUR_PROXY_PASSWORD> below with your password
// found at https://my.apify.com/proxy
curl_setopt($curl, CURLOPT_PROXYUSERPWD, 'groups-GOOGLE_SERP:<YOUR_PROXY_PASSWORD>');
$response = curl_exec($curl);
curl_close($curl);
echo $response;
?>

Python Examples

The following section contain several examples of how to use Google SERP proxy in python.

Usage with Python 3+

Get HTML of search results from the US for keyword wikipedia

import urllib.request as request
import urllib.parse as parse

# Replace <YOUR_PROXY_PASSWORD> below with your password
# found at https://my.apify.com/proxy
password = '<YOUR_PROXY_PASSWORD>'
proxy_url = f"http://groups-GOOGLE_SERP:{password}@proxy.apify.com:8000"

proxy_handler = request.ProxyHandler({
    'http': proxy_url,
})

opener = request.build_opener(proxy_handler)

query = parse.urlencode({ 'q': 'wikipedia' })
print(opener.open(f"http://www.google.com/search?{query}").read())

Get HTML of shopping results from the Czech Republic for query Apple iPhone XS 64GB

import urllib.request as request
import urllib.parse as parse

# Replace <YOUR_PROXY_PASSWORD> below with your password
# found at https://my.apify.com/proxy
password = '<YOUR_PROXY_PASSWORD>'
proxy_url = f"http://groups-GOOGLE_SERP:{password}@proxy.apify.com:8000"
proxy_handler = request.ProxyHandler({
    'http': proxy_url,
})
opener = request.build_opener(proxy_handler)

query = parse.urlencode({ 'q': 'Apple iPhone XS 64GB', 'tbm': 'shop' })
print(opener.open(f"http://www.google.cz/search?{query}").read())

Usage with Python 2+

IMPORTANT: For all examples in this section you need to have six enabled in your Python installation. Run pip install six to enable it.

Get HTML of search results from the US for keyword wikipedia

import six
from six.moves.urllib import request, urlencode

# Replace <YOUR_PROXY_PASSWORD> below with your password
# found at https://my.apify.com/proxy
password = '<YOUR_PROXY_PASSWORD>'
proxy_url = (
    'http://groups-GOOGLE_SERP:%s@proxy.apify.com:8000' %
    (password)
)
proxy_handler = request.ProxyHandler({
    'http': proxy_url,
})
opener = request.build_opener(proxy_handler)
query = parse.urlencode({ 'q': 'wikipedia' })
url = (
    'http://www.google.com/search?%s' %
    (query)
)
print(opener.open(url).read())

Get HTML of shopping results from the Czech Republic for query Apple iPhone XS 64GB

import six
from six.moves.urllib import request, urlencode

# Replace <YOUR_PROXY_PASSWORD> below with your password
# found at https://my.apify.com/proxy
password = '<YOUR_PROXY_PASSWORD>'
proxy_url = (
    'http://groups-GOOGLE_SERP:%s@proxy.apify.com:8000' %
    (password)
)
proxy_handler = request.ProxyHandler({
    'http': proxy_url,
})
opener = request.build_opener(proxy_handler)
query = parse.urlencode({ 'q': 'Apple iPhone XS 64GB', 'tbm': 'shop' })
url = (
    'http://www.google.cz/search?%s' %
    (query)
)
print(opener.open(url).read())

Troubleshooting

To view the status of the connection to Apify Proxy, open the following URL in the browser that uses the proxy:

http://proxy.apify.com/

If the proxy connection works well, the web page should look something like this:

To test that your requests are proxied and rotate the IP addresses correctly, you can open the following API endpoint via the proxy. It shows information about the client IP address:

https://api.apify.com/v2/browser-info/