Actor

apify/example-web-server

  • Builds
  • latest 0.0.6 / 2018-07-24
  • Created 2018-07-24
  • Last modified 2018-07-25
  • grade 1

Description

This example demonstrates how to use web server in actor as communication channel with outer world. Read article about this crawler in Apify knowledge base: https://kb.apify.com/actor/running-a-web-server


API

To run the actor, send a HTTP POST request to:

https://api.apify.com/v2/acts/apify~example-web-server/runs?token=<YOUR_API_TOKEN>

The POST payload will be passed as input for the actor. For more information, read the docs.


Example input

Content type: application/json; charset=utf-8

{ "hello": 123 }

Source code

Based on the apify/actor-node-chrome Docker image (see docs).

const Apify = require('apify');
const express = require('express')
const bodyParser = require('body-parser');

// Create ExpressJS app and configure body parsers that will allow us to receive form submissions.
const app = express()
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

//  Now we need to read following environment variables: 
// - APIFY_CONTAINER_PORT  contains a port number where we must start server 
// - APIFY_CONTAINER_URL  contains a URL under which we can access the container
// - APIFY_DEFAULT_KEY_VALUE_STORE_ID is simply ID of default key-value store of this 
//   actor where we can store screenshots
const {
    APIFY_CONTAINER_PORT,
    APIFY_CONTAINER_URL,
    APIFY_DEFAULT_KEY_VALUE_STORE_ID,
} = process.env;

// Create an array of the processed URLs where n-th URL has its screenshot stored under the 
// key [n].jpg in key-value store:
const processedUrls = [];

// Root path displays a HTML form to submit new URL and thumbnails of processed URLs:
app.get('/', (req, res) => {
    let listItems = '';

    // For each of the processed
    processedUrls.forEach((url, index) => {
        const imageUrl = `https://api.apify.com/v2/key-value-stores/${APIFY_DEFAULT_KEY_VALUE_STORE_ID}/records/${index}.jpg`;

        listItems += `<li>
    <a href="${imageUrl}" target="_blank">
        <img src="${imageUrl}" width="300px" />
        <br />
        ${url}
    </a>
</li>`;
    });

    const pageHtml = `
<html>
    <head><title>Example</title></head>
    <body>
        <form method="POST" action="${APIFY_CONTAINER_URL}/add-url">
            URL: <input type="text" name="url" placeholder="http://example.com" />
            <input type="submit" value="Add" />
            <hr />
            <ul>${listItems}</ul>
        </form>
    </body>
</html>`;

    res.send(pageHtml);
});

// POST route allowing us to submit new URLs. After URL is processed
// it redirects user back to root path.
app.post('/add-url', async (req, res) => {
    const { url } = req.body;
    console.log(`Got new URL: ${url}`);

    // Start chrome browser and open new page ...
    const browser = await Apify.launchPuppeteer();
    const page = await browser.newPage();

    // ... go to our URL and grab a screenshot ...
    await page.goto(url);
    const screenshot = await page.screenshot({ type: 'jpeg' });

    // ... close browser ...
    await page.close();
    await browser.close(); 

    // ... save sreenshot to key-value store and add URL to processedUrls.
    await Apify.setValue(`${processedUrls.length}.jpg`, screenshot, { contentType: 'image/jpeg' });
    processedUrls.push(url);

    res.redirect('/');
});

// Start the webserver.
app.listen(APIFY_CONTAINER_PORT, () => {
    console.log(`Application is listening at URL ${APIFY_CONTAINER_URL}.`);
});