Actor

vaclavrut/bigjson2csv

  • Builds
  • latest 0.1.17 / 2018-08-17
  • Created 2018-08-16
  • Last modified 2018-09-12
  • grade 1

Description

If you are looking for a solution that will convert your big JSON (array of JSON objects) to a CSV/Excel in JS, this is a droid you are looking for. Paste to the input an URL for a JSON and name of the dataset where we will store the records. You can provide more than 1 URL for the JSON, see input example. Actor is using our dataset to handle the converting the items, we just simply push items to the dataset and then download the whole dataset. Apify do the magic for us.


API

To run the actor, send a HTTP POST request to:

https://api.apify.com/v2/acts/vaclavrut~bigjson2csv/runs?token=<YOUR_API_TOKEN>

The POST payload will be passed as input for the actor. For more information, read the docs.


Example input

Content type: application/json; charset=utf-8

{
  "urls": [
    {
      "url": "https://apify-uploads-prod.s3.amazonaws.com/d90d0cac-7665-4f48-bddd-74698f54c02d_testData.json",
      "name": "sample-1"
    },
    {
      "url": "https://apify-uploads-prod.s3.amazonaws.com/d90d0cac-7665-4f48-bddd-74698f54c02d_testData.json",
      "name": "sample-2"
    }
  ]
}

Readme

Code

const Apify = require('apify');
const rp = require('request-promise')
const Promise = require('bluebird')
const ApifyClient = require('apify-client');

Apify.main(async () => {

    const input = await Apify.getValue('INPUT');
    const environmentVariables = await Apify.getEnv()

    const apifyClient = new ApifyClient({
        userId: environmentVariables.userId,
        token: environmentVariables.token
});
    const datasets = apifyClient.datasets;

    await Promise.map(input.urls, async (solve) => {

        if (!solve.name || !solve.url) throw new Error('Invalid input! Please provide combination of name and url');

        const inputData = await rp({uri: solve.url});

        const dataset = await datasets.getOrCreateDataset({
            datasetName: solve.name + environmentVariables.actRunId,
        });

        const parsedData = JSON.parse(inputData);
        console.log("Loaded", parsedData.length, " for ", solve.name);

        while (parsedData.length) {
            console.log("Remaining records for", solve.name, " is: ", parsedData.length)
            await datasets.putItems({datasetId: dataset.id,data: parsedData.splice(0, 1000)});
        }
        console.log(solve.name," finished.")
        console.log("Download a CSV : https://api.apify.com/v2/datasets/" + dataset.id + "/items?format=csv&attachment=1");
        console.log("Download a XLSX : https://api.apify.com/v2/datasets/" + dataset.id + "/items?format=xlsx&attachment=1");

    }, {concurrency: 10})

    console.log("Job finished, see you next time.");
});

Actor expects the file on the input in this structure:

[{
    "name":"Here is a name of the object 1.",
    "value1": 1,
    "value2":2,
    "end":"Thanks for watching!"
},
{
    "name":"Here is a name of the object 2.",
    "value1": 1,
    "value2":2,
    "end":"Thanks for watching!"
}]