Using gulp to compile and parse a bunch of JSON files.

Gulp is a great tool.  What is gulp exactly? Well it’s a node.js task runner, that is really handy for doing a lot of things that you might otherwise do with a shell script or maybe python.  Build and compile a bunch of stuff, compress and uglify javascript and css.  So many things that you can do with Gulp, that I’m making an effort over time (slowly) to replace my scattered install.sh and setup.py scripts with gulpfile.js.

So recently I had an idea, where the problem was getting a bunch of individual JSON structured files that represent complete projects for the Atmosphere IoT Ecosystem that I work on, from it’s GitHub demo project repository and creating an index file of all those projects, stripped of all the unnecessary keys.  The files were sorted in multiple directories for human readable consumption so the structure looks a bit like this:

/{platform}/{state}/{filename}.atmo 

which could be for example:

/intelCurie/official/ActivityTracker.atmo

Note: an .atmo extension is really just a cute replacement for .json that fits with the development platform name.

My goal is to sift through hundreds of these .atmo files, grab only the ones in the **/official/ folders, weed out ones that fail a few specifications and then build an index of official demos that can be loaded into a JQuery Bootgrid, which will be implemented as a demo selector within the Atmosphere IDE.  Gulp is up for the task, and in fewer lines than I expected.

Another Note: You’ll need to have installed node.js to use gulp. I’ll also assume you’ve installed Gulp at this point too. So the next thing to do is create a gulpfile.js and put in the following.

// We need gulp!
var gulp = require('gulp');

// and this gulp tool which concatenates a bunch of json into 1 json, how cool?
var jsonConcat = require('gulp-json-concat');

Now that the requirements are out of the way, time to write some code.  We’ll want to setup a gulp task that will use jsonConcat to return an index of all our .atmo’s that are just in the ‘official’ directories, and we’re also going only add want the following keys from those JSON files; version, projectType, description, and name.  So we’ll use the callback function of jsonConcat to pass only the keys we want to the output buffer like this:

gulp.task('build', function() {
    
    return gulp.src('**/official/*.atmo')
        .pipe(jsonConcat('demos.json', function(data){
            
            var newData = [];
            
            for (d in data) {                           
                newData.push({version: data[d].header.meta.version,
                    description: data[d].header.meta.description,
                    projectType: data[d].header.meta.projectType,
                    name: data[d].header.meta.name
                });
            }
            
            return new Buffer(JSON.stringify(newData));
            
        }))
        .pipe(gulp.dest('./'));
    
});

Pretty simple, this gives us a single JSON file that produces an index of all the demos.  From here however we might want to refine it some more.. one thing we are missing a reference to the actual filename of the .atmo from each demo. That could be useful, so we’ll slip that into the for…loop portion to add an a new key to our index.

for (d in data) {                           
    newData.push({version: data[d].header.meta.version,
        description: data[d].header.meta.description,
        projectType: data[d].header.meta.projectType,
        name: data[d].header.meta.name,
        file: d +".atmo"
    });
}

Interestingly, the object keys of ‘data’ from jsonConcat was **/official/* but dropped the .atmo extension, this was unexpected behavior so in the file key you’ll notice I had to put d + “.atmo”, and now the index has a reference to the actual file location.  The result set of data will end up looking something like this:

[{
    "version": "2.0.1",
    "description": "This example is a simple test of the ADC Read function",
    "projectType": "bcm20737",
    "name": "AdcTest",
    "file": "bcm20737/official/ADCReadExample.atmo"
}, {
    "version": "2.0.1",
    "description": "This example shows how to read an analog voltage using the module's Analog to Digital Converter (ADC).",
    "projectType": "bcm20737",
    "name": "ADC Example",
    "file": "bcm20737/official/ADC_Example.atmo"
}]

While you might not find this particular case useful for whatever you’re working on, I promise you that Gulp has the tools you need for many other tasks.  Along with things like a GitHub push event notification and a listener on your server, you can use Gulp in a toolset for a whole lot of automation, from testing, to builds to deployment.

Leave a Reply

Your email address will not be published. Required fields are marked *