Asset pipelines as a microservice
Over the past 10 years, delivering web assets has become increasingly more sophisticated. In the past, we might have managed this by hand, perhaps generating combined files or using tools/scripts by hand and keeping them in sync. This has then transitioned to part of the build process and there are tools especially built for this.
The most sophisticated tools for this are currently JavaScript-based, notionally around Gulp and Grunt. As part of our build process, we've standardised on Gulp to preprocess CSS (using Less) and autoprefix the output, to run Webpack on our JavaScript and build a PWA service worker with pre-cached assets, as well as do the heavy lifting around sourcemaps, minification, fingerprinting etc.
As part of a build process, this works fine. We use our CI server either run gulp by hand, or use gradle plugins that install node and run gulp through that. But what if assets are created and modified directly in our application, such as creating and editing .less files directly in our CMS?
In the past, we looked at doing the preprocessing directly in our (Java-based) CMS, using Rhino to run less.js against the input and storing the output, but that was imperfect - bringing a whole JavaScript engine into our CMS for one task seemed a bit wasteful, and this was just one part of the process; it took a lot of engineering to run less through Rhino, and this didn't get us all the other niceties that our Gulpfiles were able to.
Running a node microservice
The solution we came to was to use the same modules as in our Gulp pipeline, but as npm modules in a node application which would listen over HTTP. We're using Express as our HTTP server in the example here, but it could just easily be rewritten for anything else, it's not doing anything clever on that side of things.
We'll start out with something that just takes the input and echoes it back out:
# A directory for our project
$ mkdir asset-pipeline && cd asset-pipeline
# Create package.json with defaults
$ npm init -y
# Install express and body-parser
$ npm i express body-parser
I'm using a node ES module here but it could just as easily be written as a normal CommonJS module.
index.mjs:
import express from 'express';
import bodyParser from 'body-parser';
// Create an Express application
const app = express();
// Listen for HTTP POSTs at the root
app.post('/',
// Parse requests of Content-Type text/less and expose as req.body
bodyParser.text({ 'type': 'text/less' }),
(req, res, next) => {
const input = req.body.toString();
// Set the output content type and echo input
res.set('Content-Type', 'text/css')
.send(input)
.end();
}
);
// Listen on passed PORT or 3000 if none passed
app.listen(process.env.PORT || 3000);
We can test this out from the command line and make sure that, for the moment, it just spits out what we put in:
# Run the server
$ node --experimental-modules index.mjs
# Ctrl-C to exit. You'll need to exit and restart after making changes
# (In another terminal)
$ curl -H'Content-Type: text/less' --data 'body { background: contrast(black); }' 'http://localhost:3000'
body { background: contrast(black); }
All good so far, but not very useful. Let's install the less module, and use that to process the input and output CSS:
$ npm i less
import express from 'express';
import bodyParser from 'body-parser';
import less from 'less';
// Create an Express application
const app = express();
// Listen for HTTP POSTs at the root
app.post('/',
// Parse requests of Content-Type text/less and expose as req.body
bodyParser.text({ 'type': 'text/less' }),
(req, res, next) => {
const input = req.body.toString();
less.render(input, {
// Get the X-Filename header or use input.less (for error messages)
filename: req.get('x-filename') || 'input.less',
}).then((output) => {
// Set the output content type and output transformed CSS
res.set('Content-Type', 'text/css')
.send(output.css)
.end();
}).catch((err) => {
// Output any errors in the input as a Bad Request, as JSON
res.status(400).json(err).end();
});
}
);
// Listen on passed PORT or 3000 if none passed
app.listen(process.env.PORT || 3000);
Now, when we test, we can see our Less is transformed into CSS, and any errors are returned as JSON and can be parsed out to be displayed to the user:
$ curl -H'Content-Type: text/less' --data 'body { background: contrast(black); }' 'http://localhost:3000'
body {
background: #ffffff;
}
$ curl -H'Content-Type: text/less' --data 'body { background: }' 'http://localhost:3000'
{"message":"Unrecognised input. Possibly missing opening '{'","type":"Parse","filename":"input.less","index":19,"line":1,"column":19,"callLine":null,"extract":[null,"body { background: }",null]}
At the University of Warwick, we publish our web corporate identity so it can be installed as an npm module. This is useful because it means we can reference the less variables and mixins in our input - we'll direct the less renderer to look in the node_modules directory and then we can use @import in our less input to use them.
# Install the latest version of the corporate identity as an npm module
npm i UniversityofWarwick/id7
...
const input = req.body.toString();
less.render(input, {
// Get the X-Filename header or use input.less (for error messages)
filename: req.get('x-filename') || 'input.less',
// Allow node_modules to be referenced
paths: ['node_modules'],
}).then((output) => {
...
We can then test this out (this time using a file):
input.less:
@import (reference) "id7/less/variables.less";
body {
background: @id7-brand-purple;
}
$ curl -H'Content-Type: text/less' --data @input.less 'http://localhost:3000'
body {
background: #5b3069;
}
Now we've got the base of our pipeline working, we can apply more transformations like we would as part of the build process. For example, let's use PostCSS and autoprefixer (we could just have easily have used the autoprefixer less plugin here, but let's imagine we wanted to add more postcss transformations), and postcss-clean which uses clean-css to minify:
$ npm i postcss autoprefixer postcss-clean
import express from 'express';
import bodyParser from 'body-parser';
import less from 'less';
import postcss from 'postcss';
import autoprefixer from 'autoprefixer';
import clean from 'postcss-clean';
// Create an Express application
const app = express();
// Listen for HTTP POSTs at the root
app.post('/',
// Parse requests of Content-Type text/less and expose as req.body
bodyParser.text({ 'type': 'text/less' }),
(req, res, next) => {
const input = req.body.toString();
// Get the X-Filename header or use input.less (for error messages)
const filename = req.get('x-filename') || 'input.less';
less.render(input, {
filename,
// Allow node_modules to be referenced
paths: ['node_modules'],
}).then((output) => {
postcss()
.use(autoprefixer({ browsers: ['> 1% in GB', 'last 2 versions', 'IE 9'] }))
.use(clean({ compatibility: 'ie9', sourceMap: false }))
.process(output.css, { from: filename, to: `${filename.replace('.less', '')}.css` })
.then((result) => {
// Set the output content type and output transformed CSS
res.set('Content-Type', 'text/css')
.send(result.css)
.end();
})
.catch((err) => {
// Treat PostCSS errors as internal server errors
res.status(500).json(err).end();
});
}).catch((err) => {
// Output any errors in the input as a Bad Request, as JSON
res.status(400).json(err).end();
});
}
);
// Listen on passed PORT or 3000 if none passed
app.listen(process.env.PORT || 3000);
input.less:
@import (reference) "id7/less/variables.less";
body {
background: @id7-brand-purple;
}
::placeholder {
color: gray;
}
$ curl -H'Content-Type: text/less' --data @input.less 'http://localhost:3000'
body{background:#5b3069}::-webkit-input-placeholder{color:gray}:-ms-input-placeholder{color:gray}::-ms-input-placeholder{color:gray}::placeholder{color:gray}
In our app, we can then build a simple service that interacts with the web service and now we allow users to upload and edit .less files directly in our CMS as a result.