Refactoring a Node.js codebase using Async/Await

Multithreaded JavaScript has been published with O'Reilly!

This week I refactored a Node project of mine from the traditional Node callback pattern to the recently available Async/Await pattern. This pattern is available without transpiling code as of Node v8.3. In this post I'll walk you through a small example of this conversion.

This particular project is by no means large; it consists of 1,714 lines of JavaScript spread out across 20 files. The project is first and foremost a web server, though it performs some scheduled work as well. Data is persisted in Redis (it's not just a cache) and PostgreSQL. The project is organized into controllers, models, middleware, and services for the scheduled work. Models touch Redis and PostgreSQL and do business logic and never see a request or response object. Controllers are thin, performing validation and transforming HTTP input. The HTTP framework is Hapi.

Before the Refactor

Here's an example of a complex model operation: logging in. This function accepts a username and a password and a callback. The first thing we do is see if either the username or password is empty, if so immediately fail. Then we retrieve user data based on the username. Afterwards we use bcrypt to encrypt the password and then compare the values. Next we cache this operation in Redis for quick per-request lookups. Once all that asynchronous stuff is done we reply with an object which the client will use for auth. Here's the old login model function:

const login = ({username, password}, callback) => {
  if (!username || !password) return setImmediate(() => {
    callback(new Error('missing username or password'));
  });

  postgres.query(FIND, [username], (err, data) => {
    if (err) return callback(err);
    if (!data || !data.rows || !data.rows[0]) return callback(new Error('no user'));

    let encrypted_password = data.rows[0].password;
    let user_id = data.rows[0].id;

    bcrypt.compare(password, encrypted_password, (err, res) => {
      if (err) return callback(err);
      if (!res) return callback(new Error('invalid password'));

      let auth_code = randomstring.generate(12);

      redis.hset(keys.auth, user_id, auth_code, err => {
        if (err) return callback(err);

        let encoded = new Buffer(`${user_id}:${auth_code}`).toString('base64');

        callback(null, {
          user_id,
          auth_code,
          authorization_raw: `${user_id}:${auth_code}`,
          authorization_header: `Basic ${encoded}`
        });
      });
    });
  });
};

Of course, I've got a bit of callback hell going on, though I could have made a named function for each one of those asynchronous operations to prevent it. Anyway, that's three different asynchronous operations required before we're done. In this case the operations are rather linear; I first want to see if the user exists in the database. Next I want to see if the hashes match. Finally I want to update Redis if everything is OK. If at any time one of these operations fails I want to abort future work.

It's worth pointing out that there are 5 functions defined in this code example. In our everyday interactions with JavaScript we probably don't pay much attention to writing functions, even more now that we have the arrow function syntax. Each time we write a function we introduce additional complexity and overhead to our program.

We also need to take a look at the controller for this file to get a better feel for what's going on. The Async/Await features are most interesting in how they allow shallow layers of your application to work with deeper layers. As you might have guessed the controller is pretty simple:

server.route({
  method: 'PUT',
  path: '/v1/auth/login',
  handler: (request, reply) => {
    let username = String(request.payload.username);
    let password = String(request.payload.password);

    model.login({username, password}, (error, result) => {
      if (error) return reply({error: error.message}).code(400);

      reply(result).code(202);
    });
  }
});

This code consists of a poor-mans validation and type coercion (you should use Joi for that), pass the values into the login model, and decide if we return a sad 400 or a happy 202.

After the Refactor

When converting code to use Async/Await, the first thing to keep in mind is that we need to incorporate Promises. Lucky for me, 2/3 of these libraries overload their methods to work with callbacks or as Promises, notably the bcrypt and pg modules. redis, however, does not. It is quite easy to implement though; if we use the library pifall then we can add a few lines of boilerplate when we instantiate our Redis singleton:

const Redis = require('redis');
const pifall = require('pifall');

pifall(Redis.RedisClient.prototype);
pifall(Redis.Multi.prototype);

module.exports = Redis.createClient({});

Afterwards there will be an equivalent *Async promise method available for every method exposed in the base object. As an example, the methods .get(key, cb) and .hget(key, field, cb) now have sibling Promise methods named .getAsync(key) and .hgetAsync(key, field).

Let's take a look at what this model function looks like after converting it to Async/Await:

const login = async ({username, password}) => {
  if (!username || !password) throw new Error('missing username or password');

  let result = await postgres.query(FIND, [username]);

  if (!result || !result.rows) throw new Error('no_data');

  let user = result.rows[0];

  if (!user) throw new Error('no_user');

  let encrypted_password = user.password;
  let user_id = user.id;

  let comparison = await bcrypt.compare(password, encrypted_password);

  if (!comparison) throw new Error('invalid password');

  let auth_code = randomstring.generate(12);

  await redis.hsetAsync(keys.auth, user_id, auth_code);

  let encoded = Buffer.from(`${user_id}:${auth_code}`).toString('base64');

  return {
    user_id,
    auth_code,
    authorization_raw: `${user_id}:${auth_code}`,
    authorization_header: `Basic ${encoded}`
  };
};

Doesn't that look so much nicer? In this case, there is only 1 function defined in the entire file! One thing to point out is that the error handling here is very simple. In the first, callback pattern code example, you can see that each time an error occurs we simply halt our callbacks. We can see this because each one of the asynchronous operations immediately contains if (err) return callback(err);. If we had more complex things happening, we would need to make use of try/catch blocks. Of course, the controller has evolved a little bit too, and does include a try/catch:

server.route({
  method: 'PUT',
  path: '/v1/auth/login',
  handler: async (request, reply) => {
    let username = String(request.payload.username);
    let password = String(request.payload.password);

    try {
      var result = await model.login({username, password});
    } catch (error) {
      return reply({error: error.message}).code(400);
    }

    reply(result).code(202);
  }
});

It's worth pointing out that Hapi is playing nicely with our async function here. Specifically, Hapi is compatible with request handlers which use either a callback or return a promise. We can always swap out a promise function for an Async function (as long as our JavaScript implementation supports it, e.g. Node v8.3+) and this will work as expected. This will make it very very easy to refactor a codebase from using Promises with .then() chains to using Async/Await.

More Examples

Here's a small playground of scripts which show just how easy it is to intermingle the two:

const sleep = (sec) => {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve();
    }, sec * 1000);
  });
};

const asyncSleepWrapper = async (sec) => {
  console.log('async fn before sleep');
  await sleep(sec);
  console.log('async fn after sleep');
  return 42;
};

(async () => {
  console.log('before sleep');
  await sleep(1);
  console.log('after sleep');
})();

console.log('before sleep');
sleep(1)
  .then(() => { // ugly promise zigzag's
    console.log('after sleep');
  });

// Async functions can be used where promises can be used
asyncSleepWrapper(3)
  .then(res => { // ugly promise zigzag's
    console.log(res);
  })
  .then(() => { // ugly promise zigzag's
    console.log('async can be used where promises are used');
  });

Each of these examples so far have been doing sequential work. Here's a quick example on how to do work in parallel by using the built-in Promise.all() method. This method accepts an array of promises and will ultimately return a promise which resolves once the last bit of work has completed. The value of this resolved promise will be an array of values in the same order.

(async () => {
  let start = Date.now();
  console.log('about to do 2s and 3s things in parallel...');
  let result = await Promise.all([
    sleep(2),
    sleep(3)
  ]);

  console.log('time taken:', (Date.now() - start) / 1000); // about 3
})();

Parting Thoughts

After the refactor it was 1,585 lines, a drop in about 7.5% LoC. Personally the greatest benefit of switching projects to Async/Await is the readability goes through the roof! Our code looks much closer to the more classical, synchronous method for calling functions and assigning their results to a variable. I am confident that the presence of Async/Await will forever change the way we write JavaScript.

Some people refer to callback hell as the visually nested callbacks which continually increase indentation. Certainly this is ugly. But another way to think of them is that we aren't as much creating functions for their traditional purpose, which is to create named reusable lines of code. Instead we create anonymous callbacks with the intent of using them for asynchronous flow control. Async/Await allows us to stop writing functions for that purpose.

Chained .then() calls with Promises is really just the same thing as each .then() requires a function, unnamed or otherwise. Promises, as they've been used for the past few years, were only an incremental change from the older Callback style used in Node. The Async/Await construct offer us a new language syntax and a revolutionary new way to perform asynchronous flow control.

If you're interested in a short history lesson of these paradigms, check out my previous blog post: The long road to Async/Await in JavaScript

Thomas Hunter II Avatar

Thomas has contributed to dozens of enterprise Node.js services and has worked for a company dedicated to securing Node.js. He has spoken at several conferences on Node.js and JavaScript and is an O'Reilly published author.