Redis and Node Part 4: Lua Scripting

Multithreaded JavaScript has been published with O'Reilly!

This is the final part in this series on using Redis with Node.js. The content of these posts is partially adapted from my book, Advanced Microservices. There is also a companion presentation, Node, Redis, and You!, which I've given at several Meetups and a conference.

The nice thing about the MULTI / EXEC commands is that they allow us to atomically run commands. Unfortunately they don't allow us to take the output of one command and use it as the input for another command! Luckily for us Redis gives us the ability to load and execute Lua scripts.

These scripts allow us to execute more complex logic inside of the Redis server without having to perform this logic in our application. In many situations it just wouldn't make sense to run a Redis command, get the output, do processing in our application, run another Redis command, etc. This process is slow and can mean that the data in Redis is now in an invalid state until we're complete.

There are two main commands we're going to look at in this article. The first one is SCRIPT LOAD, which allows us to send a Lua script to Redis to have it be parsed and loaded into memory. The second command is EVALSHA, which allows us to execute our loaded scripts as well as pass arguments to said scripts. With Redis being simple and convenient to work with, we can freely run a SCRIPT LOAD multiple times with the same script and have no ill side effects. Because of this it's a normal practice to perform these loads each time a new Node instance begins.

When we perform a SCRIPT LOAD, Redis returns us a SHA1 hash to refer to the script. We can then make use of this hash to refer to and execute the script using EVALSHA. That said, we don't want to actually hash these scripts or keep track of them ourselves; that would be painful and error-prone. Instead we should always calculate these using Redis. That said, there are nice libraries we can use in Node to keep track of making these hashes for us. Personally I prefer a simple one called lured.

When executing Lua scripts it's important to pass in the names of all keys via arguments. It is technically possible to hard-code the keys inside of our Lua script, however this will lead to scaling issues down the road when we want to have a cluster of Redis instances running. We'll declaratively pass the keys in so that Redis knows how to find the data.

Lua Script

Here we'll look at a file called get-cities.lua. These files can be saved and committed into your application repository. Try to keep them in the same directory to make them easy to find. This script is fairly simple as it only executes two Redis commands.

The syntax for Lua shouldn't look too scary if you're used to working with JavaScript. When we declare a variable we use local instead of var or let or const. We have two global arrays available to us, one called KEYS and one called ARGV. KEYS is a list of the keys that we want to modify and ARGV is a list of arguments we want to programmatically work with. One notable difference from JavaScript is that Lua arrays are 1-based instead of 0-based.

When we want to call out to Redis we execute redis.call(), which is always available for communicating with Redis. The first argument is the command we execute, and the following arguments are the arguments to the command.

The first command we're going to execute is GEORADIUS, which gets us a list of GeoLocation entries based on a distance from a center point. In this case we're going to look for all cities within 10km of a supplied longitude and latitude pair. This command is executed against our GeoLocation collection which contains a list of city ID's based on their geolocation.

The second command we're going to execute is HMGET, which essentially means Hash Multi Get. This will allow us to get multiple properties from a single hash. This command is executed against a hash of cities, with the field being the same city ID from the GeoLocation, and with the value being some JSON data describing that city.

The unpack() function is similar to JavaScript's .bind() and .apply() methods. They're used for taking an array of items and converting them into arguments. Specifically the HMGET command expects multiple arguments, one for each field, so we convert the list of geolocation entries into arguments.

-- get-cities.lua: Find cities within 10km of query

local key_geo = KEYS[1]
local key_hash = KEYS[2]

local longitude = ARGV[1]
local latitude = ARGV[2]

local city_ids = redis.call('GEORADIUS', key_geo, longitude, latitude, 10, 'km')
return redis.call('HMGET', key_hash, unpack(city_ids))

Node Application

Here is our Node application code. There is a lot going on so I'll try to describe everything step by step.

The first important thing we do is build an object containing our library of Lua scripts. In this case we only have a single entry, named find, which is going to represent the above script. We load the contents of the Lua file synchronously from the filesystem (normally we want to do I/O asynchronously, but since we read the file once during application instantiation, it's acceptable). When we instantiate Lured we pass in the library as well as a Redis client instance.

Next we add some pairs of data to the GeoLocation collection as well as the Hash. In this case we're adding an entry for san-francisco as well as oakland. These IDs are also used in our Hash.

Finally we go ahead and load Lured by running lured.load(). Behind the scenes the Lured library is taking each script, passing it to Redis to be hashed, then updating the local listing of hashes. Once every script has been hashed the callback then executes.

Once we've loaded Lured we then actually go about executing our script with redis.evalsha(). The first argument is the hash of the script we want to run which has been conveniently set by Lured. After that is an interesting part where we provide a number. This number tells Redis that the next X number of arguments will represent the names of KEYS. Afterwards, any remaining arguments will be treated as ARGV. This is how we declaratively tell Redis what the keys are. Finally we provide our callback.

const redis = require('redis').createClient();
const fs = require('fs');
const GEO = 'geo-city-locations', HASH = 'hash-city-data';
let lua = {
  find: {
    script: fs.readFileSync(`${__dirname}/get-cities.lua`, {encoding: 'utf8'}),
    sha: null // This is set by lured.load()
  }
};
const lured = require('lured').create(redis, lua);

redis.geoadd(GEO, -122.419103, 37.777068, 'san-francisco');
redis.hset(HASH, 'san-francisco', JSON.stringify({name: 'San Francisco', temp: 65}));

redis.geoadd(GEO, -122.272938, 37.807235, 'oakland');
redis.hset(HASH, 'oakland', JSON.stringify({name: 'Oakland', temp: 72}));

const BERKELEY = {lon: -122.273412, lat: 37.869103};
lured.load((err) => {
  redis.evalsha(lua.find.sha, 2, GEO, HASH, BERKELEY.lon, BERKELEY.lat, (e, data) => {
    console.log('cities near Berkeley', data.map(JSON.parse));
    // [ {name:"Oakland",temp:72} ]
  });
});

When we execute this script we get a list of cities within 10 miles of Berkeley, which in this case is simply Oakland.

Lua is of course far more powerful than this simple example used in this Article. Check out the Redis Lua documentation if you're interested in how powerful it is. There are a few Lua modules which Redis always ships with, such as the ability to work with JSON.


That's it for the series! If you missed any of the previous parts you can check out the related links below which will link to them. And of course, if you found this content useful, please checkout my book Advanced Microservices.

Thomas Hunter II Avatar

Thomas has contributed to dozens of enterprise Node.js services and has worked for a company dedicated to securing Node.js. He has spoken at several conferences on Node.js and JavaScript and is an O'Reilly published author.