Redis and Node Part 1: The Basics
Support this website by purchasing prints of my photographs! Check them out here.This is part one of a four part series on using Redis with Node.js. The content of these posts is partially adapted from my book, Advanced Microservices. There is also a companion presentation, Node, Redis, and You!, which I've given at several Meetups and a conference.
Introduction to Redis
At its heart, Redis is an in-memory key/value store with support for several different data structures. Under the hood it stores data in fast and efficient ways and allows to to read that data quickly. It does a lot of the painful computer science whiteboard stuff so that you don't have to.
Redis has a really nice philosophy to create items if they don't exist, destroy items if they're empty, default numeric operations to being zero, etc. This makes Redis very easy to work with and removes the need to initialize values which is extremely useful when building distributed applications. You should build your applications in such a way that you never need to check for the presence of a key.
Redis, much like Node, is essentially single threaded as far as the core operations goes, though it does do some things in different threads, e.g. garbage collection and persistence, but as an application developer we don't need to know about that.
Communication with Redis is very simple. One could make a telnet connection to Redis and start sending simple ASCII commands to read and write data. Of course, we don't actually do this to communicate with Redis, we instead make use of the nicely featured CLI client or of course use a library for our language of choice.
By default Redis doesn't enable authentication. This is totally fine because by default Redis only listens for connections on the local machine. (This is the same philosophy used by MongoDB). Make sure you research and enable authentication if you ever choose to listen for connections from the outside world.
Redis is very useful as a cache and so we get two methods for describing expiration. One is done using expiration settings on a per-key basis, and the other is done by setting global Lease Recently Used (LRU) settings. Expect a future post from me on the intricacies of using Redis for caching.
Installing Redis
Redis is pretty easy to install, though there are a few different ways to do it depending on your platform.
macOS
First install Homebrew. Then run:
$ brew install redis
Ubuntu / Debian
$ sudo apt-get install redis
Docker
$ mkdir -p ~/data/redis
$ docker run \
--name my-redis-db \
-p 6379:6379 \
-v ~/data/redis:/data \
-d redis \
redis-server --appendonly yes
Source
Use this option as a last resort if one of the above methods doesn't work for you. This will download the source and compile it, leaving you with some binaries in the src/ directory to work with. You can copy these binaries to your system if you'd like to install it permanently.
$ curl http://download.redis.io/releases/redis-3.2.9.tar.gz -o redis-3.2.9.tar.gz
$ tar -xzvf redis-3.2.9.tar.gz
$ cd redis-3.2.9
$ make
$ ./src/redis-server
Connecting to Redis
Once you've installed Redis and have the Redis server up and running, run the following commands to test that everything is working correctly.
$ redis-cli
127.0.0.1:6379> SET xyz 'Hello'
OK
127.0.0.1:6379> GET xyz
"Hello"
127.0.0.1:6379> DEL xyz
(integer) 1
127.0.0.1:6379> GET xyz
(nil)
127.0.0.1:6379> QUIT
Introduction to Node
Node.js (Node) is an implementation of the JavaScript language for running outside of a browser. It is commonly used for, but definitely not limited to, building applications which listen for HTTP requests.
Node makes use of the V8 JavaScript engine originally built for powering the Chrome browser. JavaScript is single threaded and so is Node, for the most part. The “userland” code your application runs is limited to a single thread, however the non-blocking I/O operations it performs will happen outside of that thread.
Node also contains a method for requiring additional files into your application, as well as a powerful API for performing almost any interaction with the OS that you would need. For example there are API's for listening for HTTP, TCP, UDP requests, making outgoing requests for those same protocols, interacting with the filesystem, etc.
At the heard of Node is another library called libuv. This library abstracts the interactions with the OS and helps make them consistent across different platforms. libuv is written in C and does a lot of the heavy lifting for your application.
Node applications typically execute callbacks when an asynchronous I/O operation is complete. As an example (WARNING: Hand waving), if you want to read a text file, your JavaScript code will call out to the Node API and provide a callback, then go back to doing other things. The Node API's pass this work along to libuv which will then begin reading the file from disk. Once the data is loaded into memory it provides the data to the Node API you called which then executes your callback, allowing your JavaScript code to handle the newly read data. While your app waits for the data to be read it can do other things, or just “fall asleep” if it only has this one thing to do.
Installing Node
If you're new to Node then I suggest you simply visit the Node downloads page and find the appropriate download for you! Despite what anyone else may tell you, only once you're more advanced should you bother using a Node version manager.
Redis Data Structures
Redis supports several types of data structures. These structures provide simple yet powerful methods for describing your data. Each item that you create in Redis will exist under a different key.
Some commands can be run against keys of any type, such as checking if a key exists, deleting a key, setting / getting / removing a TTL, getting a type, and renaming keys.
Strings
Strings are the most basic type of storage we have available to us in Redis. Strings can contain any binary data that you put in them, as long as they can fit into ram. These are simple key/value pairs.
One bonus feature with strings is that you can increment and decrement numerical data, such as “100”. If you try to increment a key which doesn't exist, Redis will assume the kay has a value of “0” and act accordingly.
You can do things with strings such as get and set the value, get the length, increment and decrement values by arbitrary amounts, append strings and get sub strings.
Lists
Lists are ordered collections of string data, synonymous with a linked list or an array in JavaScript. Lists can contain duplicate strings.
You can perform operations on lists such as adding and removing items from the beginning or the end, remove or read items at arbitrary locations, get the length of a list, or move the last item in the list to be the first item.
Hashes
Hashes are field/value pairs within a single Redis key. Field names and value names are both strings. Names must be unique within a single hash but their values can repeat. Remember that since a hash is a single key that a TTL is applied to the entire key, not just a single field. These are similar to a JavaScript object / ES2015 Map.
You can get or set different fields, remove fields, see if a field exists, get a list of just keys or values, increment or even get string lengths of values.
Sets
Sets are unordered collections of unique strings. These are analogous to a JavaScript ES2015 Set. If a duplicate item is added to a set the repeated element will not be added.
With Sets you can add or remove items, count the items, see if an item exists, and even get the diff / union / intersection of multiple sets.
Sorted Sets
Sorted Sets are like a combination between a Hash and a Set. Instead of field/value pairs a Sorted Set has score/value pairs. The values do need to be unique though the scores do not.
With a Sorted Set we can get the number of items, add or remove items, get a list of items falling within a score range.
Sorted Sets almost sound like they would have a limited use-case but they're actually a powerful building blog for many applications. One common use is to create a Leaderboard, with a score representing a players score and a value representing a player id. Setting a players new score replaces their old score. We can also query a large leaderboard to know a players rank and the rank of their neighbors.
GeoLocation
GeoLocation keys are useful for storing strings which are keyed based on their latitude/longitude location. GeoLocation keys are actually implemented as Sorted Sets under the hood, however we're given some special commands to work with the data.
We can do things like insert and remove entries and get a list of entries within a radius of a specific location. This provides us with a very fast method for searching for neighboring items.
Pub/Sub
The Pub/Sub features of Redis technically aren't a data type but I've included them here for completeness. These allow consumers to subscribe to a channel based on a name or a pattern. They also allow producers to publish to a channel. Many different consumers can subscribe to many different channels. Many producers can publish to any channel they'd like.
A channel doesn't need to exist before it's subscribed or published to. If a message is published to a channel without subscribers there will be no error. We are given a numeric count of the number of subscribers of a channel when publishing a message.
Using Redis with Node
There is a pretty good Redis module available to the Node ecosystem. Visit a new directory and create a new Node project. Then, this client can be installed to the local project by running the following shell command:
$ npm init
$ npm install --save redis
Now that you have the library installed locally you can create a new JavaScript file with the following content in a file named app.js:
const redis = require('redis').createClient();
redis.set('hello', 'world', (err) => {
if (err) { throw err; }
redis.get('hello', (err, data) => {
if (err) { throw err; }
console.log(`Hello, ${data}!`); // outputs 'Hello, world!'
redis.quit();
});
});
Now you can execute your app as follows:
$ node app.js
You should see the string Hello, world! displayed in your console. In order for this to work your Node application has communicated with the locally running Redis server. If you get an error you probably need to ensure your server is running.
That's it for the first part. The next parts of this series will start to get more complex, e.g. we'll begin dealing with atomicity and distributed systems. And of course, if you found this content useful, please checkout my book Advanced Microservices.