Service Discovery

With Node.js and Consul

Presented by @tlhunter@mastodon.social

I'm writing Distributed Systems with Node.js: bit.ly/34SHToF

Naive Approach: Static Host Lookups

  • Provider (Data Service)
  • Consumer (Web Service)
  • Fine for finite number of services

Naive Approach: Static Host Lookups

haproxy.cfg

frontend http-in
  bind 0.0.0.0:80
  default_backend www
backend www
  server www1 www.example.org:80

service-www.js

const request = require('request');

request.get('http://data.example.org:80', (err, data) => {
  doStuff(data);
});

Shortcomings

  • Single Points of Failure (in particular code that we own)
  • Unable to scale services dynamically
  • Even with Node Cluster, master dies, app dies

What is Consul?

  • Open-Source Discovery tool, written by Hashicorp
  • Raft protocol, Quorum, Leader Election (3 or 5 members)
  • Keeps list of service instances, name -> host:port
  • Exposes a RESTful API, plenty of client libraries

Register and Discover Hosts

  • Data Service registers its locations to Consul
  • Data Service Heartbeats tell Consul they're alive
  • Web Service is told where Data is located

Register and Discover Hosts

service-data.js

  • Data Service Register when it is created
const CONSUL_ID = require('uuid').v4();
let details = {
  name: 'data',
  address: HOST,
  port: PORT,
  id: CONSUL_ID,
  check: {
    ttl: '10s',
    deregister_critical_service_after: '1m'
  }
};
consul.agent.service.register(details, err => {
  // schedule heartbeat
});

Register and Discover Hosts

service-data.js

  • Data Service Heartbeat runs every 10/2 seconds
setInterval(() => {
  consul.agent.check.pass({id:`service:${CONSUL_ID}`}, err => {
    if (err) throw new Error(err);
    console.log('told Consul that we are healthy');
  });
}, 5 * 1000);

Register and Discover Hosts

service-data.js

  • Data Service De-Register when it is (gracefully) destroyed
process.on('SIGINT', () => {
  console.log('SIGINT. De-Registering...');
  let details = {id: CONSUL_ID};

  consul.agent.service.deregister(details, (err) => {
    console.log('de-registered.', err);
    process.exit();
  });
});

Register and Discover Hosts

service-www.js

  • Web keeps track of changes to Data
let known_data_instances = [];

const watcher = consul.watch({
  method: consul.health.service,
  options: {
    service: 'data',
    passing: true
  }
});

watcher.on('change', data => {
  known_data_instances = [];
  data.forEach(entry => {
    known_data_instances.push(`http://${entry.Service.Address}:${entry.Service.Port}/`);
  });
});

Register and Discover Hosts

service-www.js

  • Web Service looks for a random Data Service
function getData(cb) {
  let url = known_data_instances[Math.floor(Math.random()*known_data_instances.length)];

  request(url, {json:true}, (err, res, data) => {
    if (err) return cb(err);

    cb(null, data);
  });
}

Reconigure HAProxy with Consul Template

  • Web Service now announces like Data Service
  • Consul Template dynamically reconfigures HAProxy

Reconigure HAProxy with Consul Template

haproxy.cfg.template

  • Uses the HCL "Hashicorp Configuration Language"
frontend http-in
  bind 0.0.0.0:80
  default_backend www
backend www{{range service "www"}}
  server {{.ID}} {{.Address}}:{{.Port}}{{end}}

Reconigure HAProxy with Consul Template

  • Configure Consul Template how to restart HAProxy
  • Rebuilds advanced.cfg and restarts (if needed)
$ consul-template -template "./advanced.cfg.template:./advanced.cfg:./haproxy-restart.sh"

haproxy-restart.sh

  • HAProxy restarts while handing off old port
#!/bin/bash

echo "DEBUG: restarting haproxy"

haproxy -f ./advanced.cfg -p ./haproxy.pid -D -st $(cat ./haproxy.pid)

Benefits

  • No single point of failure in our service code
    • Though, HAProxy is a single point of failure
    • However it's popular and stable
  • Easily run code on many different hosts

Drawbacks

  • Chatter: Processes need continuous heartbeats
  • Bad routing can happen up to 10 seconds after process death
  • There is more complexity in each process

Fin