HTTP API Design Part 1: Requests

This is the first of four articles on HTTP API Design. These articles are based on content from my recent book Advanced Microservices. This content is influenced by the HTTP standard itself as well as common RESTful practices.

HTTP Request Overview

HTTP is a protocol which sits a level above TCP. This protocol uses a request/response pattern where a client makes a request and a server replies. Here's an example of a complete HTTP request:

POST /v1/animal HTTP/1.1        <-- Request Line
Host: api.example.org           <-- Request Headers
Accept: application/json
Content-Type: application/json
Content-Length: 24
                                <-- Two Newlines
{                               <-- Body
  "name": "Gir",
  "animal_type": "12"
}

The first line is called a Request Line and contains three items; the HTTP method, a path (also referred to an endpoint in an API), and the HTTP version. We'll take a closer look at HTTP Methods and Endpoints later in this article.

The content which follows the Request Line are called headers. These headers are key/value pairs with the key coming first, followed by a colon and a space, and then the value. Headers are separated by newlines. Headers can technically be in any case but it's best to send them as Capital-Hyphenated-Words. If you're writing a server it's important to accept headers regardless of case. Headers can be repeated in situations where duplicate values need to be sent.

A request only needs to have a Request Line and some headers to be valid. This is mostly true of requests using the HTTP Methods GET, DELETE, HEAD, and OPTIONS. However we can also provide a request body, which requires two newlines after the last header before the body content. Bodies are mostly used with the HTTP Methods POST, PUT, and PATCH. Technically it's possible to do something like provide a body with a GET request; this is commonly used by the software Elasticsearch.

Endpoints (Paths)

When designing a RESTful HTTP API, it's important to be able to abstract the functionality of your service in such a way that all operations can be represented as performing CRUD (Create, Read, Update, Delete) operations on different resources (entities). The actions (verbs) being performed should never be part of the endpoint.

The most commonly used approach is to expose different collections of related resources. As an example, if your service contains information about different companies and employees, you could have a collection called companies and another collection called employees. We can then get more specific and refer to an individual company or employee resource within each of these collections.

We can also get more specific and use this same pattern to represent relationships between resources and sub-resources. As an example, an employee can be employed at a company. The following is how we can represent this data and their relationships:

/companies
/companies/{company_id}
/employees
/employees/{employee_id}
/companies/{company_id}/employees
/companies/{company_id}/employee/{employee_id}

If we want to act upon the collection of companies we can use /companies. If we want to act upon an individual company we can use /companies/{company_id}. If we want to act upon a specific employee in regards to their relationship with a specific company we can then use the /companies/{company_id}/employee/{employee_id} endpoint.

In these next sections on HTTP Methods we'll cover what it means to act upon resources.

GET Method

The GET Method refers to the Read portion of CRUD. GET requests are considered Safe, which means they should not alter the state of the server. GET requests are considered Idempotent, which means they should be repeatable without causing side effects. GET requests shouldn't have a body.

When applied against a collection is it used for retrieving a list of resources within that collection. We can retrieve all resources or filter specific resources based on criteria. Here's an example of how we might get a list of employees:

GET /companies/twitter/employees?maxStartDate=2016-06-23&perPage=10&offset=20

The above request could translate into “get a list of 10 employees who work at Twitter who have been employed for at least a year, skipping the first 20”. The endpoint itself (the path of the URL) follows some very common RESTful patterns, however the query parameters which follow (?key=val&key2=val2) are a bit more ad-hoc. If filtering properties are not specified then the server should reply with all matching resources. Unfortunately this could be an overwhelming about of data so it's common for servers to have default limits on returned data.

When applied against a particular resource it is then useful for getting information about that exact resource. In these situations it's common to either whitelist or blacklist parameters which the client is looking for. Here's an example request:

GET /employees/tlhunter?fields=name,age,twitter

This request could translate into “get the name, age, and Twitter handle of the employee identified as tlhunter”. If such a whitelist or blacklist of parameters is not provided then the server should respond with an entire representation of the resource.

POST Method

The POST Method corresponds to the Create portion of CRUD. POST requests are considered Unsafe as they alter the state of the server. They are considered Not Idempotent as repeated requests will result in multiple resources being created. POST requests should have a body.

When a POST is applied to a collection the server should create a new resource within that collection. The following example is how we may choose to add a new employee to our service:

POST /employees

The request body would contain information about the newly added employee.

If a POST is applied to a sub-collection within a collection it can then be used to create a new relationship. As an example, if we were to hire an employee at a company we may make the following request:

POST /companies/twitter/employees

The request body could contain some sort of reference to an existing employee, typically containing the identifier or primary key of the employee. This would then cause the employee to be hired at the company.

If the server represents a very malleable collection of data, such as a database, one could perhaps perform a POST at the root of the service (above a collection) and create a new collection.

DELETE Method

The DELETE Method obviously corresponds to the Delete portion of CRUD. DELETE requests are considered Unsafe as they alter the state of the server. They are however considered Idempotent as repeating a DELETE should have no side effect (delete an employee once, they're gone. Delete them again, well, they're still gone). A DELETE request shouldn't have a body.

In the following example we can fire/terminate an employee from our company by deleting their relationship with that company:

DELETE /companies/twitter/employees/jkup

And of course if we wanted to delete this employee entirely from the system we could do the following:

DELETE /employees/jkup

Again, if the server represents a database, we could even allow performing a DELETE request directly against a collection, which could then wipe-out the entire collection.

PUT/PATCH Methods

The PUT and PATCH Methods correspond to the Update portion of CRUD. Both of them are considered Unsafe as they alter server state. They are usually considered Idempotent as performing the same request multiple times should have the same result. This is true in situation where you are providing resource properties and values to be replaced on a document. However, if your service has the ability to modify a property, such as incrementing a property, these requests would become Not Idempotent. Both the PUT and PATCH methods should have a body.

What's the difference between PUT and PATCH? Well, it depends on whose API you're using. One approach is that a PUT request performs a full update, missing properties resulting in null values. E.g. an employee resource could have a name and role property, but performing a PUT and only supplying a name would result in a missing role. Then, a PATCH request can be used to perform partial updates. Only properties provided in the request body will be set, unlisted properties will retain their existing value. Some API's only support either PUT or PATCH for performing any updates.

If we wanted to change the name of an employee we would make the following request and provide the name in the request body:

PATCH /employees/tlhunter

If we wanted to give an employee of a company a raise perhaps we could make this request and provide their new role in the body:

PATCH /companies/twitter/employees/kng

HEAD/OPTIONS Methods

The HEAD and OPTIONS Methods are a bit special. They most closely correspond to the Read portion of CRUD but neither actually retrieve resources. They are considered Safe as they don't alter state and Idempotent as they can be repeated without ill-effect. Neither should have a request body.

The HEAD request is used for getting only the headers about a resource. These headers can contain useful meta-data such as when the resource should be expired. If the cost of retrieving a resource is high (perhaps it's a large document which is slow to transmit or perhaps it's an expensive database call) it could then make sense to use a HEAD request instead of a GET.

The OPTIONS request is useful for services which are exposed to a browser and where the host of the service is different than the host of a web application being used by the browser. Modern browsers implement a feature called CORS (Cross Origin Resource Sharing). When a web app instructs the browser to access a file from a remote host, if will first make an OPTIONS preflight request before making the real request. The server then responds with headers specifying what sort of permissions the resource has, for example can the remote domain access the resource. If the permission check fails then the browser won't make the normal request.

Request Headers

There are many different request headers which can be made. Browsers in particular send a lot of headers and new ones are being standardized all of the time. The following is a list of common request headers and their meaning, each of these should be supported by any HTTP API you build:

• Accept: This is a list of Content-Types accepted by the client. A common content type used by HTTP APIs is application/json.
• Accept-Language: This is a list of languages and fallbacks expected by the client. An example of this is en-US, en, de-DE.
• Content-Length: This is the length in bytes of the request body. If the request body size is known then this value should be sent.
• Content-Type: This represents the type of content provided in the request body. Only send if the request contains a body.
• Host: This header should always be sent with HTTP requests and represents the HTTP host. Applications will probably ignore this though, it's mostly useful for virtual hosts and request routing.
• User-Agent: This header includes identifying information about the client. In the world of Microservices this is useful to know which service is talking to which.

These are all standardized headers, but there are also a lot of ad-hoc ones as well. As a rule of thumb, when inventing your own headers you should first see if an existing header doesn't already fulfill your needs. If an appropriate one can't be found then the common pattern is to prefix it with an X, such as with X-API-Version to represent the API version.

This article is based on content from my book Advanced Microservices. There's also an accompanying HTTP API Design Presentation.

Thomas has contributed to dozens of enterprise Node.js services and has worked for a company dedicated to securing Node.js. He has spoken at several conferences on Node.js and JavaScript and is an O'Reilly published author.