Unpacking Open-Source REST Clients

Aka: Old-REST - Shootout at the 200 OK Canal

Open-source software (OSS) is free and, for the larger projects, well supported by the developer community. These traits are often reason enough for a developer to find an OSS project that fits their needs, drop it into their POM file, and essentially forget about it.

A less obvious benefit of OSS is the ability to investigate the source code when things don't quite work as expected. In this blog I'll walk through a recent OSS excavation that revealed performance issues lurking.

Case Study: POSTing to an HTTP resource

As any Java developer from the late '90's can attest, we have come a long way from java.net.URLConnection. The (seemingly) simple task of connecting to a remote resource via HTTP required a whole lot of boilerplate and custom implementation. For example, an HTTP POST to a url:

URLConnection connection = url.openConnection();
connection.setDoOutput(true);
connection.setRequestProperty("Content-Type", "application/json");

try (OutputStream out = connection.getOutputStream()) {
    out.write(json.getBytes());
}

InputStream in = connection.getInputStream();
// consume response from the server

In addition, there is no built-in facility for connection pooling or concurrency with this model. The open-source HttpClient library from the Apache Commons project corrected most of the shortcomings of the built-in JDK implementation, but was cumbersome, verbose and difficult to configure.

So it was a breath of rather fresh air to read a class file in my first month at SourceClear to see:

HttpResponse<JsonNode> response = Unirest.post(url) // 1
    .header("Content-Type", "application/json") // 2
    .body(json) // 3
    .asJson(); // 4

JsonNode node = response.getBody(); // 5

This is the Unirest HTTP Client. The code above looks pretty ideal. The POST and retrieval of a JSON resource is achieved in only 5 lines of code. More importantly the above code is expressive - it reads fluently and is easy to follow along and see exactly what is happening: we (1) initiate the request; (2) set a request header; (3) add content to the POST body; (4) get the response; and lastly (5) get the object out of the response.

Since it looked too good to be true, I wanted to see if this library was enterprise production-ready. Frequently, OSS is optimized for smaller projects with less demanding performance requirements; not out of laziness or developer oversight but simply because that what the majority of use cases require.

Concurrency and Connection Pooling

I first checked to see how this library would hold up under concurrent load. I have observed other OSS HTTP clients that fail to close connections properly (for an interesting sidebar, see: https://github.com/AsyncHttpClient/async-http-client/issues/785) leading to socket I/O exhaustion, others that fail to take advantage of HTTP 1.1 keep-alives, and others that exhibit poor concurrency.

To facilitate my investigation, in my IDE (IntelliJ IDEA) I clicked into the "Maven Projects" and then "Download Sources". This installs the Unirest project Java code in my local repo enabling me to navigate to it via "Go To Declaration." A few more hops shows us the definition of asJson():

public HttpResponse<JsonNode> asJson() throws UnirestException {
    return HttpClientHelper.request(httpRequest, JsonNode.class);
}

And one more click into HttpClientHelper.java reveals:

import org.apache.http.client.HttpClient;

// The DefaultHttpClient is thread-safe
HttpClient client = ClientFactory.getHttpClient();

This gives us some important information: under the hood this library uses the Apache HttpClient which is proven to be well-suited for enterprise, demanding web applications. We also see that the library author has considered concurrency, and (in the ClientFactory) code we find the answers to our "reasonable production configuration" query:

PoolingHttpClientConnectionManager syncConnectionManager = new PoolingHttpClientConnectionManager();
syncConnectionManager.setMaxTotal((Integer) maxTotal);
syncConnectionManager.setDefaultMaxPerRoute((Integer) maxPerRoute);

Memory

Coincidentally, the day after I first saw Unirest in our codebase, we had an unexpected failure reported in our error logs. The stack trace:

java.lang.OutOfMemoryError: Java heap space
    at com.mashape.unirest.http.HttpResponse.<init>(HttpResponse.java:84)

Paraphrasing the client code on that

InputStream responseInputStream = responseEntity.getContent();
byte[] rawBody = getBytes(responseInputStream);

The client code is unconditionally buffering the response in a byte array for use later. It is clear how this will fail when you consider that we have multiple clients, each fetching large (potentially hundreds of megabytes) REST resources. The issue is compounded because the buffered byte array remains in scope while the response is unmarshaled to a JsonNode, effectively doubling the amount of RAM needed per response. When more than two or three of these execute concurrently, we run out of memory quickly.

Buffering vs Streaming: Apache HTTP Client

Unirest buffers the HTTP response so that it can be "re-played" (as an InputStream), in addition to being transformed into the desired representation (i.e. String or JsonNode). Interestingly, the byte[] buffer is retained even if the desired representation is an InputStream. Initially I found this chagrining, and thought about filing an issue with the GitHub project - if I want streaming content, I should be able to manage it myself and dispense with the buffered input.

But, enabling clients to directly manage streaming responses exposes a host of potential pitfalls to library consumers (and the authors to whom they would complain when things go wrong). This is an issue with the very popular Apache HTTP Client library - it is incumbent on consumers to properly close the HTTP connection and dispose of network resources. Here is what its typical use looks like:

HttpClient client = getHttpClient();
HttpPost post = new HttpPost(url);
post.setEntity(json);
CloseableHttpResponse response = client.execute(post);
try {
    InputStream in = response.getEntity().getInputStream();
    try {
        processResponse(in);
    } finally {
        in.close();
    }
} finally {
    response.close();
}

This idiom is verbose and error prone; failure to properly dispose of resources results in a mess of open sockets, I/O issues and concurrency failures which can be difficult to debug. Unirest addresses this shortcoming, but fails in our case is because it tries to be the best of two worlds - it provides access to both the desired representation (an unmarshaled JsonNode) and the (buffered) streaming response.

One early discovery in my first seven weeks at SourceClear (the first time I've extensively used the Spring framework) is that if you ask, "can't Spring do this?" the answer is almost always "yes." Does Spring have an HTTP Client that is simple, expressive and performs well for an arbitrarily-sized HTTP request without excessive maintenance overhead? Yes indeed!

Spring RestTemplate

Spring comes with the built-in RestTemplate:

@Autowired
private RestTemplate restTemplate;  // configured by Spring

JsonNode responseNode = restTemplate.postForObject(url, requestNode, JsonNode.class);

It definitely passes the "easy/expressive" sniff test. What about performance and memory use? Clicking into the Spring OSS implementation code we see the lifecycle of an HTTP conversation:

ClientHttpResponse response = null;
try {
    ClientHttpRequest request = createRequest();
    // omitted for bevity
    response = request.execute();
    handleResponse(response);

    return responseExtractor.extractData();
} finally {
    if (response != null) {
        response.close();
    }
}

We see that RestTemplate properly manages I/O resources by calling response.close() in a finally block. Closely examining memory use is difficult because there are multiple implementations of the ResponseExractor interface that handle, for example, converting the response into an unmarshaled JsonNode or Java object. Clicking through the code path to create a response object, we find ourselves in AbstractJackson2HttpMessageConverter.readJavaType() which eventually calls:

return this.objectMapper.readValue(inputMessage.getBody(), javaType);

InputMessage likewise has several (nearly a dozen) implementations for different pluggable HTTP Clients (synchronous and asynchronous Apache, Netty, and others). The getBody() method returns an InputStream and auditing the implementation code shows that the open stream is directly used to convert the response; there is no in-memory buffering (note: buffering can be optionally enabled for replayable streams; see org.springframework.http.client.BufferingClientHttpResponseWrapper).

Closing thoughts

The traits of the three open-source HTTP frameworks described so far can be summarized:

Spring RestTemplate:

• Pros: expressive, simple, high performance
• Cons: requires Spring Framework

Unirest:

• Pros: expressive, simple, runs anywhere
• Cons: potential memory issues with large responses

Apache HTTP Client:

• Pros: powerful, lots of configuration options
• Cons: complex, verbose, error-prone resource management

The main takeaway is that the use case matters most, in order to optimize performance. Apache's HTTP Client is so powerful that it serves well as the underlying driver for other libraries (such as Unirest and Spring) but is difficult to use directly. For us, Unirest is a poor choice, for two reasons: we consume large API responses, and we are using Spring which has a terrific built-in option. However it might be the perfect solution for other projects.

But, especially since we already use the Spring framework here, its built-in offering is clearly the right choice given its simplicity and performance edge.