One of the key features added in the Servlet 3.1 JSR 340 is asynchronous (aka non-blocking) IO.   Servlet 3.0 introduced asynchronous servlets, which could suspend request handling to asynchronously handle server-side events.  Servlet 3.1 now adds IO with the request/response content as events that can be handled by an asynchronous servlet or filter.

The Servlet 3.1 API is available in the Jetty-9.1 branch and this blog shows how to use the API and also some Jetty extensions are shown that further increase the efficiency of asynchronous IO. Finally an full example is given that shows how asynchronous IO can be used to limit the bandwidth used by any one request.

Why use Asynchronous IO?

The key objective of being asynchronous is to avoid blocking.  Every blocked thread represents wasted resources as the memory allocated to each thread is significant and is essentially idle whenever it blocks.

Blocking also makes your server vulnerable to thread starvation. Consider a server with 200 threads in it’s thread pool.  If 200 requests for large content are received from slow clients, then the entire server thread pool may be consumed by threads blocking to write content to those slow clients.    Asynchronous IO allows the threads to be reused to handle other requests while the slow clients are handled with minimal resources.

Jetty has long used such asynchronous IO when serving static content and now Servlet 3.1 makes this feature available to standards based applications as well.

How do you use Asynchronous IO?

New methods to activate Servlet 3.1 asynchronous IO have been added to the ServletInputStream and ServletOutputStream interfaces that allow listeners to be added to the streams that receive asynchronous callbacks.  The listener interfaces are WriteListener and ReadListener.

Setting up a WriteListener

To activate asynchronous writing, it is simply a matter of starting asynchronous mode on the request and then adding your listener to the output stream.  The following example shows how this can be done to server static content obtained from the ServletContext:

@Override
protected void doGet(HttpServletRequest request, HttpServletResponse response)
  throws ServletException, IOException
{
  // Get the path of the static resource to serve.
  String info=request.getPathInfo();
  // Set the mime type of the response
  response.setContentType(getServletContext().getMimeType(info));        
  // Get the content as an input stream
  InputStream content = getServletContext().getResourceAsStream(info);
  if (content==null)
  {
    response.sendError(404);
    return;
  }
  // Prepare the async output
  AsyncContext async = request.startAsync();
  ServletOutputStream out = response.getOutputStream();
  out.setWriteListener(new StandardDataStream(content,async,out));
}

Note how this method does not actually write any output, it simple finds the content and sets up a WriteListener instance to do the actually writing asynchronously.

Implementing a WriteListener

Once added to the OutputStream, the WriteListener method onWritePossible is called back as soon as some data can be written and no other container thread is dispatched to handle the request or any async IO for it. The later condition means that the first call to onWritePossible is deferred until the thread calling doGet returns.

The actual writing of data is done via the onWritePossible callback and we can see this in the StandardDataStream implementation used in the above example:

private final class StandardDataStream implements WriteListener
{
  private final InputStream content;
  private final AsyncContext async;
  private final ServletOutputStream out;
  private StandardDataStream(InputStream content, AsyncContext async, ServletOutputStream out)
  {
    this.content = content;
    this.async = async;
    this.out = out;
  }
  public void onWritePossible() throws IOException
  {
    byte[] buffer = new byte[4096];
    // while we are able to write without blocking
    while(out.isReady())
    {
      // read some content into the copy buffer
      int len=content.read(buffer);
      // If we are at EOF then complete
      if (len < 0)
      {
        async.complete();
        return;
      }
      // write out the copy buffer. 
      out.write(buffer,0,len);
    }
  }
  public void onError(Throwable t)
  {
      getServletContext().log("Async Error",t);
      async.complete();
  }
}

When called, the onWritePossible() method loops reading content from the resource input stream and writing it to the response output stream as long as the call to isReady() indicates that the write can proceed without blocking.    The ‘magic’ comes when isReady() returns false and breaks the loop, as in that situation the container will call onWritePossible() again once writing can proceed and thus to loop picks up from where it broke to avoid blocking.

Once the loop has written all the content, it calls the AsyncContext.complete() method to finalize the request handling.    And that’s it! The content has now been written without blocking (assuming the read from the resource input stream does not block!).

Byte Arrays are so 1990s!

So while the asynchronous APIs are pretty simply and efficient to use, they do suffer from one significant problem.  JSR 340 missed the opportunity to move away from byte[] as the primary means for writing content!  It would have been a big improvement to add an write(ByteBuffer) method to ServletOutputStream.

Without a ByteBuffer API, the content data to be written has to be copied into a buffer and then written out.   If a direct ByteBuffer could be used for this copy, then at least this data would not enter user space and would avoid an extra copies by the operating system.  Better yet, a file mapped buffer could be used and thus the content could be written without the need to copy any data at all!

So while this method was not added to the standard, Jetty does provide it if you are willing to down caste to our HttpOutput class.  Here is how the above example can be improved using this method and no data copying at all:

protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException
{
  String info=request.getPathInfo();                
  response.setContentType(getServletContext().getMimeType(info));
  File file = new File(request.getPathTranslated());
  response.setContentLengthLong(file.length());
  // Look for a file mapped buffer in the cache
  ByteBuffer mapped=cache.get(path);
  if (mapped==null)
  {
    try (RandomAccessFile raf = new RandomAccessFile(file, "r"))
    {
      ByteBuffer buf = raf.getChannel().map(MapMode.READ_ONLY,0,raf.length());
      mapped=cache.putIfAbsent(path,buf);
      if (mapped==null)
        mapped=buf;
    }
  }
  // write the buffer asynchronously
  final ByteBuffer content=mapped.asReadOnlyBuffer();
  final ServletOutputStream out = response.getOutputStream();
  final AsyncContext async=request.startAsync();
  out.setWriteListener(new WriteListener()
  {
     public void onWritePossible() throws IOException
     {            
       while(out.isReady())
       {
         if (!content.hasRemaining())
         {              
           async.complete();
           return;
         }
         out.write(content);
      }
      public void onError(Throwable t)
      {
        getServletContext().log("Async Error",t);
        async.complete();
      }
  });
}

Note how the file mapped buffers are stored in a ConcurrentHashMap cache to be shared between multiple requests.  The call to asReadOnlyBuffer() only creates a position/limit indexes and does not copy the underlying data, which is written directly by the operating system from the file system to the network.

Managing Bandwidth – Limiting Data Rate.

Now that we have seen how we can break up the writing of large content into asynchronous writes that do not block, we can consider some other interesting use-cases for asynchronous IO.

Another problem frequently associated with large uploads and downloads is the data rate.  Often you do not wish to transfer data for a single request at the full available bandwidth for reasons such as:

• The large content is a streaming movie and there is no point paying the cost of sending all of the data if the viewer ends up stopping the video 30 seconds in.  With streaming video, it is ideal to send the data at just over the rate that it is consumed by a viewer.
• Large downloads running at full speed may consume a large proportion of the available bandwidth within a data centre and can thus impact other traffic.  If the large downloads are low priority it can be beneficial to limit their bandwidth.
• Large uploads or requests for large downloads can be used as part of a DOS attack as they are requests that can consume significant resources. Limiting bandwidth can reduce the impact of such attacks and cost the attacker more resources/time themselves.

We have added the DataRateLimitedServlet to Jetty-9.1 as an example of how asynchronous writes can be slowed down with a scheduler to limit the data rate allocated to any one requests.  The servlet uses both the standard byte[] API and the extended Jetty ByteBuffer API.   Currently it should be considered example code, but we are planning on developing it into a good utility servlet as Jetty-9.1 is release in the next few months.