Pre-release 0 of Jetty 7.0.0 is now available and includes a preview of the proposed Servlet 3.0 API for asynchronous servlets. This blog looks at 4 cool things you can do with asynchronous servlets and how they can be implemented using the proposed API.

The new APIs proposed for servlet 3.0 included in 7.0.0pre0 are:

ServletContext:
addServlet, addServletMapping, addFilter, addFilterMapping
Cookie:
setHttpOnly, isHttpOnly
ServletRequest:
getServletContext, getServletResponse, suspend, resume, complete, isSuspended, isResumed, isTimeout, isInitial
ServletResponse:
disable, enable, isDisabled
ServletRequestListener:
requestSuspended, requestResumed, requestCompleted
The key methods for the purposes of this blog, are the suspend and resume on ServletRequest. These are inspired by the suspend/resume aspects of Jetty Continuations and allow a servlet to return a request to the container to be handled later.  The unliked thrown exception of Continuations is gone and replaced with the ability to disable responses if suspend-unaware code needs to be traversed.

The following 4 use-cases show some of the diverse ways this API can greatly improve your web-1.0 and web-2.0 applications.

Ajax Cometd

OK boring you say! Yes I have been ranting and ranting about this use-case for some time: 20,000 simultaneous users etc. etc.  So I wont say much more about this other than that porting the cometd servlet to the 3.0 API from Continuations was trivial and resulted in simpler more easily readable code that will soon be portable between all servlet-3.0 containers.

Quality of Service Filter

I have previously blogged about how slow resources (eg JDBC) can cause thread starvation in synchronous web applications.  The Jetty  ThrottleFilter  was developed to allow only a fixed number of requests to access a given slow resource, and for excess requests to asynchronous wait to proceed.  This both protects the slow resource from over-use and protects the webapp from thread starvation.

With the QoSFilter in 7.0.0pre0, this is taken a step further, so that requests may be assigned a priority based on extensible criteria (eg  authenticated user or type of user), and higher priority users get preferential access to the protected resource.

Requests that enter the filter for the first time try to acquire the _passes semaphore, which limits the number of requests that can simultaneously acquire it:

  if (request.isInitial())
{
accepted=_passes.tryAcquire(_waitMs,TimeUnit.MILLISECONDS);
If a request is accepted by the semaphore, it simply allows the request to continue down the filter chain:
    if (accepted)
chain.doFilter(request,response);
If a request is not accepted by the semaphore in the _waitMs time, then the request is suspended using the new 3.0 API and the filter returns the request after queueing it on the appropriate priority queue:
    if (!accepted)
{
request.suspend();
int priority = getPriority(request);
_queue[priority].add(request);
return;
}
Note that the thread continues to execute after the suspend. It has not suspended the thread, only the request. The suspend is done before adding to the _queue to prevent a resume occurring before the suspend! The priority queues are handled by accepted requests as they exit the filter in a finally clause. The priority queues are searched for the next highest priority request that is suspended and that request is resumed before the the _passes semaphore is released:
finally
{
if (accepted)
{
for (int p=_queue.length;p-->0;)
{
ServletRequest req=_queue[p].poll();
if (req!=null)
{
req.resume();
break;
}
}
_passes.release();
}
}
The requests so resumed, are run again and re-enter the filter. This time they do not pass the isInitial() and thus forcefully acquire the semaphore and proceed to call the filter chain:
  if (request.isInitial())
{
...
}
else if (request.isResumed())
{
_passes.acquire();
accepted=true;
}
...

if (accepted)
chain.doFilter(request,response);
The ability to favour the processing of some requests over others is a significant new ability available in servlet-3.0.

Asynchronous Web Services

Jesse McConnell has already blogged about his demo showing the CFX asynchronous web services clients working with the Jetty 7.0.0pre0 to allow web service calls in parallel and without
threadful waiting for the responses:

The demo shows the thread time taken to access the ebay webservices interface. With a single call, both synchronous and asynchronous techniques took about 860ms to produce the  response. But with the synchronous client, a thread was held the entire time of that call, and could not be used to service other requests. With the asynchronous api, the thread is only held for 2ms to send the ws request, suspend and then process the response. This thread could be used to handle hundreds of other requests while waiting for the ws response!

It get’s even better if multiple ws calls are required. The synchronous approach must do them in series, so the total time and thread hold time blows out to 2700ms for 3 requests. With the asynchronous API, the three ws requests are sent in parallel and the total time is 890ms, almost the same for a single request, and the thread is only held for 8ms. Again the thread could be used to service many other requests, rather than waiting idly  wasting resources!

Asynchronous Web Proxy

The WS demo above uses the CFX asynchronous client. Jetty now includes it own asynchronous HTTP client and this can be used in a very similar way to proxy HTTP requests to another server. The AsyncProxyServlet  demonstrates how a response can be generated for a suspended request without a resume.  The servlet-3.0 has a request.complete method that allows a asynchronous callbacks to generate the response and complete a suspended request:

The request to be proxied is copied into a HttpExchange, with a little bit of process (deleted here) for standard proxy stuff:

HttpExchange exchange = new HttpExchange()
{
...
};
exchange.setMethod(request.getMethod());
exchange.setURI(uri);
...
Enumeration enm = request.getHeaderNames();
while (enm.hasMoreElements())
{
String hdr=(String)enm.nextElement();
...
Enumeration vals = request.getHeaders(hdr);
while (vals.hasMoreElements())
{
String val = (String)vals.nextElement();
exchange.setRequestHeader(lhdr,val);
}
}
if (hasContent)
exchange.setRequestContentSource(in);

Once the HttpExchange has been constructed and configured it is asynchronously sent and the current requests is suspended while a responses is waited for:

_client.send(exchange);
request.suspend();

The processing of the proxy response is all done in call back methods supplied to the construction of the HttpExchange (not shown above):

HttpExchange exchange = new HttpExchange()
{
protected void onResponseStatus(Buffer version,
int status,
Buffer reason)
{
if (reason!=null && reason.length()>0)
response.setStatus(status,reason.toString());
else
response.setStatus(status);
}

protected void onResponseHeader(Buffer name,
Buffer value)
{
...
String s = name.toString().toLowerCase();
if (!_DontProxyHeaders.contains(s))
response.addHeader(name.toString(),value.toString());
}

protected void onResponseContent(Buffer content)
{
content.writeTo(out);
}

protected void onResponseComplete() throws IOException
{
request.complete();
}
}

The callbacks allow the real response status to be set, the headers to be copied over (with some more hidden processing), the content to be written and eventually the original response is completed.
Because the response has already been given a status, headers and content, there is no need to resume the request in order to generate a response.  

This approach will allow scalable proxies to be implemented as standard java servlets, which will in turn allow some arbitrary fancy business logic to be incorporated into these proxies.

Conclusion

Of these use-cases, only Cometd is a real web-2.0. The other use-cases: JDBC access, web services and proxying are all pretty standard parts of many web-1.0 applications. Thus these examples demonstrate how widely asynchronous servlets may be applied to both existing and new web applications to improve their scalability and quality of service.

If these topics interest you, come to the Jetty BOF at JavaOne.  Tuesday 7:30 pm 6 May 2008 San Francisco, or visit the Webtide booth.
Use-Cases for Async Servlets