A Continuation of another
Greg has blogged about the ThrottlingFilter I wrote for Jetty. He's covered the situation quite well.
I wrote the filter (with Greg's input) based a situation that was quite similar to what he describes. There are a few other ways you can approach it though.
I've not tried it, but I believe it would be feasible to push the same logic down into the connection pool (or any other pooled resource).
Many pooling implementations (e.g. commons-pool/commons-dbcp) offer configurable behaviour when the pool is full - a common behaviour is to block as Greg describes in his blog entry. What could be useful is to be able to do lost cost waiting - i.e. continuations from within the pool.
The filter is nice in that it stops the unwanted request at the beginning of the chain, before any processing time has been wasted on it, and when the impact of retrying is quite low. The cost to that is that you need to be able to split your application by URL. The filter needs to be able to tell whether to throttle any given request.
Putting that logic into the datasource would mean that some processing would have already been performed before the request is suspended, and you would have to be confident that the application is safe to retry. However it has the advantage of being very precise - requests are only suspended when they attempt to access an exhausted pool.
I'm sure there are a lot of other possible ways to make use of continuations. The use of continuations in the ThrottlingFilter was a suggestion by Greg - I had originally intended to simply reject requests that exceeded the maximum size until he reminded me how useful continutations would be in that context. I'm sure that with a bit of imagination many other uses could also be found.