For those who are interested, here is some background reading on Roomkey:<p>60,000% growth in 7 months using Clojure and AWS<p><a href="http://www.colinsteele.org/post/27929539434/60000-growth-in-7-months-using-clojure-and-aws" rel="nofollow">http://www.colinsteele.org/post/27929539434/60000-growth-in-...</a><p>Against the Grain: How We Built the Next Generation Online Travel Agency using Amazon, Clojure, and a Comically Small Team<p><a href="http://www.colinsteele.org/post/23103789647/against-the-grain-aws-clojure-startup" rel="nofollow">http://www.colinsteele.org/post/23103789647/against-the-grai...</a>
I wonder why they approached the problem from the way they did -- that is to say the monitor the effect of the problem but not the cause (# open fd caused by stuck threads). Why not monitor the # JVM threads.<p>I'm somewhat familiar with a tool we use at work for monitoring JVM - wily introscope, but I'm sure there are other options available (newrelic java agent? ps can check nwlp on Linux? prstat available on linux? other jmx solutions?). Again I'm not familiar with tomcat as much as websphere/weblogic but I see tomcat has this option <a href="https://tomcat.apache.org/tomcat-7.0-doc/config/valve.html#Stuck_Thread_Detection_Valve" rel="nofollow">https://tomcat.apache.org/tomcat-7.0-doc/config/valve.html#S...</a> which you can use in conjunction with other monitoring to get alerts I think for situations like this (your app threads getting stuck). I'm not sure if tomcat will write a warning log msg or actually mark the the thread as "STUCK" in thread dump.<p>The author did mention Java Mission Control but I wanted to point out that's not the only option for JVM monitoring.
"You can use the free Java Mission Control instead to do this investigation, but I do not have our production JVMs configured to accept remote connections"
The free JMC is only for dev time not for production use, as far as I know -- not sure if Oracle has a way to determine what is dev and what is production.