Too many open files: Tracking down a bug in production

For those who are interested, here is some background reading on Roomkey:<p>60,000% growth in 7 months using Clojure and AWS<p><a href="http:&#x2F;&#x2F;www.colinsteele.org&#x2F;post&#x2F;27929539434&#x2F;60000-growth-in-7-months-using-clojure-and-aws" rel="nofollow">http:&#x2F;&#x2F;www.colinsteele.org&#x2F;post&#x2F;27929539434&#x2F;60000-growth-in-...</a><p>Against the Grain: How We Built the Next Generation Online Travel Agency using Amazon, Clojure, and a Comically Small Team<p><a href="http:&#x2F;&#x2F;www.colinsteele.org&#x2F;post&#x2F;23103789647&#x2F;against-the-grain-aws-clojure-startup" rel="nofollow">http:&#x2F;&#x2F;www.colinsteele.org&#x2F;post&#x2F;23103789647&#x2F;against-the-grai...</a>

I wonder why they approached the problem from the way they did -- that is to say the monitor the effect of the problem but not the cause (# open fd caused by stuck threads). Why not monitor the # JVM threads.<p>I&#x27;m somewhat familiar with a tool we use at work for monitoring JVM - wily introscope, but I&#x27;m sure there are other options available (newrelic java agent? ps can check nwlp on Linux? prstat available on linux? other jmx solutions?). Again I&#x27;m not familiar with tomcat as much as websphere&#x2F;weblogic but I see tomcat has this option <a href="https:&#x2F;&#x2F;tomcat.apache.org&#x2F;tomcat-7.0-doc&#x2F;config&#x2F;valve.html#Stuck_Thread_Detection_Valve" rel="nofollow">https:&#x2F;&#x2F;tomcat.apache.org&#x2F;tomcat-7.0-doc&#x2F;config&#x2F;valve.html#S...</a> which you can use in conjunction with other monitoring to get alerts I think for situations like this (your app threads getting stuck). I&#x27;m not sure if tomcat will write a warning log msg or actually mark the the thread as &quot;STUCK&quot; in thread dump.<p>The author did mention Java Mission Control but I wanted to point out that&#x27;s not the only option for JVM monitoring.

&quot;You can use the free Java Mission Control instead to do this investigation, but I do not have our production JVMs configured to accept remote connections&quot; The free JMC is only for dev time not for production use, as far as I know -- not sure if Oracle has a way to determine what is dev and what is production.

excellent ! eye opener !<p>the last i faced this problem on a million request site, i just fixed the ulimit