We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

This problem is not going to go away so easily. Numerous core Java classes (like BufferedInputStream) use synchronized. I count 1600+ usages in java.base. The blocking issue means it&#x27;s _much_ easier to accidentally run into this, rather than waving it away as an unlikely edge case.<p>I personally ran into this Using the built in com.sun webserver, with a virtual thread executor. My VPS only has two CPUs which means the FJP that virtual threads run on only have 2 active threads at a time. I ran into this hang when some of the connection hung, blocking any further requests from being processed.

Totally off topic but I am getting tired of the AI generated images used on nearly all blog posts nowadays. They are instantly recognisable, it just seems low effort and lowers the feeling of quality one might otherwise have

It is a known caveat that virtual threads do not work well with long running synchronization by pinning the thread. That unfortunately means that for many applications it may be premature to adopt them, but it is mature enough for broader evaluation by the libraries and frameworks. The Java team provided a status of their efforts recently [1].<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=WoQJnnMIlFY&amp;t=421s" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=WoQJnnMIlFY&amp;t=421s</a>

Curious if you considered switching to a different connection pooling library. These days I usually use HikariCP which is fast an actively maintained. c3p0 hasn&#x27;t had any activity for years, I&#x27;m not sure if it&#x27;s still maintained.

Go has a mechanism to spawn a new thread (m in ho runtime parlance) if it thinks one of its threads might be blocked in a cgo (go’s “native function” equivalent). That prevents stuff like this.

The warning shots across the bow where heard with this statement from the devs:<p>&quot;Don&#x27;t replace platform&#x2F;native threads with virtual ones, replace tasks (without further explanation) instead&quot;?!<p>Combine that with the fact that they chose to implement the scheduler in Java instead of C(++) and you&#x27;re set for performance problems.<p>Remember that NIO took from 1.5 to 1.7 to be usable&#x2F;performant and that was native!<p>Edit: Finally figured out why: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39010648">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39010648</a>

This is a common problem when migrating a system from threads to virtual threads. In general, using primitives which block the current thread and prevent forward progress can quickly lead to deadlocks. It’s a hard issue to catch because in the past usually this would get “solved” by spawning a new thread to complete the task but in a world with virtual threads the runtime is usually reluctant to spawn more threads, so there’s nothing that can service more work if you’ve blocked all the threads.

I am extremely confused.<p>&gt; There are two scenarios in which a virtual thread cannot be unmounted during blocking operations because it is pinned to its carrier:<p>&gt; When it executes code inside a synchronized block or method<p>Isn&#x27;t &#x27;synchronized&#x27; effectively sugar for taking a kind of lock? Why can&#x27;t it be treated uniformly by the scheduler?

Did they get a deadlock again? <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=38939165">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=38939165</a>

I wonder if HikariCP, currently the best Java DB Connection Pooling library, suffer the same issue as c3p0.

Why not treat this as a bug, and fix it in Java 21? For compliance reasons we can only use LTS versions, and the next one isn&#x27;t until September 2025, according to <a href="https:&#x2F;&#x2F;www.oracle.com&#x2F;java&#x2F;technologies&#x2F;java-se-support-roadmap.html" rel="nofollow">https:&#x2F;&#x2F;www.oracle.com&#x2F;java&#x2F;technologies&#x2F;java-se-support-roa...</a>.

Seems to be a similiar problem field as writing blocking functions that call async functions in C# and co-existence of synchronous and asynchronous code.<p>There are numerous recommendations such as<p><a href="https:&#x2F;&#x2F;learn.microsoft.com&#x2F;en-us&#x2F;archive&#x2F;msdn-magazine&#x2F;2015&#x2F;july&#x2F;async-programming-brownfield-async-development#the-blocking-hack" rel="nofollow">https:&#x2F;&#x2F;learn.microsoft.com&#x2F;en-us&#x2F;archive&#x2F;msdn-magazine&#x2F;2015...</a><p>Final phase is &quot;I hope these techniques will help you adopt async into your existing applications in a way that works best for you.&quot;

I thought the blog was great but &quot;in summary&quot; conclusion bad.<p>The summary merely stated that Java virtual thread are great. I expected a summary of the problem and solution, for example something like:<p>When using Java 21 virtual threads, you can end-up starved of carrier threads due to all carrier threads waiting on a pool exhausted resources with no thread available to free such resources. The solution is to wrap those resources in a virtual-thread aware object. In our case, we solved our problem by wrapping connections in semaphores.

Concurrency, parallelism. These are among the most misunderstood concepts in programming&#x2F;software development.<p>TLS (especially mutual TLS) and Oauth also join this club.

Why synchronized block are not preemptible? When compiling<p>public void syncMethod() {<p><pre><code> synchronized(lock) { &#x2F;&#x2F; some code } </code></pre> }<p>they could translate to<p>public void syncMethod() {<p><pre><code> await reentrantLockAsync.lockAsync(); try { await somecodeAsync(); } finally { await lock.unlockAsync() } }</code></pre>

Why was c3p0 used (its latest version was released in Dec 2019)? Those tests existed for a while and people were too lazy to replace c3p0 with something newer? I guess that they spent all their time to use virtual threads in those tests and had no time left to look at c3p0.

Why couldn&#x27;t JVM detect when all carrier threads are blocked, and just spawn more of them?

Sounds similar to the quirks you get with TPL in .NET under some circumstances. For library code, a ConfigureAwait(false) invoke should be considered to signify that the execution does not need to resume on the original thread.

Can someone explain what these virtual threads are in 2 sentences please?

Why dining philosophers from the image have more than two hands?

BTW, dining philosophers is an extremely clunky example.

Java virtual threads did not cause a deadlock here.<p>The deadlock was a usage error.<p>A better title would be: Naively switching to Java virtual threads caused a deadlock in TPC-C for Progress SQL.

&quot;Our PostgreSQL TPC-C implementation utilizes c3p0 for connection pooling...The problem is that...synchronized code might be deeply embedded within the libraries you use. In our case, it was within the c3p0 library. So, the fix is straightforward: we simply wrapped the connection with a java.util.concurrent.Semaphore. With this change, virtual threads are blocked on the semaphore and, crucially, release the carrier thread instead of delving inside c3p0. Thus, we never block inside c3p0 because we enter c3p0 code only when there is a free session available.&quot;

&quot;Why not to use java virtual threads&quot; -&gt; Fundamental features of the language treated as implementation details in libraries for more than two decades cause deadlock.

Personally I am curious how these features like virtual threads are tested when developed.

I think i have a more elegant solution for this deadlock:<p>&quot;Switch to haskell&quot;.

Erlang is based on virtual threads (confusingly called processes). The Erlang virtual machine schedules them on OS threads. Erlang processes communicate using message passing, preventing deadlocks. You can use millions of Erlang processes without problems, e.g., to handle millions of Elixir LiveView sessions.