I think LLMs have a lot of therapeutic potential, but I am worried about how easily they can cause harm in this area. CBT is the gold standard treatment for many problems, but applying it correctly requires some nuance and an accurate diagnosis of the problem.<p>Consider a teenager who is fixated on the idea that they're a "bad person" because they're having homosexual thoughts. You might think they need positive affirmations or exercises to challenge their fears with evidence. But these symptoms are sometimes a manifestation of OCD. If so, using CBT to "argue away" the fear could end up reinforcing the OCD cycle and causing more and more self-doubt. For this person, the better treatment would be a different CBT tool: exposure. It may seem odd, but they would be better served with exercises such as repeatedly thinking sexual thoughts and then telling themselves, "I might be a horrible person because of this." (The purpose is to desensitize them to the idea - eventually, they will just get bored.) Needless to say, this type of treatment needs to be implemented with care.<p>I think it's beyond the capabilities of LLMs to reliably distinguish problems like this. So then, I think the systems have to be designed so that their output is at least harmless for all people, and that sounds really hard.