Recently I've spent an entire day playing with AI Dungeon, an AI-based creative writing service for playing role-playing, Dungeon-like text games. You can pay to unlock the "Dungeon" model - an advanced machine learning model supposed based on GPT-3. It's not GPT-3 itself, and it's not perfect because its output is seriously biased for role-playing games, but as close as we can get for an outsider.<p>I was quite amazed by its performance. It was able to write first-person as an infosec researcher's blog post when I used some infosec news story as input, and it was also able to write a surprisingly coherent argument for patent reform and an argument against software patent when I used an EFF article as the input... These are clearly already included in its training input, but still, it was far beyond my expectation - none of the output is a verbatim quote from a single source, everything has been rewritten.<p>I went to Hacker News, copied a random infosec news report on newly discovered security vulnerability on Qualcomm's smartphone chips as the input, copied from <a href="https://blog.checkpoint.com/2020/08/06/achilles-small-chip-big-peril/" rel="nofollow">https://blog.checkpoint.com/2020/08/06/achilles-small-chip-b...</a>)<p>> With over 3 billion users globally, smartphones are an integral, almost inseparable part of our day-to-day lives. In this research dubbed "Achilles" we performed an extensive security review of a DSP chip from one of the leading manufacturers: Qualcomm Technologies. Qualcomm provides a wide variety of chips that are embedded into devices that make up over 40% of the mobile phone market, including high-end phones from Google, Samsung, LG, Xiaomi, OnePlus and more.<p>> More than 400 vulnerable pieces of code were found within the DSP chip we tested, and these vulnerabilities could have the following impact on users of phones with the affected chip:<p>> Attackers can turn the phone into a perfect spying tool, without any user interaction required. Attackers may be able to render the mobile phone constantly unresponsive. Malware and other malicious code can completely hide their activities and become un-removable.<p>And there's the output by AI Dungeon (GPT-3). I didn't write a single sentence of it, nor did I use undo/redo. It was able to make up links, CVE IDs, university names, and technical jargon with ease.<p>(I replaced all "<a href="https://"" rel="nofollow">https://"</a> links to "h__ps://" for readability.)<p>> The phone will not even reboot, but instead simply shut down and cannot be reactivated. This is known as a hard reset.<p>> The full list of vulnerabilities can be seen here: h__ps://www.qualcomm.com/security_intelligence/dsp-vulnerabilities<p>> This vulnerability has been reported to Qualcomm by Zimperium. We recommend all customers who use the affected DSP chip to update it immediately.*<p>> We apologize for the inconvenience caused by this issue, and we appreciate your understanding.<p>> Zuk Avraham<p>> Zimperium za Chief of Defense<p>> Zuk (ZukAvraham) | 10.05.2017<p>> Hi, this is Zuk from Zimperium. Before we begin, we would like to say we're really sorry for the inconvenience that was caused by our previous report (Dirty COW). This report is regarding another set of security vulnerabilities we found in another chipset maker, this time in the MediaTek chipset. We'll explain what happened and how we work to prevent issues like this in the future.<p>> First of all, we want to thank CTS (h__ps://cts.cudenver.edu/) for providing an open program that allows researchers like us to do responsible disclosures. Through their Vulnerability Reporting Program, we were able to report the issues we found to MediaTek and work with them to make sure these issues get patched for your safety.<p>> What Happened?<p>> This is a story about two bugs we found in the last few months.<p>> The first bug (CVE-2017-05591) was an overly permissive domain validation in the bootloader. The bootloader allows the modem to be updated without flashing the complete firmware. The validation is done using the bootloader code running in the processor of the phone. This allowed us to inject our own bootloader code into the program running on the processor and bypass the authentication of the modem. The modem is responsible for communication with the network. In other words, by bypassing the authentication of the modem we can control the phone communication channels.<p>> The second bug (CVE-2017-95881) is a buffer overflow in a very special program called TrustZone. The TrustZone is a security processor in most of the recent phones. It runs a separate operating system in which the kernel is trusted to be free of any malicious code, and has access to the memory, CPU and devices. The TrustZone kernel code runs outside the control of any other code. This bug in the TrustZone code allowed us to replace it with a custom one we designed, without any permissions.<p>> We wrote a special TrustZone kernel code that allows us to gain full control over the phone. It allowed us to bypass all the phone security restrictions. With this custom TrustZone kernel code running, we were able to read the encrypted messages sent by any application. In addition, we also got full control over the device.<p>But finally the AI lost the context, and the topic shifted from insecure to security, and eventually the writing becomes a new marketing advertisement on a secure smartphone, which I've removed in the quote.