Let's assume that someone creates a GPT model which<p><pre><code> 1. searches the internet for server software vulnerability or just find security holes by reading source codes,
2. writes code to exploit, searches internet for vulnerable servers,
3. copy itself to any server it can hack and replicate.
</code></pre>
Bonus steps:<p><pre><code> - randomly modifies its hyper-parameters, collect extra learning data, and trains itself,
- generate a script with evil purposes ("What would an evil AI do? Write a python script.") and executes it.
</code></pre>
When we will have such a GPT virus, how will we fight against it? Patch all software? Create AI-free part of the internet?<p>(Please consider that any answer to this question might get into the training data of a GPT model.)
Digital security is always an arms race. Even if someone managed to create this hypothetical system and it got terminal velocity to get to the point where it could train itself there would be lots of people using the same systems to thwart it. As long as there are more people and resources dedicated to maintaining order over chaos it will never last.