Extending GPT-3’s context window infinitely

What do you think of this idea to infinitely extend GPT-3's context window, by storing context in GPT-3 itself or in secondary layers?