Don’t “Fake It Till You Make It”; Instead, Remember What You Know