What he said is, if you dont create tests first, refactoring is just changing shit. The behavior is supposed to be invariant across refactoring. The only way you can be sure is to test the before and after.
You'd also want to make sure you're getting some <i>benefit</i> from your refactoring activity. Are you winding up with less code? Is it more readable? Most important: are you getting closer to Don't Repeat Yourself?