We all love continuous refactoring (... ok, not all).
But what happens when that becomes an obsession? I assume no code will never be 100% perfect. When should you stop refactoring? and ... continue ahead with more coding?
Refactor AS you continue ahead when you notice specifically bad code/design issues. Also if you find performance issues when profiling. Don't let refactoring be your primary task, keep it on a secondary track but running along side your primary task.
refactoring usually has diminishing returns, providing you're doing it well. it comes down to what will benefit you more? refactoring again for the 3rd time, which will take you even longer and/or result in less benefit than the previous 2 times, or something else like testing or development?