I did this a couple of months back. I realized that WordPress adds a lot of overhead and a pretty large attack surface, and the only benefit it was giving me was a medium amount of convenience.<p>In other words, my cost/benefit ratio for WP was very unfavorable, so I ditched it. Right now, I've fallen back to largely static sites that use a minimalist CMS that I wrote over a decade ago. Eventually, I'll settle on a better solution.
Yes, I've been moving in this direction as well for personal blogging. Markdown is nicer to write than HTML, especially for lists, but HTML feels workable. Just copy a template to start a new post so you only need to write the content.<p>I don't think the "edit on GitHub" experience is that great, you can't preview changes in any efficient way. That works ok if you are just writing text, instead of tweaking format.