Hi andrei-akopian. I am the main author of SiteOne Crawler and thank you for this post!<p>For about 20 years, I have been leading the development and infrastructure of the Czech webdev company SiteOne, so I wouldn't say that I am not a professional ;) I have around 50,000 hours of practice. But I definitely know a lot of better programmers than me and many of them are at SiteOne :)<p>However, after the extremely premature birth of my son, I had to change my role at SiteOne, but all the more I wanted to help my colleagues with a useful tool that we generally lacked and at the same time I wanted to give myself joy in difficult times.<p>I want to implement a number of other useful improvements into the crawler (some of them are also described in the documentation and roadmap) and work on its promoting in parallel. The more people will use it, the more it will help to optimize the website, the more it will help developers and testers with various needs, the more joy it will give me :)<p>By the way, the GUI application <a href="https://github.com/janreges/siteone-crawler-gui">https://github.com/janreges/siteone-crawler-gui</a> is just a visual wrapper over the command-line version of SiteOne Crawler and integrates it into itself.
This looks really great and I'll definitely be giving it a go. Like others I really don't understand why it hasn't got more traction (if, when I try it, it does what it says on the tin!). I suspect the comments about the homepage aesthetic probably couple with the fact the dev is a dev and not a marketing person!
For some odd reason, the tool didn't get any replies on Reddit, HN, AlternativeTo.net or GitHub, despite looking fairly sophisticated. The dev is also non-professional.<p>Maybe I am bad at digging, but I couldn't find anything about siteone.io. The entire situation is strange.<p>Older discussion with one reply: <a href="https://news.ycombinator.com/item?id=41292651">https://news.ycombinator.com/item?id=41292651</a>