Want to sanity-check the generated nut-website

[jimklimov]

I have a FOSS project whose web site is generated by asciidoc and some custom scripts as an horde (thousands) of static files locally in the source files' repo, copied into another workspace and uploaded to github.io style repository, and eventually is rendered as an HTTP server for browsers around the world to see.

Users occasionally report that some of the links between site pages end up broken (lead nowhere).

The website build platform is generally POSIX-ish, although most often the agent doing the regular work is a Debian/Linux one. Maybe the platform differences cause the "page outages"; maybe this bug is platform-independent.

I had a thought about crafting a check for the two local directories as well as the resulting site to crawl all relative links (and/or absolute ones starting with its domain name(s)), and report any broken pages so I could focus on finding why they fail and/or avoiding publication of "bad" iterations - same as with compilers, debuggers and warnings elsewhere.

The general train of thought is about using some wget spider mode, though any other command-line tool (curl, lynx...), python script, shell with sed, etc. would do as well. Surely this particular wheel has been invented too many times for me to even think about making my own? A quick and cursory googling session while on commute did not come up with any good fit however.

So, suggestions are welcome :)

Posted as a question at https://unix.stackexchange.com/questions/775994/how-to-check-consistency-of-a-generated-web-site-using-recursive-html-parsing

[jimklimov]

One promising suggestion was https://github.com/gjtorikian/html-proofer (packaged in Debian in ruby-html-proofer) - it does at least report a few hundred issues, inside the site and outside it (with third-party pages we refer to), so definitely something for me to chew on :)