blogfetch/README.md

blogfetch

tiny crystal web crawler to crawl a blogspot blog and all other blogspot blogs it links recursively until a set amount of urls (5000) is reached

this is especially helpful for archiveteam.

how to

...build

1. clone this repo, cd to it.
2. (linux) docker run --rm -it -v .:/tmp/blogfetch 84codes/crystal:latest-alpine build /tmp/blogfetch/blogfetch.cr --release --static --no-debug -o /tmp/blogfetch/blogfetch && sudo chown 1000:1000 blogfetch && strip blogfetch will init it all and chown the resulting (enter your pass when prompted) to your user.
3. (other) install crystal from your package manager, then crystal build blogfetch.cr --release --static --no-debug -o blogfetch, strip it (optional), then copy to anywhere in your path.

...use

1. find any blogspot blog with a blogroll.
2. affix it to the command, like blogfetch [url of that blog here].
3. affix > blogfetchoutput.txt to put output in a file.
4. wait for it to be done crawling, then transfer your file to the place of your choice.

license

public domain when legally possible in your jurisdiction, otherwise 0BSD.