# blogfetch

tiny crystal web crawler to crawl a blogspot blog and all other blogspot blogs it links recursively until a set amount of unique blogspot blog urls (5000) is reached

this is especially helpful for archiveteam.

## how to
### ...build
1. clone this repo, cd to it.
2. (linux) ```docker run --rm -it -v .:/tmp/blogfetch 84codes/crystal:latest-alpine build /tmp/blogfetch/blogfetch.cr --release --static --no-debug -o /tmp/blogfetch/blogfetch && sudo chown 1000:1000 blogfetch && strip blogfetch``` will init it all and chown the resulting (enter your pass when prompted) to your user.
2. (other) install crystal from your package manager, then ``crystal build blogfetch.cr --release --static --no-debug -o blogfetch``, strip it (optional), then copy to anywhere in your path.
### ...use
1. find any blogspot blog with a blogroll.
2. affix it to the command, like ``blogfetch [url of that blog here]``.
3. affix ``> blogfetchoutput.txt`` to put output in a file.
4. wait for it to be done crawling, then transfer your file to the place of your choice.

## license
public domain when legally possible in your jurisdiction, otherwise 0BSD.