mirror of
https://codeberg.org/gigirassy/blogfetch
synced 2026-08-30 23:37:42 +00:00
19 lines
1.1 KiB
Markdown
19 lines
1.1 KiB
Markdown
# blogfetch
|
|
|
|
tiny crystal web crawler to crawl a blogspot blog and all other blogspot blogs it links recursively until a set amount of unique blogspot blog urls (5000) is reached
|
|
|
|
this is especially helpful for archiveteam.
|
|
|
|
## how to
|
|
### ...build
|
|
1. clone this repo, cd to it.
|
|
2. (linux) ```docker run --rm -it -v .:/tmp/blogfetch 84codes/crystal:latest-alpine build /tmp/blogfetch/blogfetch.cr --release --static --no-debug -o /tmp/blogfetch/blogfetch && sudo chown 1000:1000 blogfetch && strip blogfetch``` will init it all and chown the resulting (enter your pass when prompted) to your user.
|
|
2. (other) install crystal from your package manager, then ``crystal build blogfetch.cr --release --static --no-debug -o blogfetch``, strip it (optional), then copy to anywhere in your path.
|
|
### ...use
|
|
1. find any blogspot blog with a blogroll.
|
|
2. affix it to the command, like ``blogfetch [url of that blog here]``.
|
|
3. affix ``> blogfetchoutput.txt`` to put output in a file.
|
|
4. wait for it to be done crawling, then transfer your file to the place of your choice.
|
|
|
|
## license
|
|
public domain when legally possible in your jurisdiction, otherwise 0BSD. |