diff --git a/README.md b/README.md index 00abc9e..fb6875f 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # blogfetch -tiny crystal web crawler to crawl a blogspot blog and all other blogspot blogs it links recursively until a set amount of urls (5000) is reached +tiny crystal web crawler to crawl a blogspot blog and all other blogspot blogs it links recursively until a set amount of unique blogspot blog urls (5000) is reached this is especially helpful for archiveteam.