riptide

quick, recursive, sitemap url extractor

riptide, with 16 workers, can rip through a 121,000+ link sitemap in less than 6 seconds.

nune@nunes-pc ~/sitemaps2> time riptide \
                                                              --sitemap https://www.************.com/sitemap.axd \
                                                              --output ****.txt \
                                                              --scheme https \
                                                              --workers 16 \
________________________________________________________
Executed in    5.99 secs      fish           external
   usr time  528.67 millis    1.14 millis  527.53 millis
   sys time  201.95 millis    0.21 millis  201.74 millis

installation and usage:

install (linux static binary, x86_64)

wget https://codeberg.org/gigirassy/riptide/releases/download/0.1.0/riptide && chmod +x riptide && mv riptide .local/bin/

install (cargo)

cargo install --git https://codeberg.org/gigirassy/riptide/

usage

riptide \
                               --sitemap https://example.com/sitemap.xml \
                               --output examplecomurls.txt \
                               --scheme https \
                               --workers 16 \
                               --user-agent "Mozilla/5.0 (X11; Linux i686; rv:151.0) Gecko/20100101 Firefox/151.0"

for searching for terms in urls after output is completed, i recommend ripgrep.

by default, riptide identifies itself in user agent, the above example overrides it.

license

isc