wls

WLS（web ls）使得轻松抓取多个网站地图并列出 URL 变得容易。它甚至可以使用 robots.txt 自动找到域的网站地图。

用法

wls 接受多个域名/网站地图作为参数，并将所有找到的 URL 打印到标准输出

$ wls docs.rs > urls.txt

$ head -n 6 urls.txt 
https://docs.rs/A-1/latest/A_1/
https://docs.rs/A-1/latest/A_1/all.html
https://docs.rs/A5/latest/A5/
https://docs.rs/A5/latest/A5/all.html
https://docs.rs/AAAA/latest/AAAA/
https://docs.rs/AAAA/latest/AAAA/all.html

$ grep /all.html urls.txt | wc -l
113191
# that's a lot of crates!

如果参数不包含斜杠，则将其视为域名，wls 将自动尝试使用 robots.txt 查找网站地图。例如，docs.rs 在其 robots.txt 文件 中使用 Sitemap: 指令，所以以下命令是等效的

$ wls docs.rs
$ wls https://docs.rs/robots.txt
$ wls https://docs.rs/sitemap.xml

启用 -v/--verbose 时，wls 将将日志打印到标准错误

$ wls -v docs.rs
   Found 1 sitemaps
    in robotstxt with url: https://docs.rs/robots.txt

   Found 26 sitemaps
    in sitemap with url: https://docs.rs/sitemap.xml
    in robotstxt with url: https://docs.rs/robots.txt

   Found 15934 URLs
    in sitemap with url: https://docs.rs/-/sitemap/a/sitemap.xml
    in sitemap with url: https://docs.rs/sitemap.xml
    in robotstxt with url: https://docs.rs/robots.txt

   Found 11170 URLs
    in sitemap with url: https://docs.rs/-/sitemap/b/sitemap.xml
    in sitemap with url: https://docs.rs/sitemap.xml
    in robotstxt with url: https://docs.rs/robots.txt

  ...

还有更多选项可供选择

$ wls --help
Usage: wls [OPTIONS] <URLS>...

Arguments:
  <URLS>...  Domains/sitemaps to crawl

Options:
  -T, --timeout <SECONDS>  Maximum response time [default: 30]
  -w, --wait <SECONDS>     Delay between requests [default: 0]
  -v, --verbose            Enable logs
  -h, --help               Print help
  -V, --version            Print version