CC-Downloader

这是一个用Rust编写的实验性礼貌下载器，用于下载Common Crawl数据。目前它从Cloudfront下载Common Crawl数据。

待办事项

• 添加重试支持
• 添加Python绑定
• 添加测试
• 重构CLI子命令
• 添加对s3的支持

使用方法

Usage: cc-downloader [COMMAND]

Commands:
  download-paths  Download paths for a given snapshot
  download        Download files from a crawl
  help            Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version

------

cc-downloader download -h                                                             
Download files from a crawl

Usage: cc-downloader download --path-file <PATHS> --output <OUTPUT> [PROGRESS]

Arguments:
  [PROGRESS]  Print progress #[arg(short, long)] [possible values: true, false]

Options:
      --path-file <PATHS>  Path file
  -o, --output <OUTPUT>    Otput folder
  -h, --help               Print help

------

cc-downloader download-paths -h                                                               
Download paths for a given snapshot

Usage: cc-downloader download-paths --snapshot <SNAPSHOT> --data-type <PATHS> --output <OUTPUT> [PROGRESS]

Arguments:
  [PROGRESS]  Print progress #[arg(short, long)] [possible values: true, false]

Options:
      --snapshot <SNAPSHOT>  Crawl reference
      --data-type <PATHS>    Data type
  -o, --output <OUTPUT>      Otput folder
  -h, --help                 Print help