Lib.rs

用 Rust 编写的更好的去重工具。

基本用法： dedup <INPUT> [-o <OUTPUTFILE>]

运行 dedup --help 查看帮助信息

USAGE:
    dedup.exe [FLAGS] [OPTIONS] [INPUT]

FLAGS:
    -l, --count-lines        If flag is set only print the number of unique entries found.
        --mmap               Enables use of memory mapped files. This is enabled by default.
        --no-mmap            Prohibits usage of memory mapped files. This will slow down the deduplication process
                             significantly!
    -z, --zero-terminated    Specifies that entries should be intepreted as being separated by a null byte rather than a
                             newline.
    -h, --help               Prints help information
    -V, --version            Prints version information

OPTIONS:
    -o, --output <OUTPUT>
        --terminator <TERMINATOR>    Specifies the single-byte pattern to separate entries by. Default is newline.
                                     [default: \n]

ARGS:
    <INPUT>    Specifies the input file to read from. Omit or supply '-' to read from stdin.

要运行基准测试，运行 python benchsuite/benchrunner。这将下载一个大型（400MB+）的文本文件作为基准测试案例。

欢迎提出功能请求和错误报告！请在 Github 仓库中创建一个问题来提出。