#哈希 #模糊 #ngrams #shingles

nightly fingles

基于哈希的文本相似度分析库

1个不稳定版本

0.1.0 2022年8月27日

#1788算法 分类中

GPL-3.0-or-later

16KB
107

fingles

schindel库的功能性重写,目标是使库更具可扩展性和开放性。性能是首要考虑,我只是认为这样一个简单的库不应该有太多的模板代码

算法描述请参考原始仓库的README

待办事项

  • 更合理的哈希(基于函数),支持多种哈希算法

示例

#![feature(generic_arg_infer)]
use fingles::*;

const HASH_LEN: usize = 100;
const NGRAM_LEN: usize = 5;

fn main() {
    let original = "\
        “My sight is failing,” she said finally. “Even when I was young I could not have read what was written there. \
        But it appears to me that that wall looks different. Are the Seven Commandments the same as they used to be, \
        Benjamin?” For once Benjamin consented to break his rule, and he read out to her what was written on the wall. \
        There was nothing there now except a single Commandment. It ran:\
        ALL ANIMALS ARE EQUAL BUT SOME ANIMALS ARE MORE EQUAL THAN OTHERS";

    let plagiarism = "\
        “My sight is failing,” she said finally. “When I was young I could not have read what was written there. \
        But it appears to me that that wall looks different. Are the Seven Commandments the same as they used to be” \
        Benjamin read out to her what was written. There was nothing there now except a single Commandment. \
        It ran: ALL ANIMALS ARE EQUAL BUT SOME ANIMALS ARE MORE EQUAL THAN OTHERS";

    let other = "\
        Throughout the spring and summer they worked a sixty-hour week, and in August Napoleon announced that there \
        would be work on Sunday afternoons as well. This work was strictly voluntary, but any animal who absented \
        himself from it would have his rations reduced by half. Even so, it was found necessary to leave certain \
        tasks undone. The harvest was a little less successful than in the previous year, and two fields which \
        should have been sown with roots in the early summer were not sown because the ploughing had not been \
        completed early enough. It was possible to foresee that the coming winter would be a hard one.";

    let original_hash = hash_from_string::<HASH_LEN, NGRAM_LEN>(original.chars());

    let plagiarism_hash = hash_from_string::<HASH_LEN, NGRAM_LEN>(plagiarism.chars());
    println!("plagiarism similarity: {}", compare_hashes::<_, NGRAM_LEN>(&original_hash, &plagiarism_hash));

    let other_hash = hash_from_string::<HASH_LEN, NGRAM_LEN>(other.chars());
    println!("other text similarity: {}", compare_hashes::<_, NGRAM_LEN>(&original_hash, &other_hash));
}

依赖项