lrtc：低资源文本分类

此crate是Jiang等（2023）中介绍的低资源文本分类方法的Rust实现。此实现允许您在各种压缩级别上选择gzip、zstd、zlib或deflate压缩算法。

use lrtc::{CompressionAlgorithm, classify};

let training = vec!["some normal sentence".to_string(), "godzilla ate mars in June".into(),];
let training_labels = vec!["normal".to_string(), "godzilla".into(),];
let queries = vec!["another normal sentence".to_string(), "godzilla eats marshes in August".into(),];
// Using a compression level of 3, and 1 nearest neighbor:
println!("{:?}", classify(&training, &training_labels, &queries, 3i32, CompressionAlgorithm::Gzip, 1usize));

此方法似乎对于相对稀疏的训练集表现良好，并且不需要像神经网络方法那样多的调整。

use csv::Reader;
use lrtc::{classify, CompressionAlgorithm};
use std::fs::File;
let imdb = File::open("./data/imdb.csv").unwrap();
let mut reader = Reader::from_reader(imdb);

    let imdb = File::open("./data/imdb.csv").unwrap();
    let mut reader = Reader::from_reader(imdb);

    let mut content = Vec::with_capacity(50000);
    let mut label = Vec::with_capacity(50000);
    for record in reader.records() {
        content.push(record.as_ref().unwrap()[0].to_string());
        label.push(record.unwrap()[1].to_string());
    }

let predictions = classify(
    &content[0..1000],
    &label[0..1000],
    &content[40000..50000],
    3i32,
    CompressionAlgorithm::Zstd,
    5usize,
)

let correct = predictions
    .iter()
    .zip(label[40000..50000].to_vec().iter())
    .filter(|(a, b)| a == b)
    .count();
println!("{}", correct as f64 / 1000f64)
// 0.623

参考文献

Zhiying Jiang，Matthew Yang，Mikhail Tsirlin，Raphael Tang，Yiqin Dai，和Jimmy Lin. 2023. “Low-Resource” Text Classification: A Parameter-Free Classification Method with Compressors. In Findings of the Association for Computational Linguistics: ACL 2023，第6810-6828页，多伦多，加拿大。计算语言学协会. https://aclanthology.org/2023.findings-acl.426