vaporetto_rules

Vaporetto是一个快速轻量级的基于点预测的tokenizer。vaporetto_rules是Vaporetto的基于规则的过滤器。

示例

use std::fs::File;
use std::io::BufReader;
use std::rc::Rc;

use vaporetto::{CharacterType, Model, Predictor, Sentence};
use vaporetto_rules::{
    SentenceFilter, StringFilter,
    sentence_filters::{ConcatGraphemeClustersFilter, KyteaWsConstFilter},
    string_filters::KyteaFullwidthFilter,
};

let mut f = BufReader::new(File::open("model.bin").unwrap());
let model = Model::read(&mut f).unwrap();
let mut predictor = Predictor::new(model, false).unwrap();

let pre_filters: Vec<Box<dyn StringFilter<String>>> = vec![
    Box::new(KyteaFullwidthFilter),
];
let post_filters: Vec<Box<dyn SentenceFilter>> = vec![
    Box::new(ConcatGraphemeClustersFilter),
    Box::new(KyteaWsConstFilter::new(CharacterType::Digit)),
];

let input = "Vaporettoは仲良し家族👨‍👨‍👧‍👦を離れ離れにさせません。"
    .to_string();

let preproc_input = pre_filters.iter().fold(input, |s, filter| filter.filter(s));

let mut sentence = Sentence::from_raw(preproc_input).unwrap();
predictor.predict(&mut sentence);

post_filters.iter().for_each(|filter| filter.filter(&mut sentence));

let mut buf = String::new();
sentence.write_tokenized_text(&mut buf);
assert_eq!(
    "Ｖａｐｏｒｅｔｔｏ は 仲良 し 家族 👨‍👨‍👧‍👦 を 離れ離れ に さ せ ま せ ん 。",
    buf,
);

许可证

许可协议为以下之一

• Apache许可证2.0版本 (LICENSE-APACHE 或 https://apache.ac.cn/licenses/LICENSE-2.0)
• MIT许可证 (LICENSE-MIT 或 http://opensource.org/licenses/MIT)

任选其一。

贡献

除非你明确表示，否则任何有意提交并包含在作品中的贡献，如Apache-2.0许可证中定义的，将按照上述方式双许可，不附加任何额外的条款或条件。