lexer-generator

从正则表达式模式派生的 Lexer crate，具有用户自定义标记

示例：基本标记化

可能会用于计算器标记令牌的潜在代码

key.json:

{
    "literals": {
        "number": "[0-9]*(\\.[0-9]*){0, 1}",
        "subtract": "-",
        "add": "\\+",
        "divide": "/",
        "multiply": "\\*" 
    },
    "whitespace": "\n| |\r|\t"
}

main.rs:

let json: String = std::fs::read_to_string("key.json").unwrap();
let source: String = String::from("123 + 456 * 789");

let mut lexer = Lexer::from(json, source);
while !lexer.done() {
    println!("{}", lexer.next_token().unwrap());
}

number(123)
add(+)     
number(456)
multiply(*)
number(789)

lib.rs:

lexer-generator

此 crate 是一个小型 lexer 包，它从 JSON 解析

示例：基本标记化

可能会用于计算器标记令牌的潜在代码

key.json:

{
    "literals": {
        "number": "[0-9]*(\\.[0-9]*){0, 1}",
        "subtract": "-",
        "add": "\\+",
        "divide": "/",
        "multiply": "\\*" 
    },
    "whitespace": "\n| |\r|\t"
}

main.rs:

let json: String = std::fs::read_to_string("key.json").unwrap();
let source: String = String::from("123 + 456 * 789");

let mut lexer = Lexer::from(json, source);
// parsing, runtime, whatever one would want to do with their tokens

"123 + 456 * 789" -> Token("number", "123"), Token("add", "*"), Token("number", "456"), Token("multiply", "*"), Token("number", "789") // ignoring line position and the incremental nature of the lexer