Rust 中的 RDF 数据集标准化

进行中

这是一个 RDF 标准化算法版本 1.0 (RDFC-1.0) 的 Rust 实现。其目的是为了理解和评估规范，并且 不打算用于生产环境。请注意，它目前非常不稳定，可能会发生未经通知的重大更改。

先决条件

请使用 Rust 1.70 或更高版本。

此实现依赖于处理 RDF 数据结构的 Oxrdf 预发布版本。如果您旨在标准化 N-Quads 文档而不是 Oxrdf 数据集，您还需要 Oxttl 来解析 N-Quads。请注意，这些 crate 目前仅作为预发布版本可用。

用法

将以下依赖项添加到您的 Cargo.toml 中

[dependencies]
rdf-canon = "0.15.0-alpha.6"
oxrdf = "0.2.0-alpha.6"
oxttl = "0.1.0-alpha.7"

然后，您可以使用 canonicalize 函数将 Oxrdf Dataset 转换为标准化的 N-Quads。

示例

use oxrdf::Dataset;
use oxttl::NQuadsParser;
use rdf_canon::canonicalize;
use std::io::Cursor;

let input = r#"_:e0 <http://example.org/vocab#next> _:e1 _:g .
_:e0 <http://example.org/vocab#prev> _:e2 _:g .
_:e1 <http://example.org/vocab#next> _:e2 _:g .
_:e1 <http://example.org/vocab#prev> _:e0 _:g .
_:e2 <http://example.org/vocab#next> _:e0 _:g .
_:e2 <http://example.org/vocab#prev> _:e1 _:g .
<urn:ex:s> <urn:ex:p> "\u0008\u0009\u000a\u000b\u000c\u000d\u0022\u005c\u007f" _:g .
"#;

let expected = r#"<urn:ex:s> <urn:ex:p> "\b\t\n\u000B\f\r\"\\\u007F" _:c14n0 .
_:c14n1 <http://example.org/vocab#next> _:c14n2 _:c14n0 .
_:c14n1 <http://example.org/vocab#prev> _:c14n3 _:c14n0 .
_:c14n2 <http://example.org/vocab#next> _:c14n3 _:c14n0 .
_:c14n2 <http://example.org/vocab#prev> _:c14n1 _:c14n0 .
_:c14n3 <http://example.org/vocab#next> _:c14n1 _:c14n0 .
_:c14n3 <http://example.org/vocab#prev> _:c14n2 _:c14n0 .
"#;

let input_quads = NQuadsParser::new()
    .parse_read(Cursor::new(input))
    .map(|x| x.unwrap());
let input_dataset = Dataset::from_iter(input_quads);
let canonicalized = canonicalize(&input_dataset).unwrap();

assert_eq!(canonicalized, expected);

高级用法

标准化图和四元组

我们提供了 canonicalize_graph 和 canonicalize_quads 函数来分别标准化 Graph 和 Vec<Quad>。例如，您可以使用 canonicalize_graph 如下标准化 RDF 图

use oxrdf::Graph;
use oxttl::NTriplesParser;
use rdf_canon::canonicalize_graph;
use std::io::Cursor;

let input = r#"_:e0 <http://example.org/vocab#next> _:e1 .
_:e0 <http://example.org/vocab#prev> _:e2 .
_:e1 <http://example.org/vocab#next> _:e2 .
_:e1 <http://example.org/vocab#prev> _:e0 .
_:e2 <http://example.org/vocab#next> _:e0 .
_:e2 <http://example.org/vocab#prev> _:e1 .
<urn:ex:s> <urn:ex:p> "\u0008\u0009\u000a\u000b\u000c\u000d\u0022\u005c\u007f" .
"#;

let expected = r#"<urn:ex:s> <urn:ex:p> "\b\t\n\u000B\f\r\"\\\u007F" .
_:c14n0 <http://example.org/vocab#next> _:c14n2 .
_:c14n0 <http://example.org/vocab#prev> _:c14n1 .
_:c14n1 <http://example.org/vocab#next> _:c14n0 .
_:c14n1 <http://example.org/vocab#prev> _:c14n2 .
_:c14n2 <http://example.org/vocab#next> _:c14n1 .
_:c14n2 <http://example.org/vocab#prev> _:c14n0 .
"#;

let input_triples = NTriplesParser::new()
    .parse_read(Cursor::new(input))
    .map(|x| x.unwrap());
let input_graph = Graph::from_iter(input_triples);
let canonicalized = canonicalize_graph(&input_graph).unwrap();

assert_eq!(canonicalized, expected);

在此，我们将输入图解释为包含它作为默认图的数据集，然后执行标准化算法。输出是从结果标准化数据集中获得的默认图。

标准化后的数据集

规范化算法还可以返回一个 规范化数据集，而不是序列化的规范化N-Quads。

RDF数据集规范化

一个 规范化数据集 是以下内容的组合

• 一个 RDF数据集 — 即 输入数据集，
• 输入空白节点标识符映射 — 将输入数据集中的空白节点映射到 空白节点标识符，以及
• 来自 规范化发行者 的 已发行标识符映射 — 将输入数据集中的标识符映射到规范化标识符

规范化数据集的具体序列化必须使用规范化 空白节点 标识符为所有空白节点打标签。

如果您希望使用规范化数据集，可以使用 issue 函数来获取 已发行标识符映射，该映射可以与 输入数据集（包含嵌入的 输入空白节点标识符映射）结合使用来构建规范化数据集。

use oxrdf::Dataset;
use oxttl::NQuadsParser;
use rdf_canon::issue;
use std::collections::HashMap;
use std::io::Cursor;

let input = r#"
_:e0 <http://example.org/vocab#next> _:e1 _:g .
_:e0 <http://example.org/vocab#prev> _:e2 _:g .
_:e1 <http://example.org/vocab#next> _:e2 _:g .
_:e1 <http://example.org/vocab#prev> _:e0 _:g .
_:e2 <http://example.org/vocab#next> _:e0 _:g .
_:e2 <http://example.org/vocab#prev> _:e1 _:g .
"#;

let expected = HashMap::from([
    ("g".to_string(), "c14n0".to_string()),
    ("e0".to_string(), "c14n1".to_string()),
    ("e1".to_string(), "c14n2".to_string()),
    ("e2".to_string(), "c14n3".to_string()),
]);

let input_quads = NQuadsParser::new()
    .parse_read(Cursor::new(input))
    .map(|x| x.unwrap());
let input_dataset = Dataset::from_iter(input_quads);
let issued_identifiers_map = issue(&input_dataset).unwrap();

assert_eq!(issued_identifiers_map, expected);

使用备用散列函数

RDF规范化算法版本1.0（RDFC-1.0） 使用内部散列函数来确定规范化数据集。虽然SHA-256被定义为默认散列函数，但它也允许在必要时使用备用散列函数。如果您想使用SHA-256以外的内部散列函数，可以使用如下所示的 canonicalize_with 函数。这里我们使用SHA-384而不是默认的SHA-256。

use oxrdf::Dataset;
use oxttl::NQuadsParser;
use rdf_canon::{canonicalize_with, CanonicalizationOptions};
use sha2::Sha384;
use std::io::Cursor;

let input = r#"_:e0 <http://example.org/vocab#next> _:e1 _:g .
_:e0 <http://example.org/vocab#prev> _:e2 _:g .
_:e1 <http://example.org/vocab#next> _:e2 _:g .
_:e1 <http://example.org/vocab#prev> _:e0 _:g .
_:e2 <http://example.org/vocab#next> _:e0 _:g .
_:e2 <http://example.org/vocab#prev> _:e1 _:g .
<urn:ex:s> <urn:ex:p> "\u0008\u0009\u000a\u000b\u000c\u000d\u0022\u005c\u007f" _:g .
"#;

let expected = r#"<urn:ex:s> <urn:ex:p> "\b\t\n\u000B\f\r\"\\\u007F" _:c14n0 .
_:c14n1 <http://example.org/vocab#next> _:c14n3 _:c14n0 .
_:c14n1 <http://example.org/vocab#prev> _:c14n2 _:c14n0 .
_:c14n2 <http://example.org/vocab#next> _:c14n1 _:c14n0 .
_:c14n2 <http://example.org/vocab#prev> _:c14n3 _:c14n0 .
_:c14n3 <http://example.org/vocab#next> _:c14n2 _:c14n0 .
_:c14n3 <http://example.org/vocab#prev> _:c14n1 _:c14n0 .
"#;

let input_quads = NQuadsParser::new()
    .parse_read(Cursor::new(input))
    .map(|x| x.unwrap());
let input_dataset = Dataset::from_iter(input_quads);
let options = CanonicalizationOptions::default();
let canonicalized = canonicalize_with::<Sha384>(&input_dataset, &options).unwrap();

assert_eq!(canonicalized, expected);

请注意，根据散列函数的选择，输出的规范化数据集可能会有所不同；这是因为散列函数会影响空白节点的顺序，进而影响规范化算法的输出。

防止毒害数据集

如 https://www.w3.org/TR/rdf-canon/#dataset-poisoning 中所述，存在一些恶意数据集会导致规范化算法消耗大量计算时间。我们为Hash N-Degree Quads算法的执行提供了调用限制，以防止它因毒害数据而无限期运行。默认限制设置为4000。如果您想提高或降低此限制，可以使用如下所示的 canonicalize_with 函数指定限制。

use oxrdf::Dataset;
use oxttl::NQuadsParser;
use rdf_canon::{canonicalize_with, CanonicalizationOptions};
use sha2::Sha256;
use std::io::Cursor;

let input = r#"_:e0 <http://example.org/vocab#next> _:e1 _:g .
_:e0 <http://example.org/vocab#prev> _:e2 _:g .
_:e1 <http://example.org/vocab#next> _:e2 _:g .
_:e1 <http://example.org/vocab#prev> _:e0 _:g .
_:e2 <http://example.org/vocab#next> _:e0 _:g .
_:e2 <http://example.org/vocab#prev> _:e1 _:g .
<urn:ex:s> <urn:ex:p> "\u0008\u0009\u000a\u000b\u000c\u000d\u0022\u005c\u007f" _:g .
"#;
let expected = r#"<urn:ex:s> <urn:ex:p> "\b\t\n\u000B\f\r\"\\\u007F" _:c14n0 .
_:c14n1 <http://example.org/vocab#next> _:c14n2 _:c14n0 .
_:c14n1 <http://example.org/vocab#prev> _:c14n3 _:c14n0 .
_:c14n2 <http://example.org/vocab#next> _:c14n3 _:c14n0 .
_:c14n2 <http://example.org/vocab#prev> _:c14n1 _:c14n0 .
_:c14n3 <http://example.org/vocab#next> _:c14n1 _:c14n0 .
_:c14n3 <http://example.org/vocab#prev> _:c14n2 _:c14n0 .
"#;

let input_quads = NQuadsParser::new()
    .parse_read(Cursor::new(input))
    .map(|x| x.unwrap());
let input_dataset = Dataset::from_iter(input_quads);
let options = CanonicalizationOptions {
    hndq_call_limit: Some(10000),
};
let canonicalized = canonicalize_with::<Sha256>(&input_dataset, &options).unwrap();

assert_eq!(canonicalized, expected);

调试日志功能

通过启用 log 功能，可以获取YAML格式的调试日志。

[dependencies]
rdf-canon = { version = "0.15.0-alpha.6", features = ["log"] }
oxrdf = "0.2.0-alpha.6"
oxttl = "0.1.0-alpha.7"

use oxrdf::Dataset;
use oxttl::NQuadsParser;
use rdf_canon::{canonicalize, logger::YamlLayer};
use std::io::Cursor;

// setup for debug logger
use tracing::metadata::LevelFilter;
use tracing_subscriber::prelude::*;
const INDENT_WIDTH: usize = 2;
fn init_logger(level: tracing::Level) {
    let _ = tracing_subscriber::registry()
        .with(YamlLayer::new(INDENT_WIDTH).with_filter(LevelFilter::from_level(level)))
        .try_init();
}

fn main() {
    // initialize debug logger
    init_logger(tracing::Level::DEBUG);

    let input = r#"_:e0 <http://example.com/#p1> _:e1 .
_:e1 <http://example.com/#p2> "Foo" .
"#;
    let expected = r#"_:c14n0 <http://example.com/#p1> _:c14n1 .
_:c14n1 <http://example.com/#p2> "Foo" .
"#;

    // get dataset from N-Quads document
    let input_quads = NQuadsParser::new()
        .parse_read(Cursor::new(input))
        .map(|x| x.unwrap());
    let input_dataset = Dataset::from_iter(input_quads);

    // canonicalize the dataset
    let canonicalized = canonicalize(&input_dataset).unwrap();

    assert_eq!(canonicalized, expected);
}

上述代码生成以下调试日志

ca:
  log point: Entering the canonicalization function (4.4.3).
  ca.2:
    log point: Extract quads for each bnode (4.4.3 (2)).
    Bnode to quads:
      e0:
        - _:e0 <http://example.com/#p1> _:e1 .
      e1:
        - _:e0 <http://example.com/#p1> _:e1 .
        - _:e1 <http://example.com/#p2> "Foo" .
  ca.3:
    log point: Calculated first degree hashes (4.4.3 (3)).
    with:
      - identifier: e0
        h1dq:
          log point: Hash First Degree Quads function (4.6.3).
          nquads:
            - _:a <http://example.com/#p1> _:z .
          hash: 24da9a4406b4e66dffa10ad3d4d6dddc388fbf193bb124e865158ef419893957
      - identifier: e1
        h1dq:
          log point: Hash First Degree Quads function (4.6.3).
          nquads:
            - _:z <http://example.com/#p1> _:a .
            - _:a <http://example.com/#p2> "Foo" .
          hash: a994e40b576809985bc0f389308cd9d552fd7c89d028c163848a6b2d33a8583a
  ca.4:
    log point: Create canonical replacements for hashes mapping to a single node (4.4.3 (4)).
    with:
      - identifier: e0
    hash: 24da9a4406b4e66dffa10ad3d4d6dddc388fbf193bb124e865158ef419893957
    canonical label: c14n0
      - identifier: e1
    hash: a994e40b576809985bc0f389308cd9d552fd7c89d028c163848a6b2d33a8583a
    canonical label: c14n1
  ca.5:
    log point: Calculate hashes for identifiers with shared hashes (4.4.3 (5)).
    with:
  ca.6:
    log point: Replace original with canonical labels (4.4.3 (6)).
    issued identifiers map: {e0: c14n0, e1: c14n1}
    hndq_call_counter:  { counter: 0, limit: 4000 }