ar_row-rs

针对 Apache Arrow 的行访问

目前，它只允许读取数组，而不允许构建它们。

Arrow 是一种列式数据存储格式，旨在存储在内存中。虽然列式非常高效，但使用起来可能比较麻烦，因此此 crate 通过将列“压缩”成经典的 Rust 结构来提供在行上工作的功能。

此 crate 是由 ORC 解析库 orcxx 分支出来的，通过删除对底层 ORC C++ 库的绑定，并重写高级 API 以操作 Arrow 而不是 ORC 特定结构。

ar_row_derive crate 提供了一个自定义的 derive 宏。

extern crate ar_row;
extern crate ar_row_derive;
extern crate orc_rust;

use std::fs::File;
use std::num::NonZeroU64;

use orc_rust::projection::ProjectionMask;
use orc_rust::{ArrowReader, ArrowReaderBuilder};

use ar_row::deserialize::{ArRowDeserialize, ArRowStruct};
use ar_row::row_iterator::RowIterator;
use ar_row_derive::ArRowDeserialize;

// Define structure
#[derive(ArRowDeserialize, Clone, Default, Debug, PartialEq, Eq)]
struct Test1 {
    long1: Option<i64>,
}

// Open file
let orc_path = "../test_data/TestOrcFile.test1.orc";
let file = File::open(orc_path).expect("could not open .orc");
let builder = ArrowReaderBuilder::try_new(file).expect("could not make builder");
let projection = ProjectionMask::named_roots(
    builder.file_metadata().root_data_type(),
    &["long1"],
);
let reader = builder.with_projection(projection).build();
let rows: Vec<Option<Test1>> = reader
    .flat_map(|batch| -> Vec<Option<Test1>> {
        <Option<Test1>>::from_record_batch(batch.unwrap()).unwrap()
    })
    .collect();

assert_eq!(
    rows,
    vec![
        Some(Test1 {
            long1: Some(9223372036854775807)
        }),
        Some(Test1 {
            long1: Some(9223372036854775807)
        })
    ]
);

RowIterator API

此 API 允许在记录批次之间重用缓冲区，但需要使用 RecordBatch 而不是 Result<RecordBatch, _> 作为输入。

extern crate ar_row;
extern crate ar_row_derive;
extern crate orc_rust;

use std::fs::File;
use std::num::NonZeroU64;

use orc_rust::projection::ProjectionMask;
use orc_rust::{ArrowReader, ArrowReaderBuilder};

use ar_row::deserialize::{ArRowDeserialize, ArRowStruct};
use ar_row::row_iterator::RowIterator;
use ar_row_derive::ArRowDeserialize;

// Define structure
#[derive(ArRowDeserialize, Clone, Default, Debug, PartialEq, Eq)]
struct Test1 {
    long1: Option<i64>,
}

// Open file
let orc_path = "../test_data/TestOrcFile.test1.orc";
let file = File::open(orc_path).expect("could not open .orc");
let builder = ArrowReaderBuilder::try_new(file).expect("could not make builder");
let projection = ProjectionMask::named_roots(
    builder.file_metadata().root_data_type(),
    &["long1"],
);
let reader = builder.with_projection(projection).build();
let mut rows: Vec<Option<Test1>> = RowIterator::new(reader.map(|batch| batch.unwrap()))
    .expect("Could not create iterator")
    .collect();

assert_eq!(
    rows,
    vec![
        Some(Test1 {
            long1: Some(9223372036854775807)
        }),
        Some(Test1 {
            long1: Some(9223372036854775807)
        })
    ]
);

嵌套结构

上述两个示例也适用于嵌套结构

extern crate ar_row;
extern crate ar_row_derive;

use ar_row_derive::ArRowDeserialize;

#[derive(ArRowDeserialize, Default, Debug, PartialEq)]
struct Test1Option {
    boolean1: Option<bool>,
    byte1: Option<i8>,
    short1: Option<i16>,
    int1: Option<i32>,
    long1: Option<i64>,
    float1: Option<f32>,
    double1: Option<f64>,
    bytes1: Option<Box<[u8]>>,
    string1: Option<String>,
    list: Option<Vec<Option<Test1ItemOption>>>,
}

#[derive(ArRowDeserialize, Default, Debug, PartialEq)]
struct Test1ItemOption {
    int1: Option<i32>,
    string1: Option<String>,
}