ai_utils

• 目的：此库提供了简单的AI工具，用于神经网络关联分析和连接OpenAI JSON模式和批量处理API。
• 功能:
  • 使用与Levenshtein距离计算和模糊匹配相关的原生Rust实现进行简单AI-like分析
  • 与启用JSON模式的OpenAI模型交互
  • 与启用批量处理的OpenAI模型交互

public_url_utils

• 目的：此库提供简单工具，用于从流行的公共接口（如可公开查看的Google Sheets）检索数据。
• 功能:
  • 从Google Sheets中检索数据

api_utils

• 目的：优雅地进行并缓存API调用。
• 功能:
  • ApiCallBuilder：轻松进行并缓存API调用，并管理缓存数据以实现高效的API使用。

python_utils

• 目的：Python是互操作性的爱情语言，非常适合RUST与其他语言编写的库良好协作。此实用工具包含RGWML在裸金属上运行的python脚本和pip包，以简化与XGBOOST、Clickhouse、Google Big Query等集成的调试。
• 功能:
  • DB_CONNECT_SCRIPT：存储便于Google Big Query和Clickhouse集成的db_connect.py脚本。
  • DC_CONNECT_SCRIPT：存储便于H5文件解析集成的dc_connect.py脚本，以及从数据容器中提取元数据的实用工具。
  • XGB_CONNECT_SCRIPT：存储便于XGBOOST集成的xgb_connect.py脚本。
  • DASK_GROUPER_CONNECT_SCRIPT：连接Python Dask API，便于实现复杂但内存效率高的数据分组功能。
  • DASK_PIVOTER_CONNECT_SCRIPT：连接Python Dask API，便于实现复杂但内存效率高的数据转换功能。

2. 重要！通过安装裸金属依赖项开始

与其他坚持使用原生Rust的方法不同，RGWML利用Python作为API来扩展Rust的功能，这些方法往往使简单任务变得复杂，与10-15行Python代码相比。RGWML假设如果Rust是牛排，Python就是土豆。土豆永远不应该成为任何烹饪活动的中心，就像牛排永远不会扮演次要角色一样。这种方法也适用于在原始“第一”语言中表现最好/经过长时间尝试和测试的软件依赖。

RGWML需要以下UNIX系统库

sudo apt-get update
sudo apt-get install python3-pip libxgboost-dev libhdf5-dev

以及以下RGWML的python依赖项，按设计需要在裸金属上安装（虚拟环境被高估了）

pip3 install google-cloud-bigquery clickhouse-driver pandas xgboost scikit-learn numpy h5py tables dask dask[dataframe] dask[distributed]

python_utils实用工具包含RGWML在裸金属上运行的python脚本和pip包，以简化与XGBOOST、scikit-learn、Clickhouse、Google Big Query等集成的调试。RGWML自动放置和更新这些脚本，对应于包的版本号，存储在/home/RGWML/executables/。

• DB_CONNECT_SCRIPT：存储便于Google Big Query和Clickhouse集成的db_connect.py脚本。
• XGB_CONNECT_SCRIPT：存储便于XGBOOST集成的xgb_connect.py脚本。
• CLUSTERING_CONNECT_SCRIPT：存储便于scikit-learn集成的clustering_connect.py脚本。
• DASK_GROUPER_CONNECT_SCRIPT：连接Python Dask API，便于实现复杂但内存效率高的数据分组功能。

3. csv_utils

csv_utils模块包含一组旨在简化与CSV文件相关的各种任务的实用工具。这些实用工具包括用于创建和管理CSV文件的CsvBuilder、用于将JSON数据转换为CSV格式的CsvConverter以及用于高效数据缓存和检索的CSV结果缓存。每个实用工具都针对在不同场景下处理CSV数据的生产力和便捷性进行了定制。

• CsvBuilder：提供了一种流畅的接口，用于创建、分析和保存CSV文件。它简化了与CSV数据的交互，无论从头开始还是修改现有文件等。

• CsvConverter：提供了一种将JSON数据转换为CSV格式的方法。此实用工具特别适用于处理和保存JSON API响应为CSV文件，提供了一种简单直接的数据转换方法。要使用CsvConverter，只需使用JSON数据和所需的输出文件路径作为参数调用from_json方法。

• CSV结果缓存：有助于避免不必要的重复数据生成。想象一下你有一个记录每日温度的CSV文件。你不想每次访问它时都生成此文件，尤其是在一天中数据变化不大的情况下。

CsvBuilder

实例化

示例1：创建一个新的对象

use rgwml::csv_utils::CsvBuilder;

let builder = CsvBuilder::new()
    .set_header(&["Column1", "Column2", "Column3"])
    .add_rows(&[&["Row1-1", "Row1-2", "Row1-3"], &["Row2-1", "Row2-2", "Row2-3"]])
    .save_as("/path/to/your/file.csv");
builder.print_table();

示例2：从现有文件加载

use rgwml::csv_utils::CsvBuilder;

let builder = CsvBuilder::from_csv("/path/to/existing/file.csv");
builder.print_table();

示例3：从公开可查看的Google Sheets URL加载

use rgwml::csv_utils::CsvBuilder;
use tokio::runtime::Runtime;

let rt = Runtime::new().unwrap();
rt.block_on(async {
    let csv_builder = CsvBuilder::from_publicly_viewable_google_sheet("https://docs.google.com/spreadsheets/d/1U9ozNFwV__c15z4Mp_EWorGwOv6mZPaQ9dmYtjmCPow/edit#gid=272498272").await;

    csv_builder.print_table();
});

示例4：从裸金属Python可执行文件加载

# A bare metal python executable should:
# - Be executable without an virtual environment, with the `python3 <file_name>.py <arguments>` format;
# - Specify a --uid flag accepting a string value, for the library to retrieve the output correctly
# - Save the output to rgwml_{uid}.json file
# For instance:

import os
import argparse
import json
import mmap
import pandas as pd

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Process some data")
    parser.add_argument('--uid', type=str, help='A unique identifier to name the output json file', required=True)

    parser.add_argument('--file_a_path', type=str, required=True, help='Path to the first CSV file')
    parser.add_argument('--file_b_path', type=str, required=True, help='Path to the second CSV file')
    parser.add_argument('--join_type', type=str, required=True, choices=['LEFT_JOIN', 'RIGHT_JOIN', 'OUTER_FULL_JOIN', 'UNION', 'BAG_UNION'], help='Type of join operation to perform')
    parser.add_argument('--file_a_ref_column', type=str, required=False, help='Reference column in the first CSV file')
    parser.add_argument('--file_b_ref_column', type=str, required=False, help='Reference column in the second CSV file')


    # Some processing logic that creates `df`

    headers = df.columns.tolist()
    rows = df.values.tolist()

    output = {
        "headers": headers,
        "rows": [[str(item) for item in row] for row in rows],
    }

    json_output = json.dumps(output, indent=4)
    filename = f"rgwml_{uid}.json"

    with open(filename, 'wb') as f:
        f.write(b' ' * len(json_output))

    with open(filename, 'r+b') as f:
        mm = mmap.mmap(f.fileno(), 0)
        mm.write(json_output.encode('utf-8'))
        mm.close()

// Now, you can load the result of the above directly into your RGWML workflow, in the manner shown below.

use rgwml::csv_utils::CsvBuilder;
use tokio::runtime::Runtime;
use std::path::PathBuf;

let current_dir = std::env::current_dir().unwrap();
let executable_path = current_dir.join("python_executables/dask_joiner_connect.py");
let executable_path_str = executable_path.to_str().unwrap();

// Append the file name to the directory path
let csv_path_a = current_dir.join("test_file_samples/joining_test_files/join_file_a.csv");
let csv_path_a_str = csv_path_a.to_str().unwrap();

let csv_path_b = current_dir.join("test_file_samples/joining_test_files/join_file_b.csv");
let csv_path_b_str = csv_path_b.to_str().unwrap();

let rt = Runtime::new().unwrap();
rt.block_on(async {
    let args = vec![
        ("--file_a_path", csv_path_a_str),
        ("--file_b_path", csv_path_b_str),
        ("--join_type", "LEFT_JOIN"),
        ("--file_a_ref_column", "id"),
        ("--file_b_ref_column", "id")
    ];

    let mut builder = CsvBuilder::from_bare_metal_python_executable(
        &executable_path_str,
        args,
        ).await;

});

示例5：从xls/xlsx/h5文件加载

use rgwml::csv_utils::CsvBuilder;

// Load from a sheet in an .xls file
let builder_1 = CsvBuilder::from_xls("/path/to/existing/file.xls", "Sheet1", "SHEET_NAME"); // Loads from the sheet named "Sheet1" of the .xls file.
builder_1.print_table();
let builder_2 = CsvBuilder::from_xls("/path/to/existing/file.xls", "1", "SHEET_ID"); // Loads from the seond sheet of the .xls file i.e. having an id of 1 (since the first sheet has an id of 0).
builder_2.print_table();

// Load from a sheet in an .xlsx file
let builder_1 = CsvBuilder::from_xlsx("/path/to/existing/file.xlsx", "Sheet1", "SHEET_NAME"); // Loads from the sheet named "Sheet1" of the .xlsx file.
builder_1.print_table();
let builder_2 = CsvBuilder::from_xlsx("/path/to/existing/file.xlsx", "1", "SHEET_ID"); // Loads from the seond sheet of the .xlsx file i.e. having an id of 1 (since the first sheet has an id of 0).       
builder_2.print_table();

// Load from a dataset in an .h5 file
let builder_1 = CsvBuilder::from_h5("/path/to/existing/file.h5", "Dataset1", "DATASET_NAME").await; // Loads from the dataset named "Dataset1" of the .h5 file.
builder_1.print_table();
let builder_2 = CsvBuilder::from_h5("/path/to/existing/file.h5", "1", "DATASET_ID").await; // Loads from the seond sheet of the .h5 file i.e. having an id of 1 (since the first sheet has an id of 0).
builder_2.print_table();

示例6：从原始数据加载

use rgwml::csv_utils::CsvBuilder;

let headers = vec!["Header1".to_string(), "Header2".to_string(), "Header3".to_string()];
let data = vec![
    vec!["Row1-1".to_string(), "Row1-2".to_string(), "Row1-3".to_string()],
    vec!["Row2-1".to_string(), "Row2-2".to_string(), "Row2-3".to_string()],
];

let builder = CsvBuilder::from_raw_data(headers, data);
builder.print_table();

示例7：从MSSQL/MYSQL服务器查询加载

use rgwml::csv_utils::CsvBuilder;

let result = CsvBuilder::from_mssql_query(            // Also available: .from_mysql_query
    "username", 
    "password", 
    "server", 
    "database", 
    "SELECT * from your_table")
    .await
    .expect("Failed to create CsvBuilder query");

result.print_table();

// To load the column description of a particular table into a CsvBuilder object
let _ = CsvBuilder::get_mssql_table_description(
    "username", 
    "password", 
    "server", 
    "in_focus_database", 
    "table_name")
    .await
    .expect("Failed to create CsvBuilder query");

result.print_table();

示例8：从MSSQL/MYSQL服务器查询加载，以分块形式接收数据，合并为联合查询

use rgwml::csv_utils::CsvBuilder;

let result = CsvBuilder::from_chunked_mssql_query_union(    // Also available: .from_chunked_mysql_query_union
    "username",
    "password",
    "server",
    "database",
    "SELECT * from your_table"
    "10000" // Get data in chunks of 10000 rows at a time
    )
    .await
    .expect("Failed to create CsvBuilder query");

result.print_table();

示例9：从MSSQL/MYSQL服务器查询加载，以分块形式接收数据，合并为包联合查询

use rgwml::csv_utils::CsvBuilder;

let result = CsvBuilder::from_chunked_mssql_query_bag_union(    // Also available: .from_chunked_mysql_query_bag_union
    "username",
    "password",
    "server",
    "database", 
    "SELECT * from your_table"
    "10000" // Get data in chunks of 10000 rows at a time
    )
    .await
    .expect("Failed to create CsvBuilder query");

result.print_table();

示例10：从Clickhouse服务器查询加载

use rgwml::csv_utils::CsvBuilder;

let result = CsvBuilder::from_clickhouse_query(  
    "username",
    "password",
    "server",
    "SELECT * from your_table")
    .await
    .expect("Failed to create CsvBuilder query");

result.print_table();


// To load the column description of a particular table into a CsvBuilder object
let result = CsvBuilder::get_clickhouse_table_description(
    "username",
    "password",
    "server",
    "table_name")
    .await
    .expect("Failed to create CsvBuilder query");

result.print_table();

示例11：从Clickhouse服务器查询加载，以分块形式接收数据，合并为联合查询

use rgwml::csv_utils::CsvBuilder;

let result = CsvBuilder::from_chunked_clickhouse_query_union(
    "username",
    "password",
    "server",
    "SELECT * from your_table"
    "10000" // Get data in chunks of 10000 rows at a time
    )
    .await
    .expect("Failed to create CsvBuilder query");

result.print_table();

示例12：从Clickhouse服务器查询加载，以分块形式接收数据，合并为包联合查询

use rgwml::csv_utils::CsvBuilder;

let result = CsvBuilder::from_chunked_clickhouse_query_bag_union(
    "username",
    "password",
    "server",
    "SELECT * from your_table"
    "10000" // Get data in chunks of 10000 rows at a time
    )
    .await
    .expect("Failed to create CsvBuilder query");

result.print_table();

示例13：从Google Big Query服务器加载

use rgwml::csv_utils::CsvBuilder;

let result = CsvBuilder::from_google_big_query_query(  
    "path/to/your/json/credentials",
    "SELECT * from your_table")
    .await
    .expect("Failed to create CsvBuilder query");

result.print_table();

// To load the column description of a particular table into a CsvBuilder object
let result = CsvBuilder::get_google_big_query_table_description(
    "path/to/your/json/credentials",
    "your_project_id",
    "your_dataset_name",
    "your_table_name")
    .await
    .expect("Failed to create CsvBuilder query");

result.print_table();

示例14：从Google Big Query服务器查询加载，以分块形式接收数据，合并为联合查询

use rgwml::csv_utils::CsvBuilder;

let result = CsvBuilder::from_chunked_google_big_query_query_union(
    "path/to/your/json/credentials",
    "SELECT * from your_table"
    "10000" // Get data in chunks of 10000 rows at a time
    )
    .await
    .expect("Failed to create CsvBuilder query");

result.print_table();

示例15：从Google Big Query服务器查询加载，以分块形式接收数据，合并为包联合查询

use rgwml::csv_utils::CsvBuilder;

let result = CsvBuilder::from_chunked_google_big_query_query_bag_union(
    "path/to/your/credentials",
    "SELECT * from your_table"
    "10000" // Get data in chunks of 10000 rows at a time
    )
    .await
    .expect("Failed to create CsvBuilder query");

result.print_table();

示例16：从现有实例加载新的实例

use rgwml::csv_utils::CsvBuilder;

let builder_instance_1 = CsvBuilder::from_xls("/path/to/existing/file.xls", 1);
let builder_instance_2 = CsvBuilder::from_copy(builder_instance_1);

用于分析或保存的CsvBuilder对象的操作

use rgwml::csv_utils::{Exp, ExpVal, CsvBuilder, CsvConverter};

let _ = CsvBuilder::from_csv("/path/to/your/file.csv")
    .rename_columns(vec![("OLD_COLUMN", "NEW_COLUMN")])
    .drop_columns(vec!["UNUSED_COLUMN"])
    .cascade_sort(vec![("COLUMN".to_string(), "ASC".to_string())])
    .reverse_rows() // Reverses the order of the rows
    .reverse_columns() // Reverses the order of columns
    .where_(
        vec![
            ("Exp1", Exp {
                column: "customer_type".to_string(),
                operator: "==".to_string(),
                compare_with: ExpVal::STR("REGULAR".to_string()),
                compare_as: "TEXT".to_string() // Also: "NUMBERS", "TIMESTAMPS"
            }),
            ("Exp2", Exp {
                column: "invoice_data".to_string(),
                operator: ">".to_string(),
                compare_with: ExpVal::STR("2023-12-31 23:59:59".to_string()),
                compare_as: "TEXT".to_string()
            }),
            ("Exp3", Exp {
                column: "invoice_amount".to_string(),
                operator: "<".to_string(),
                compare_with: ExpVal::STR("1000".to_string()),
                compare_as: "NUMBERS".to_string()
            }),
            ("Exp4", Exp {
                column: "address".to_string(),
                operator: "FUZZ_MIN_SCORE_60".to_string(),
                compare_with: ExpVal::VEC(vec!["public school".to_string()]),
                compare_as: "TEXT".to_string()
            })
        ],
        "Exp1 && (Exp2 || Exp3) && Exp4",
    )
    .print_row_count()
    .save_as("/path/to/modified/file.csv");

链式选项

use rgwml::csv_utils::{CalibConfig, CsvBuilder, CsvConverter, Exp, ExpVal, Train};
use rgwml::xgb_utils::XgbConfig;
use rgwml::dask_utils::{DaskGrouperConfig, DaskPivoterConfig, DaskCleanerConfig, DaskJoinerConfig, DaskIntersectorConfig, DaskDifferentiatorConfig};

CsvBuilder::from_csv("/path/to/your/file1.csv")
// A. Calibrating an irrugularly formatted file
.calibrate(
    CalibConfig {
        header_is_at_row: "21".to_string(),
        rows_range_from: ("23".to_string(), "*".to_string())
    }) // sets the row 21 content as the header, and row 23 to last row content as the data

// B. Setting and adding headers
.set_header(vec!["Header1", "Header2", "Header3"])
.add_column_header("NewColumn1")
.add_column_headers(vec!["NewColumn2", "NewColumn3"])

// C. Set an Index
.resequence_id_column("account_id") // Sets the values of the specified column sequentially from 1 onwards, ensuring each entry is uniquely numbered in ascending order until the last row.

// D. Assuming a single row csv, set the value of a column
.set("column_name", "value");

// E. Ordering columns
.order_columns(vec!["Column1", "...", "Column5", "Column2"])
.order_columns(vec!["...", "Column5", "Column2"])
.order_columns(vec!["Column1", "Column5", "..."])

// F. Overriding data from another builder object
.override_with(other_csv_builder_object);

// G. Modifying columns
.drop_columns(vec!["Column1", "Column3"])
.retain_columns(vec!["Column1", "Column3"])
.rename_columns(vec![("Column1", "NewColumn1"), ("Column3", "NewColumn3")])

// H. Adding and modifying rows
.add_row(vec!["Row1-1", "Row1-2", "Row1-3"])
.add_rows(vec![vec!["Row1-1", "Row1-2", "Row1-3"], vec!["Row2-1", "Row2-2", "Row2-3"]])
.update_row_by_row_number(2, vec!["Bob", "36", "San Francisco"])
.update_row_by_id(2, vec!["Bob", "36", "San Francisco"]) // Updates a row by id in the CSV, assuming the first column is 'id'
.delete_row_by_row_number(2)
.delete_row_by_id(2) // Deletes a row by id in the CSV, assuming the first column is 'id'
.remove_duplicates()

// I. Cleaning/ Replacing Cell values
.trim_all() // Trims white spaces at the beginning and end of all cells in all columns.
.replace_header_whitespaces_with_underscores()
.replace_all(vec!["Column1".to_string(), "Column2".to_string()], vec![("null".to_string(), "".to_string()), ("NA".to_string(), "-".to_string())]) // In specified columns
.replace_all(vec!["*".to_string()], vec![("null".to_string(), "".to_string()), ("NA".to_string(), "-".to_string())]) // In all columns
replace_all_empty_string_cells_with(vec!["Column1", "Column2"], "0")
.clean_or_test_clean_by_eliminating_rows_subject_to_column_parse_rules(
    DaskCleanerConfig {
        rules: "Column1:IS_VALID_TEN_DIGIT_INDIAN_MOBILE_NUMBER;Column2:IS_NUMERICAL_VALUE".to_string(),  // Avalable Rules: IS_NUMERICAL_VALUE, IS_POSITIVE_NUMERICAL_VALUE, IS_LENGTH:n (for instance: IS_LENGTH:9), IS_MIN_LENGTH:n, IS_MAX_LENGTH:n, IS_VALID_TEN_DIGIT_INDIAN_MOBILE_NUMBER, IS_NOT_AN_EMPTY_STRING, IS_DATETIME_PARSEABLE
        action: "ANALYZE_AND_CLEAN".to_string(), // Avalailable Actions: CLEAN, ANALYZE, ANALYZE_AND_CLEAN
        show_unclean_values_in_report: "TRUE".to_string(), // Options: TRUE, FALSE
    })

// J. Limiting and sorting
.limit(10)
.limit_distributed_raw(10)  //  limit rows distributed as evenly as possible across the dataset
.limit_distributed_category(10, "Colum7")  //  limit rows distributed as evenly as possible across the dataset, to maximize variance in values of the indicated column
.limit_rand(10)         // limit rows randomly
.limit_where(
    10,
    vec![
        ("Exp1", Exp {
            column: "Withdrawal Amt.".to_string(),
            operator: "<".to_string(),
            compare_with: ExpVal::STR("1000".to_string()),
            compare_as: "NUMBERS".to_string() // Also: "TEXT", "TIMESTAMPS"
        }),
        ("Exp2", Exp {
            column: "Withdrawal Type".to_string(),
            operator: "==".to_string(),
            compare_with: ExpVal::STR("Urgent".to_string()),
            compare_as: "TEXT".to_string()
        }),
    ],
    "Exp1 && Exp2",
    "TAKE:FIRST" // Also: TAKE:LAST, TAKE:RANDOM
    )
.cascade_sort(vec![("Column1".to_string(), "DESC".to_string()), ("Column3".to_string(), "ASC".to_string())])

// K. Search operations
.print_contains_search_results("needle") // Prints rows where any cell contains the needle
.print_not_contains_search_results("needle") // Prints rows where no cell contains the needle
.print_starts_with_search_results("needle") // Prints rows where any cell starts with the needle
.print_not_starts_with_search_results("needle") // Prints rows where no cell starts with the needle

// L. Search operations
.print_contains_search_results("needle", vec!["*"]) // Prints rows where any cell in all columns contains the needle
.print_contains_search_results("needle", vec!["column1", "column2"]) // Same as above, but only specific columns targetted
.print_not_contains_search_results("needle", vec!["*"]) // Prints rows where no cell in all columns contains the needle
.print_not_contains_search_results("needle", vec!["column1", "column2"]) // Same as above, but only specific columns targetted
.print_starts_with_search_results("needle", vec!["*"]) // Prints rows where any cell in all columns starts with the needle
.print_starts_with_search_results("needle", vec!["column1", "column2"]) // Same as above, but only specific columns targetted
.print_not_starts_with_search_results("needle", vec!["*"]) // Prints rows where no cell in all columns starts with the needle
.print_not_starts_with_search_results("needle", vec!["column1", "column2"]) // Same as above, but only specific columns targetted
.print_raw_levenshtein_search_results("needle", 10, ["column1", "column2"]) // Prints rows where cells in column1, column2 have a levenshtein distance of less than 10 vis-a-vis the needle
.print_vectorized_levenshtein_search_results(["awesome", "good job"], max_lev_distance, ["column1", "column2"]) // Dynamically compares each needle against successive combinations of words within the cell values from the indicated columns, considering the minimum word count of the needle. It computes the Levenshtein distance for each needle qua the cell value, and for each such comparison the cell value is considered based on every combination of constituent words accruing from the minimum distance found within a specified maximum distance (max_lev_distance). This approach allows matching based on the proximity of words, providing a more contextually relevant search. For instance, if the cell contains "django is a good boy", it generates and compares distances for combinations like "django is", "is a", "a good", "good boy", up to the full cell content, ultimately considering the closest match. The minimum levenshtein distance acorss all needles for that cell value is then considered as the basis for filtering.

// M. Applying conditional operations
.where_(
    vec![
        ("Exp1", Exp {
            column: "customer_type".to_string(),
            operator: "==".to_string(),
            compare_with: ExpVal::STR("REGULAR".to_string()),
            compare_as: "TEXT".to_string() // Also: "NUMBERS", "TIMESTAMPS"
        }),
        ("Exp2", Exp {
            column: "invoice_data".to_string(),
            operator: ">".to_string(),
            compare_with: ExpVal::STR("2023-12-31 23:59:59".to_string()),
            compare_as: "TEXT".to_string()
        }),
        ("Exp3", Exp {
            column: "invoice_amount".to_string(),
            operator: "<".to_string(),
            compare_with: ExpVal::STR("1000".to_string()),
            compare_as: "NUMBERS".to_string()
        }),
        ("Exp4", Exp {
            column: "address".to_string(),
            operator: "FUZZ_MIN_SCORE_60".to_string(),
            compare_with: ExpVal::VEC(vec!["public school".to_string()]),
            compare_as: "TEXT".to_string()
        }),
        ("Exp5", Exp {
            column: "status".to_string(),
            operator: "CONTAINS".to_string(), // Also: "DOES_NOT_CONTAIN"
            compare_with: ExpVal::STR("REJECTED".to_string()),
            compare_as: "TEXT".to_string()
        }),
        ("Exp6", Exp {
            column: "status".to_string(),
            operator: "STARTS_WITH".to_string(), // Also: "DOES_NOT_START_WITH"
            compare_with: ExpVal::STR("VERIFIED".to_string()),
            compare_as: "TEXT".to_string()
        }),
    ],
    "Exp1 && (Exp2 || Exp3 || Exp4) && Exp5 && Exp6")
.where_set(
    vec![
        // Same as .where() 
    ],
    "Exp1 && (Exp2 || Exp3 || Exp4) && Exp5 && Exp6",
    "Column10",
    "IS OKAY")

// N. Analytical Prints for data inspection

.print_columns()
.print_row_count()
.print_first_row("75").await // Shows the first row via a dask vectorization for object sizes of more than 75 mb, and via a String approach for smaller tables
.print_first_row_small_file()
.print_first_row_big_file().await

.print_last_row("75").await // Shows the last row via a dask vectorization for object sizes of more than 75 mb, and via a String approach for smaller tables
.print_last_row_small_file()
.print_last_row_big_file().await

.print_first_n_rows("2", "75").await // Shows the first 2 rows via a dask vectorization for object sizes of more than 75 mb, and via a String approach for smaller tables
.print_first_n_rows_small_file("2")
.print_first_n_rows_big_file("2").await 

.print_last_n_rows("2", "75").await // Shows the last 2 rows via a dask vectorization for object sizes of more than 75 mb, and via a String approach for smaller tables
.print_last_n_rows_small_file("2")
.print_last_n_rows_big_file("2").await

.print_rows_range("2","5","75").await // Shows results per a spreadsheet row range via a dask vectorization for object sizes of more than 75 mb, and via a String approach for smaller tables
.print_rows_range_small_file("2","5")
.print_rows_range_big_file("2","5").await
.print_rows() // Shows results as per a spreadsheet row range
.print_rows_where(
    vec![
        // Same as .where()
    ],
    "Exp1 && (Exp2 || Exp3 || Exp4) && Exp5 && Exp6")
.print_table("75").await // Prints a truncated table to the terminal, via a dask vectorization for object sizes of more than 75 mb, and via a String approach for smaller tables
.print_table_big_file().await // Uses a dask vectorization output efficiently
.print_table_small_file().await // Uses a String approach to output efficiently
.print_table_all_rows() // Prints a truncated table to the terminal, with all rows
.print_cells(vec!["Column1", "Column2"])
.print_unique("column_name")
.print_unique_count("column_name")
.print_column_numerical_analysis(vec!["Column1", "Column2"]) // Prints the min, max, range, mean, median, mode, variance, standard deviation, sum of squared deviations, and list non-numerical values, if any, for each of the indicated columns
.print_freq(vec!["Column1", "Column2"])
.print_freq_cascading(vec!["Column1", "Column2"]) // Prints cascading frequency tables for selected columns of a dataset.
.print_freq_mapped(vec![
        ("Column1", vec![
            ("Delhi", vec!["New Delhi", "Delhi"]),
            ("UP", vec!["Ghaziabad", "Noida"])
        ]),
        ("Column2", vec![("NO_GROUPINGS", vec![])])
    ])
.print_unique_values_stats(vec!["Column1", "Column2"]) // Prints the number of unique values in a column, along with the mean and median of their frequencies
.print_count_where(
    vec![
        // Same as .where()
    ],
    "Exp1 && (Exp2 || Exp3 || Exp4) && Exp5 && Exp6")

// O. Transforming Data
.transpose_transform() // Transposes the headers with the first row
.split_as("ColumnNameToGroupBy", "/output/folder/for/grouped/csv/files/") // Groups data by a specified column and saves each group into a separate CSV file in a given folder
.grouped_index_transform(
    DaskGrouperConfig {
        group_by_column_name: "Column7".to_string(),
        count_unique_agg_columns: "".to_string(),
        numerical_max_agg_columns: "Column8, Column9".to_string(), 
        numerical_min_agg_columns: "".to_string(),
        numerical_sum_agg_columns: "".to_string(),
        numerical_mean_agg_columns: "".to_string(),
        numerical_median_agg_columns: "".to_string(),
        numerical_std_deviation_agg_columns: "".to_string(),
        mode_agg_columns: "".to_string(),
        datetime_max_agg_columns: "".to_string(),
        datetime_min_agg_columns: "".to_string(),
        datetime_semi_colon_separated_agg_columns: "".to_string(),
        bool_percent_agg_columns: "".to_string(),
    })
.pivot(
    DaskPivoterConfig {
        group_by_column_name: "Column7".to_string(),
        values_to_aggregate_column_name: "Column9".to_string(),
        operation: "NUMERICAL_MEAN".to_string(), // Options: COUNT, COUNT_UNIQUE, NUMERICAL_MAX, NUMERICAL_MIN, NUMERICAL_SUM, NUMERICAL_MEAN, NUMERICAL_MEDIAN, NUMERICAL_STANDARD_DEVIATION, BOOL_PERCENT
        segregate_by_column_names: "Column3, Column5".to_string()
    })

// P. Basic Set Theory Operations 

// P.1. WITH CSV FILES (DYNAMIC THRESHOLD)
.union_with_csv_file("/path/to/set_b/file.csv", 
    "UNION", // Options: UNION, BAG_UNION, LEFT_JOIN, RIGHT_JOIN, OUTER_FULL_JOIN
    "id",    // Table A reference column
    "id",    // Table B reference column
    "75"     // If the file size is above 75 MB, Dask vectorization will be favored over a String based approach
    ).await
.intersection_with_csv_file("/path/to/set_b/file.csv",    // Analogus to 'INNER_JOIN' 
    "id",    // Table A reference column
    "id",    // Table B reference column
    "75"     // If the file size is above 75 MB, Dask vectorization will be favored over a String based approach
    ).await
.difference_with_csv_file("/path/to/set_b/file.csv",     
    "NORMAL", // Options: NORMAL, SYMMETRIC
    "id",    // Table A reference column
    "id",    // Table B reference column
    "75"     // If the file size is above 75 MB, Dask vectorization will be favored over

基于字符串的方法).await

// P.2. WITH CSV BUILDER (DYNAMIC THRESHOLD)
.union_with_csv_builder(set_b_csv_builder,     
    "UNION", // Options: UNION, BAG_UNION, LEFT_JOIN, RIGHT_JOIN, OUTER_FULL_JOIN
    "id",    // Table A reference column
    "id",    // Table B reference column
    "75"     // If the builder size is above 75 MB, Dask vectorization will be favored over a String based approach
    ).await
.intersection_with_csv_builder(set_b_csv_builder,    // Analogus to 'INNER_JOIN'
    "id",    // Table A reference column
    "id",    // Table B reference column
    "75"     // If the builder size is above 75 MB, Dask vectorization will be favored over a String based approach
    ).await
.difference_with_csv_builder(set_b_csv_builder,
    "NORMAL", // Options: NORMAL, SYMMETRIC
    "id",    // Table A reference column
    "id",    // Table B reference column
    "75"     // If the builder size is above 75 MB, Dask vectorization will be favored over a String based approach
    ).await

// P.3. WITH CSV FILES (SMALL)
.union_with_csv_file_small("/path/to/set_b/file.csv",
    "UNION", // Options: UNION, BAG_UNION, LEFT_JOIN, RIGHT_JOIN, OUTER_FULL_JOIN
    "id",    // Table A reference column
    "id",    // Table B reference column
    ).await
.intersection_with_csv_file_small("/path/to/set_b/file.csv",    // Analogus to 'INNER_JOIN'
    "id",    // Table A reference column
    "id",    // Table B reference column
    ).await
.difference_with_csv_file_small("/path/to/set_b/file.csv",
    "NORMAL", // Options: NORMAL, SYMMETRIC
    "id",    // Table A reference column
    "id",    // Table B reference column
    ).await

// P.4. WITH CSV BUILDER (SMALL)
.union_with_csv_builder_small(set_b_csv_builder,
    "UNION", // Options: UNION, BAG_UNION, LEFT_JOIN, RIGHT_JOIN, OUTER_FULL_JOIN
    "id",    // Table A reference column
    "id",    // Table B reference column
    ).await
.intersection_with_csv_builder_small(set_b_csv_builder,    // Analogus to 'INNER_JOIN'
    "id",    // Table A reference column
    "id",    // Table B reference column
    ).await
.difference_with_csv_builder_small(set_b_csv_builder,
    "NORMAL", // Options: NORMAL, SYMMETRIC
    "id",    // Table A reference column
    "id",    // Table B reference column
    ).await

// P.5. WITH CSV FILES (BIG)
.union_with_csv_file_big("/path/to/set_b/file.csv", 
    DaskJoinerConfig {
        join_type: "LEFT_JOIN".to_string(), // Options: UNION, BAG_UNION, LEFT_JOIN, RIGHT_JOIN, OUTER_FULL_JOIN
        table_a_ref_column: "id".to_string(), // Leave as empty "" for UNION/ BAG_UNION
        table_b_ref_column: "id".to_string(), // Leave as empty "" for UNION/ BAG_UNION
    }).await
.intersection_with_csv_file_big("/path/to/set_b/file.csv", 
    DaskIntersectorConfig {
        table_a_ref_column: "id".to_string(), 
        table_b_ref_column: "id".to_string(),
    }).await
.difference_with_csv_file_big("/path/to/set_b/file.csv",
    DaskDifferentiatorConfig {
        difference_type: "NORMAL".to_string(), // Options: NORMAL, SYMMETRIC
        table_a_ref_column: "id".to_string(), 
        table_b_ref_column: "id".to_string(), 
    }).await

// P.6. WITH CSV BUILDER (BIG)
.union_with_csv_builder_big(set_b_csv_builder, 
    DaskJoinerConfig {
        join_type: "LEFT_JOIN".to_string(), // Options: UNION, BAG_UNION, LEFT_JOIN, RIGHT_JOIN, OUTER_FULL_JOIN
        table_a_ref_column: "id".to_string(), // Leave as empty "" for UNION/ BAG_UNION
        table_b_ref_column: "id".to_string(), // Leave as empty "" for UNION/ BAG_UNION
    }).await
.intersection_with_csv_file_big(set_b_csv_builder,
    DaskIntersectorConfig {
        table_a_ref_column: "id".to_string(), 
        table_b_ref_column: "id".to_string(), 
    }).await
.difference_with_csv_file_big(set_b_csv_builder,
    DaskDifferentiatorConfig {
        difference_type: "NORMAL".to_string(), // Options: NORMAL, SYMMETRIC
        table_a_ref_column: "id".to_string(),
        table_b_ref_column: "id".to_string(),
    }).await

// Q. Append Analytical Columns
.append_static_value_column("static_value_across_all_rows", "new_column_name")
.append_derived_boolean_column(
    "is_qualified_for_discount",
    vec![
        // Same as .where() 
    ],
    "Exp1 && (Exp2 || Exp3 || Exp4) && Exp5 && Exp6")
.append_inclusive_exclusive_numerical_interval_category_column(
        "column_name",           // Values in this column should be numerically parseable
        "interval_points",       // For instance 0,5,10,15 .... and so on, will create the intervals 00 to 05, 05 to 10, 10 to 15 ... and so on, with the lower limit being inclusive  , and the upper limit being exclusive
        "new_column_name"        // The name of the new column 
    )
.append_inclusive_exclusive_numerical_interval_category_column(
        "column_name",           // Values in this column should be numerically parseable
        "interval_points",       // For instance 0,5,10,15 .... and so on, will create the intervals 00 to 05, 05 to 10, 10 to 15 ... and so on, with the lower limit being inclusive , and the upper limit being exclusive
        "new_column_name"        // The name of the new column
    )
.append_derived_category_column(
    "EXPENSE_RANGE",
    vec![
        (
            "< 1000",
            vec![
                ("Exp1", Exp {
                    column: "Withdrawal Amt.".to_string(),
                    operator: "<".to_string(),
                    compare_with: ExpVal::STR("1000".to_string()),
                    compare_as: "NUMBERS".to_string() // Also: "TEXT", "TIMESTAMPS"
                }),
            ],
            "Exp1"
        ),
        (
            "1000-5000",
            vec![
                ("Exp1", Exp {
                    column: "Withdrawal Amt.".to_string(),
                    operator: ">=".to_string(),
                    compare_with: ExpVal::STR("1000".to_string()),
                    compare_as: "NUMBERS".to_string()
                }),
                ("Exp2", Exp {
                    column: "Withdrawal Amt.".to_string(),
                    operator: "<".to_string(),
                    compare_with: ExpVal::STR("5000".to_string()),
                    compare_as: "NUMBERS".to_string()
                }),
            ],
            "Exp1 && Exp2"
        )
    )
.append_derived_concatenation_column("NewColumnName", vec!["Column1", " ", "Column2", "@"]) // Items in the vector that are not column names will be concatenated as strings
.append_derived_openai_analysis_columns(
    vec!["column7", "column9"],     // Names of the columns to be analyzed 
    std::collections::HashMap::from([
        ("noun".to_string(), "extract the noun from the sentence".to_string()),
        ("verb".to_string(), "extract the verb from the sentence".to_string()),
    ]),
    "YOUR_OPEN_AI_API_KEY",
    "gpt-3.5-turbo-0125"            // Any OpenAI model with the JSON mode feature
    ).await
.append_derived_smartcore_linear_regression_column(
    "predictions",                  // name of new column to store predictions
    vec![                           // predictor combinations/ feature sets - length should be 2x the number of predictors/features
        vec!["90", "good"],         // predictor/ feature values can also be text strings. The model uses a Levenshtein distance based approach to tokenize strings.
        vec!["70", "bad"], 
        vec!["60", "great"], 
        vec!["40", "awful"]
    ], 
    vec![72.0, 65.0, 63.0, 56.0],   // labels mapped to the above predictors
    vec![0.0, 100.0],               // normalization range of minimum and maximum prediction value
    vec!["Column1", "Column7"])     // names of columns whose values are to be used to make predictions as the 'test' data set 
.append_openai_batch_analysis_columns(
    "YOUR_OPEN_AI_API_KEY",
    "output_file_id"
    )
.append_fuzzai_analysis_columns(
    "Column1", // Name of column to be analyzed
    "sales_analysis", // Identifier for newly created columns
    vec![
        Train {
            input: "I want my money back".to_string(),
            output: "refund".to_string()
        },
        Train {
            input: "I want a refund immediately".to_string(),
            output: "refund".to_string()
        },
    ],
    "WORD_SPLIT:2", // The minimum length of word combinations that training data is to be broken into
    "WORD_LENGTH_SENSITIVITY:0.8", // Multiplies differences in word length between training data input and the value being analyzed by 0.8
    "GET_BEST:2" // Get the top 2 results, max value is 3
    )
.append_fuzzai_analysis_columns_with_values_where(
    "Column1", // Name of column to be analyzed
    "sales_analysis", // Identifier for newly created column
    vec![
        Train {
            input: "I want my money back".to_string(),
            output: "refund".to_string()
        },
        Train {
            input: "I want a refund immediately".to_string(),
            output: "refund".to_string()
        },
    ],
    "WORD_SPLIT:2", // The minimum length of word combinations that training data is to be broken into
    "WORD_LENGTH_SENSITIVITY:0.8", // Multiplies differences in word length between training data input and the value being analyzed by 0.8
    "GET_BEST:2", // Get the top 2 results, max value is 3
    vec![
        ("Exp1", Exp {
            column: "Deposit Amt.".to_string(),
            operator: ">".to_string(),
            compare_with: ExpVal::STR("500".to_string()),
            compare_as: "NUMBERS".to_string() // Also: "TEXT", "TIMESTAMPS"
        }),
    ],
    "Exp1", // Filters rows where fuzzai analysis would be applied
    )

// R. Append Analytical Date/Timestamp Columns
.append_semi_colon_separated_timestamp_count_after_date_column(
    "semi_colon_separated_timestamps_column_name", 
    "date_column_name", 
    "new_column_name"
    )   
.append_semi_colon_separated_timestamp_count_before_date_column(
    "semi_colon_separated_timestamps_column_name", 
    "date_column_name", 
    "new_column_name"
    )   
.append_added_days_column_relative_to_adjacent_column(
    "days_column_name",         // Should be float parseable
    "timestamp_column_name",    // Should be timestamp/date parseable
    "new_column_name"
    )
.append_subtracted_days_column_relative_to_adjacent_column(
    "days_column_name",         // Should be float parseable
    "timestamp_column_name",    // Should be timestamp/date parseable
    "new_column_name"
    )
.append_added_days_column(
    "date_column_name",             // Should be timestamp/date parseable
    "number_of_days_to_add",        // Should be float parseable
    "new_column_name"
    )
.append_subtracted_days_column(
    "date_column_name",              // Should be timestamp/date parseable
    "number_of_days_to_subtract",    // Should be float parseable
    "new_column_name"
    )
.append_day_difference_column(
    "date_column_1_name",
    "date_column_2_name",
    "new_column_name"
    )
.split_date_as_appended_category_columns("Column10", "%d/%m/%y") // Appends additional columns splitting a date/timestamp into categorization columns by year, month and week


// S. Plot charts
.print_dot_chart("Column3", "Column5") // X axis column followed by the Y axis column
.print_cumulative_dot_chart("Column3", "Column5") // X axis column followed by the Y axis column
.print_smooth_line_chart("Column3", "Column5") // X axis column followed by the Y axis column
.print_cumulative_smooth_line_chart("Column3", "Column5") // X axis column followed by the Y axis column

// T. Save
.save_as("/path/to/your/file2.csv")

// U. Die
.die() // Gracefully terminates execution of a CsvBuilder chain

提取数据

这些方法返回特定数据，而不是可变的CsvBuilder对象，因此不能连续链式调用。

let builder = CsvBuilder::from_csv("/path/to/your/file1.csv");

builder
.get_unique("column_name"); // Returns a Vec<String>
.get("column_name"); // Returns cell content as a String, if the csv has been filtered to single row. See the chainable ".set()" method above for set a value in such a circumstance
.get_freq(vec!["Column1", Column2]) // Returns a HashMap where keys are column names and values are vectors of sorted (value, frequency) pairs.
.get_freq_mapped(vec![
        ("Column1", vec![
            ("Delhi", vec!["New Delhi", "Delhi"]),
            ("UP", vec!["Ghaziabad", "Noida"])
        ]),
        ("Column2", vec![("NO_GROUPINGS", vec![])])
    ])
.has_data() // Returns `true` if either headers or data rows are present, `false` otherwise.
.has_headers() // Returns `true` if headers are present, `false` otherwise.
.get_headers().unwrap() // Returns an Option<&[String]> containing a reference to the headers if present, `None` otherwise.
.get_data().unwrap() // Returns an Option<&Vec<Vec<String>>> containing a reference to the data contained in the builder.

.get_numeric_min("Column1").unwrap() // Returns a String value of the minimum numeric value - assuming all values of the column can be consistently parsed as such
.get_numeric_max("Column1").unwrap() // Returns a String value of the maximum numeric value - assuming all values of the column can be consistently parsed as such
.get_datetime_min("Column1").unwrap() // Returns a String value of the minimum numeric value - assuming all values of the column can be consistently parsed as such
.get_datetime_max("Column1").unwrap() // Returns a String value of the maximum numeric value - assuming all values of the column can be consistently parsed as such
.get_range("Column1").unwrap() // Returns an `Option<f64>` the range (difference between the maximum and minimum) in a numerically parseable column. 
.get_sum("Column1").unwrap() // Returns an `Option<f64>` the sum of all values in a numerically parseable column.
.get_mean("Column1").unwrap() // Returns an `Option<f64>` - the mean of all values in a numerically parseable column.
.get_median("Column1").unwrap() // Returns an `Option<f64>` - the median of all values in a numerically parseable column.
.get_mode("Column1").unwrap() // Returns an `Option<f64>` - the mode of all values in a numerically parseable column.
.get_variance("Column1").unwrap() // Returns an `Option<f64>` - the variance of all values in a numerically parseable column.
.get_standard_deviation("Column1").unwrap() // Returns an `Option<f64>` - the standard deviation of all values in a numerically parseable column.
.get_sum_of_squared_deviations("Column1").unwrap() // Returns an `Option<f64>` - the getsum of squared deviations of all values in a numerically parseable column.
.get_get_non_numeric_values("Column1").unwrap() // Returns an `Option<Vec<String>>` - the non numeric values in a column. 

// Send data to OpenAI for batch analysis, returning a batch_id as `Result<String, Box<dyn std::error::Error>>`
.send_columns_for_openai_batch_analysis(
    vec!["column7", "column9"],     // Names of the columns to be analyzed
    std::collections::HashMap::from([
        ("noun".to_string(), "extract the noun from the sentence".to_string()),
        ("verb".to_string(), "extract the verb from the sentence".to_string()),
    ]),
    "YOUR_OPEN_AI_API_KEY",
    "gpt-3.5-turbo-0125"            // Any OpenAI model with the JSON mode feature
    "night_job"                     // Name of the batch

)
.get_all_csv_files("path/to/your/directory") // Returns a `Result<CsvBuilder, Box<dyn Error>>` of all CSV files in the directory specifying their file name, last modified time, and file size in mb

CsvBuilder-XGBoost操作

// A. Creating a Model
use rgwml::csv_utils::CsvBuilder;
use rgwml::xgb_utils::XgbConfig;

let mut builder = CsvBuilder::from_csv("/path/to/your/training_data.csv");

let (builder, report) =  builder.create_xgb_model(
    "ticket_count, tickets_after_last_payment",     // Param column names
    "churn_day",                                    // Target column name
    "PREDICTION",                                   // Prediction column name
    "/home/rgw/Desktop/csv_db/xgb_models",          // Dir to save model
    "churn_v8",                                     // Model name without extension
    XgbConfig {
        objective: "reg:squarederror".to_string(), // Leave as empty string for binary classification
        xgb_max_depth: "".to_string(),
        xgb_learning_rate: "".to_string(),
        xgb_n_estimators: "".to_string(),
        xgb_gamma: "".to_string(),
        xgb_min_child_weight: "".to_string(),
        xgb_subsample: "".to_string(),
        xgb_colsample_bytree: "".to_string(),
        xgb_reg_lambda: "".to_string(),
        xgb_reg_alpha: "".to_string(),
        xgb_scale_pos_weight: "".to_string(),
        xgb_max_delta_step: "".to_string(),
        xgb_booster: "".to_string(),
        xgb_tree_method: "".to_string(),
        xgb_grow_policy: "".to_string(),
        xgb_eval_metric: "".to_string(),
        xgb_early_stopping_rounds: "".to_string(),
        xgb_device: "".to_string(),
        xgb_cv: "".to_string(),
        xgb_interaction_constraints: "".to_string(),
        hyperparameter_optimization_attempts: "".to_string(),
        hyperparameter_optimization_result_display_limit: "".to_string(),
        dask_workers: "".to_string(),
        dask_threads_per_worker: "".to_string(),
    },
)
.await;

builder
    .order_columns(vec!["...", "churn_day", "PREDICTION"])
    .print_table();
dbg!(report);

// B. Invoking a Model
use rgwml::csv_utils::CsvBuilder;
use rgwml::xgb_utils::XgbConfig;

let mut builder = CsvBuilder::from_csv("/home/rgw/Desktop/csv_db/churn_predictions_data.csv");

builder.append_xgb_model_predictions_column(
    "ticket_count, tickets_after_last_payment",             // Param column names
    "PREDICTION",                                           // Prediction column name
    "/path/to/your/model/churn_v8.json"     // Model absolute path
).await;

let _ = builder
    .order_columns(vec!["...", "ticket_count", "tickets_after_last_payment", "PREDICTION"])
    .print_table()
    .save_as("/path/to/your/predictions/churn_v8_predictions_current_portfolio.csv");

CsvBuilder-聚类（scikit-learn）操作

// A. Creating a Model
use rgwml::csv_utils::CsvBuilder;
use rgwml::clustering_utils::ClusteringConfig;
use tokio::runtime::Runtime;
use std::path::PathBuf;
use std::env::current_dir;

let headers = vec![
    "customer_id".to_string(),
    "age".to_string(),
    "annual_income".to_string(),
    "spending_score".to_string(),
];

let data = vec![
    vec!["1".to_string(), "19".to_string(), "15".to_string(), "39".to_string()],
    vec!["2".to_string(), "21".to_string(), "15".to_string(), "81".to_string()],
    vec!["3".to_string(), "20".to_string(), "16".to_string(), "6".to_string()],
    vec!["4".to_string(), "23".to_string(), "16".to_string(), "77".to_string()],
    vec!["5".to_string(), "31".to_string(), "17".to_string(), "40".to_string()],
    vec!["6".to_string(), "22".to_string(), "17".to_string(), "76".to_string()],
    vec!["7".to_string(), "35".to_string(), "18".to_string(), "6".to_string()],
    vec!["8".to_string(), "23".to_string(), "18".to_string(), "94".to_string()],
    vec!["9".to_string(), "64".to_string(), "19".to_string(), "3".to_string()],
    vec!["10".to_string(), "30".to_string(), "19".to_string(), "72".to_string()],
];

let mut builder = CsvBuilder::from_raw_data(headers, data);
let rt = Runtime::new().unwrap();
rt.block_on(async {
    let param_column_names = "age, annual_income, spending_score";
    let cluster_column_name = "CLUSTERING";
    let clustering_config = ClusteringConfig {
        operation: "KMEANS".to_string(),        // Options: KMEANS, DBSCAN, AGGLOMERATIVE, MEAN_SHIFT, GMM, SPECTRAL, BIRCH
        optimal_n_cluster_finding_method: "ELBOW".to_string(),  // FIXED:{n}, ELBOW, SILHOUETTE; Not relevant for MEAN_SHIFT and DBSCAN
        dbscan_eps: "".to_string(),             // Only relevant for DBSCAN
        dbscan_min_samples: "".to_string(),     // Only relevant for DBSCAN
    };
    builder.append_clustering_column(param_column_names, cluster_column_name, clustering_config).await;
});

CsvConverter

use serde_json::json;
use tokio;
use rgwml::csv_utils::CsvConverter;
use rgwml::api_utils::ApiCallBuilder;

// Function to fetch sales data from an API
async fn fetch_sales_data_from_api() -> Result<String, Box<dyn std::error::Error>> {
    let method = "POST";
    let url = "http://example.com/api/sales"; // API URL to fetch sales data

    // Payload for the API call
    let payload = json!({
        "date": "2023-12-21"
    });

    // Performing the API call
    let response = ApiCallBuilder::call(method, url, None, Some(payload))
        .execute()
        .await?;

    Ok(response)
}

// Main function with tokio's async runtime
#[tokio::main]
async fn main() {
    // Fetch sales data and handle potential errors inline
    let sales_data_response = fetch_sales_data_from_api().await.unwrap_or_else(|e| {
        eprintln!("Failed to fetch sales data: {}", e);
        std::process::exit(1); // Exit the program in case of an error
    });

    // Convert the fetched JSON data to CSV
    CsvConverter::from_json(&sales_data_response, "path/to/your/file.csv")
        .expect("Failed to convert JSON to CSV"); // Handle errors in CSV conversion
}

使用was_last_modified_within方法缓存CsvBuilder

示例1：与API调用一起使用

use rgwml::api_utils::ApiCallBuilder;
use rgwml::csv_utils::CsvBuilder;
use serde_json::json;
use tokio;

async fn generate_daily_sales_report() -> Result<(), Box<dyn std::error::Error>> {
    async fn fetch_sales_data_from_api() -> Result<String, Box<dyn std::error::Error>> {
        let method = "POST";
        let url = "http://example.com/api/sales"; // API URL to fetch sales data

        let payload = json!({
            "date": "2023-12-21"
        });

        let response = ApiCallBuilder::call(method, url, None, Some(payload))
            .execute()
            .await?;

        Ok(response)
    }

    let sales_data_response = fetch_sales_data_from_api().await?;

    // Convert the JSON response to CSV format using CsvBuilder
    let mut csv_builder = CsvBuilder::new();
    csv_builder.set_header(vec!["column1", "column2", "column3"]); // Set your headers appropriately
    // Add your rows based on the API response
    // csv_builder.add_row(vec!["value1", "value2", "value3"]);

    csv_builder.save_as("/path/to/daily/sales/report/cache.csv");

    Ok(())
}

#[tokio::main]
async fn main() {
    let cache_path = "/path/to/daily_sales_report.csv";
    let cache_duration_minutes = 1440; // Cache duration set to 1 day

    if !CsvBuilder::was_last_modified_within(cache_path, &cache_duration_minutes.to_string()) {
        if let Err(e) = generate_daily_sales_report().await {
            eprintln!("Failed to generate sales report: {}", e);
        }
    }

    println!("Sales report is ready.");
}

示例2：与从外部数据库检索的CsvBuilder一起使用

use rgwml::csv_utils::CsvBuilder;
use tokio;

async fn generate_builder(cache_path: &str) -> Result<(), Box<dyn std::error::Error>> {
    let query = r#"
    SELECT * FROM your_table
    "#;

    let mut builder = CsvBuilder::from_mysql_query(
        "username",
        "password",
        "host",
        "database",
        &query,
    )
    .await
    .expect("Failed to create CsvBuilder from query");

    builder.save_as(cache_path);

    Ok(())
}

#[tokio::main]
async fn main() {
    let cache_path = "/path/to/cache/your/file.csv";
    let cache_duration_minutes = 1440; // Cache duration set to 1 day

    if !CsvBuilder::was_last_modified_within(cache_path, &cache_duration_minutes.to_string()) {
        if let Err(e) = generate_builder(cache_path).await {
            eprintln!("Failed to generate data: {}", e);
        }
    }

    let mut builder = CsvBuilder::from_csv(cache_path);
    builder.print_table();
}

5. heavy_csv_utils

实例化

示例1：创建一个新的对象

use rgwml::heavy_csv_utils::HeavyCsvBuilder;

let builder = HeavyCsvBuilder::heavy_new()
    .heavy_set_header(vec!["Column1".to_string(), "Column2".to_string(), "Column3".to_string()])
    .heavy_set_data(vec![
        vec![b"Row1-1".to_vec(), b"Row1-2".to_vec(), b"Row1-3".to_vec()],
        vec![b"Row2-1".to_vec(), b"Row2-2".to_vec(), b"Row2-3".to_vec()]
    ])
.heavy_save_as("/path/to/your/file.csv");

builder.heavy_print_table();

示例2：从现有文件加载

use rgwml::heavy_csv_utils::HeavyCsvBuilder;

let builder = HeavyCsvBuilder::heavy_from_csv("/path/to/existing/file.csv");
builder.heavy_print_table();

示例3：从公开可查看的Google Sheets URL加载

use rgwml::heavy_csv_utils::HeavyCsvBuilder;
use tokio::runtime::Runtime;

let rt = Runtime::new().unwrap();
rt.block_on(async {
    let csv_builder = HeavyCsvBuilder::from_publicly_viewable_google_sheet("https://docs.google.com/spreadsheets/d/1U9ozNFwV__c15z4Mp_EWorGwOv6mZPaQ9dmYtjmCPow/edit#gid=272498272").await;

    csv_builder.heavy_print_table();
});

示例4：从裸金属Python可执行文件加载

# A bare metal python executable should:
# - Be executable without an virtual environment, with the `python3 <file_name>.py <arguments>` format;
# - Specify a --uid flag accepting a string value, for the library to retrieve the output correctly
# - Save the output to rgwml_{uid}.json file
# For instance:

import os
import argparse
import json
import mmap
import pandas as pd

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Process some data")
    parser.add_argument('--uid', type=str, help='A unique identifier to name the output json file', required=True)

    parser.add_argument('--file_a_path', type=str, required=True, help='Path to the first CSV file')
    parser.add_argument('--file_b_path', type=str, required=True, help='Path to the second CSV file')
    parser.add_argument('--join_type', type=str, required=True, choices=['LEFT_JOIN', 'RIGHT_JOIN', 'OUTER_FULL_JOIN', 'UNION', 'BAG_UNION'], help='Type of join operation to perform')
    parser.add_argument('--file_a_ref_column', type=str, required=False, help='Reference column in the first CSV file')
    parser.add_argument('--file_b_ref_column', type=str, required=False, help='Reference column in the second CSV file')


    # Some processing logic that creates `df`

    headers = df.columns.tolist()
    rows = df.values.tolist()

    output = {
        "headers": headers,
        "rows": [[str(item) for item in row] for row in rows],
    }

    json_output = json.dumps(output, indent=4)
    filename = f"rgwml_{uid}.json"

    with open(filename, 'wb') as f:
        f.write(b' ' * len(json_output))

    with open(filename, 'r+b') as f:
        mm = mmap.mmap(f.fileno(), 0)
        mm.write(json_output.encode('utf-8'))
        mm.close()

// Now, you can load the result of the above directly into your RGWML workflow, in the manner shown below.

use rgwml::heavy_csv_utils::HeavyCsvBuilder;
use tokio::runtime::Runtime;
use std::path::PathBuf;

let current_dir = std::env::current_dir().unwrap();
let executable_path = current_dir.join("python_executables/dask_joiner_connect.py");
let executable_path_str = executable_path.to_str().unwrap();

// Append the file name to the directory path
let csv_path_a = current_dir.join("test_file_samples/joining_test_files/join_file_a.csv");
let csv_path_a_str = csv_path_a.to_str().unwrap();

let csv_path_b = current_dir.join("test_file_samples/joining_test_files/join_file_b.csv");
let csv_path_b_str = csv_path_b.to_str().unwrap();

let rt = Runtime::new().unwrap();
rt.block_on(async {
    let args = vec![
        ("--file_a_path", csv_path_a_str),
        ("--file_b_path", csv_path_b_str),
        ("--join_type", "LEFT_JOIN"),
        ("--file_a_ref_column", "id"),
        ("--file_b_ref_column", "id")
    ];

    let mut builder = HeavyCsvBuilder::heavy_from_bare_metal_python_executable(
        &executable_path_str,
        args,
        ).await;

});