Ai00-Core

这是Ai00服务器的核心库。它提供了以下功能：

• 模型/LoRA/初始状态加载，自动检测版本；
• 采样器；
• BNF集成；
• 状态缓存；
• 会话管理。

此crate的目的是提供一个无状态的本地推理API。

指南

首先启动运行时，一个异步任务，用于处理所有后台操作（例如，模型运行时、缓存、会话队列）

use ai00_core::model_route;

let (sender, receiver) = flume::unbounded::<ThreadRequest>();
tokio::spawn(model_route(receiver));

然后用户可以通过发送ThreadRequest与运行时通信，这些是请求运行时执行各种操作的命令。查看其定义

pub enum ThreadRequest {
    /// Acquire a list of current available adapters.
    Adapter(Sender<AdapterList>),
    /// Get the current runtime info.
    Info(Sender<RuntimeInfo>),
    /// Request the runtime to complement a prompt.
    Generate {
        request: Box<GenerateRequest>,
        tokenizer: Arc<Tokenizer>,
        sender: Sender<Token>,
    },
    /// Reload the runtime with custom config.
    Reload {
        request: Box<ReloadRequest>,
        sender: Option<Sender<bool>>,
    },
    /// Unload the runtime.
    Unload,
    /// Additionally load an initial state.
    StateLoad {
        request: reload::State,
        sender: Option<Sender<bool>>,
    },
    /// Unload an initial state given its id.
    StateUnload(StateId),
    /// Save the current model with config.
    Save {
        request: SaveRequest,
        sender: Sender<bool>,
    },
}