🏋️‍♀️ bevy_rl

为Bevy引擎提供强化学习

使用Bevy引擎构建OpenAI Gym环境，以训练能够从屏幕像素或定义的观察状态中学习的AI代理。

📝功能

• 一套API用于实现OpenAI Gym接口，例如 reset、step、render 及相关模拟器状态
• 多代理支持
• 将屏幕像素渲染到RAM缓冲区 — 用于训练使用原始像素的代理
• REST API用于控制代理

👩‍💻用法

1. 定义动作和观察空间

观察空间需要是 Serializable 以便REST API能够工作。

// Action space
#[derive(Default)]
pub struct Actions {
    // actuator_signals: [f32; 3],
}

// Observation space
#[derive(Default, Serialize, Clone)]
pub struct Observations {
    // agent_coords: [(f32, f32); 16],
}

2. 启用AI Gym插件

宽度和高度应超过256，否则wgpu会panic。

// Setup bevy_rl
let ai_gym_state = AIGymState::<Actions, State>::new(AIGymSettings {
    width: u32,              // Width and height of the screen
    height: u32,             // ...
    num_agents: 1,           // Number of agents — each will get a camera handle
    render_to_buffer: false, // You can disable rendering to buffer
    pause_interval: 0.01,    // 100 Hz
    ..default()
});
app.insert_resource(ai_gym_state)
    .add_plugins(AIGymPlugin::<Actions, Observations>::default());

2.1（可选）启用缓冲区渲染

如果您的环境需要导出原始像素，您需要将渲染目标附加到您想要从中导出它们的每个相机。渲染目标会在每一帧从GPU内存复制到RAM缓冲区，以便可以使用REST API访问。

pub(crate) fn spawn_cameras(
    ai_gym_state: Res<AIGymState<Actions, Observations>>,
) {
    let mut ai_gym_state = ai_gym_state.lock().unwrap();
    let ai_gym_settings = ai_gym_state.settings.clone();

    for i in 0..ai_gym_settings.num_agents {
        let render_image_handle = ai_gym_state.render_image_handles[i as usize].clone();
        let render_target = RenderTarget::Image(render_image_handle);
        let camera_bundle = Camera3dBundle {
            camera: Camera {
                target: render_target,  // Render target is baked in bevy_rl and used to export pixels
                priority: -1,           // set to -1 to render at the firstmost pass
                ..default()
            },
            ..default()
        };
        commands.spawn(camera_bundle);
    }
}

4. 处理bevy_rl事件

bevy_rl将通过事件与您的环境通信。这些事件由REST API或 bevy_rl.SimulationPauseTimer（以给定的间隔暂停模拟 AIGymSettings.pause_interval）启动。

以下是如何处理这些事件的示例

// EventPauseResume
fn bevy_rl_pause_request(
    mut pause_event_reader: EventReader<EventPauseResume>,
    ai_gym_state: Res<AIGymState<Actions, State>>,
) {
    for _ in pause_event_reader.iter() {
        // Pause simulation (physics engine)
        // ...
        // Collect state into serializable struct
        let env_state = Observations(...);
        // Set bevy_rl gym state
        let mut ai_gym_state = ai_gym_state.lock().unwrap();
        ai_gym_state.set_env_state(env_state);
    }
}

// EventControl
fn bevy_rl_control_request(
    mut pause_event_reader: EventReader<EventControl>,
    mut simulation_state: ResMut<State<SimulationState>>,
) {
    for control in pause_event_reader.iter() {
        let unparsed_actions = &control.0;
        for i in 0..unparsed_actions.len() {
            if let Some(unparsed_action) = unparsed_actions[i].clone() {
                let action: Vec<f64> = serde_json::from_str(&unparsed_action).unwrap();
                // Pass control inputs to your agents
                // ...
            }
        }
        // Resume simulation (physics engine)
        // ...
        // Return to running state; note that it uses pop/push to avoid
        // entering `SystemSet::on_enter(SimulationState::Running)` which initialized game world anew
        simulation_state.pop().unwrap();
    }
}

/// Handle bevy_rl::EventReset
pub(crate) fn bevy_rl_reset_request(
    mut reset_event_reader: EventReader<EventReset>,
    mut commands: Commands,
    mut walls: Query<Entity, &Wall>,
    mut players: Query<(Entity, &Actor)>,
    mut simulation_state: ResMut<State<SimulationState>>,
    ai_gym_state: Res<AIGymState<Actions, Observations>>,
) {
    if reset_event_reader.iter().count() == 0 {
        return;
    }

    // Reset envrionment state here

    // Return simulation in Running state
    simulation_state.set(SimulationState::Running).unwrap();

    // tell bevy_rl that environment is reset and return response to REST API
    let ai_gym_state = ai_gym_state.lock().unwrap();
    ai_gym_state.send_reset_result(true);
}

注册系统以处理bevy_rl事件。

app.add_systems(
    Update,
    (
        .with_system(bevy_rl_control_request)
        .with_system(bevy_rl_reset_request)
        .with_system(bevy_rl_pause_request),
    ).in_set(SimulationState::PausedForControl)
);

💻 AIGymState API

这些方法在 AIGymState 资源上可用。您应该使用它们来更改bevy_rl的内部状态。

🌐 REST API

可以通过 REST API 访问 bevy_rl 启用的环境。以下是可用端点的列表

bevy_rl_shooter 实现了一个 Python 包装器的示例。

✍️ 示例

• bevy_rl_shooter — 示例 FPS 项目
• bevy_quadruped_neural_control — 使用 bevy_mujoco 和 bevy_rl 的四足运动