Windsock - 一个数据库基准测试框架

Windsock是一个通用的数据库基准测试框架。

你将做什么

• 提供你自己的Rust异步兼容数据库驱动程序
• 定义你的基准逻辑，该逻辑将一些简单的统计信息返回给Windsock
• 定义你的基准池

Windsock将做什么

• 提供命令行界面，你可以
  • 查询可用的基准
  • 有选择地运行基准
  • 将基准结果处理成可读的表格

Windsock适合于

• 在数据库或服务开发过程中迭代测试性能（对于微基准测试，你可能需要不同的工具）
• 调查你打算使用的数据库上不同工作负载的性能。

定义基准

要使用Windsock，创建一个导入Windsock的Rust crate

windsock = "0.1"

然后像这样实现crate（简化版）

fn main() {
    // Define our benchmarks and give them to windsock
    Windsock::new(vec![
        Box::new(CassandraBench { topology: Topology::Cluster3 }),
        Box::new(CassandraBench { topology: Topology::Single })
    ])
    // Hand control of the app over to windsock
    // Windsock processes CLI args, possibly running benchmarks and then terminates.
    .run();
}

pub struct CassandraBench { topology: Topology }

#[async_trait]
impl Bench for CassandraBench {
    // define tags that windsock will use to filter and name the benchmark instance
    fn tags(&self) -> HashMap<String, String> {
        [
            ("name".to_owned(), "cassandra".to_owned()),
            (
                "topology".to_owned(),
                match &self.topology {
                    Topology::Single => "single".to_owned(),
                    Topology::Cluster3 => "cluster3".to_owned(),
                },
            ),
        ]
        .into_iter()
        .collect()
    }

    // the benchmark logic for this benchmark instance
    async fn run(&self, runtime_seconds: usize, operations_per_second: Option<u64>, reporter: UnboundedSender<Report>) {
        // bring up the DB
        let _handle = init_cassandra();

        // create the DB driver session
        let session = init_session().await;

        // spawn tokio tasks to concurrently hit the database
        // The exact query is defined in `run_one_operation` below
        BenchTaskCassandra { session }.spawn_tasks(reporter.clone(), operations_per_second).await;

        // tell windsock to begin benchmarking
        reporter.send(Report::Start).unwrap();
        let start = Instant::now();

        // run the bench for the time requested by the user on the CLI (defaults to 15s)
        tokio::time::sleep(Duration::from_secs(runtime_seconds)).await;

        // tell windsock to finalize the benchmark
        reporter.send(Report::FinishedIn(start.elapsed())).unwrap();
    }
}

// This struct is cloned once for each tokio task it will be run in.
#[derive(Clone)]
struct BenchTaskCassandra {
    session: Arc<Session>,
}

#[async_trait]
impl BenchTask for BenchTaskCassandra {
    async fn run_one_operation(&self) -> Result<(), String> {
        self.session.query("SELECT * FROM table").await
    }
}

待办事项：记录以独立crate和cargo bench运行Windsock的方法。

此示例简化以供演示，请参阅 examples/cassandra.rs 以获取完整的工作示例。

运行基准

然后我们运行我们的crate以运行基准并查看结果，如下所示

> cargo windsock run-local
... benchmark running logs
> cargo windsock results
Results for cassandra
           topology   ──single ─cluster3
Measurements ═══════════════════════════
   Operations Total     750762    372624
 Operations Per Sec      83418     41403
             Min       0.255ms   0.229ms
               1       0.389ms   0.495ms
               2       0.411ms   0.571ms
               5       0.460ms   0.714ms
              10       0.567ms   0.876ms
              25       1.131ms   1.210ms
              50       1.306ms   1.687ms
              75       1.519ms   2.600ms
              90       1.763ms   4.881ms
              95       2.132ms   7.542ms
              98       2.588ms  14.008ms
              99       2.951ms  19.297ms
              99.9     7.952ms  40.896ms
              99.99   25.559ms  80.692ms

待办事项：将其转换为更炫酷的比较，并使用图片包含着色

如何在Windsock中执行各种任务

运行所有基准

> cargo windsock run-local

运行带有匹配标签的基准，并在一个表格中查看所有结果

> cargo windsock run-local db=kafka OPS=1000 topology=single # run benchmarks matching some tags
> cargo windsock results # view the results of the benchmarks with the same tags in a single table

迭代比较与先前实现的差异

> git checkout main # checkout original implementation
> cargo windsock run-local # run all benchmarks
> cargo windsock baseline-set # set the last benchmark run as the baseline
> vim src/main.rs # modify implementation
> cargo windsock run-local # run all benchmarks, every result is compared against the baseline
> cargo windsock results # view those results in a nice table
> vim src/main.rs # modify implementation again
> cargo windsock run-local # run all benchmarks, every result is compared against the baseline

在云端运行基准（简单）

# create cloud resources, run benchmarks and then cleanup - all in one command
> cargo windsock cloud-setup-run-cleanup

迭代比较与先前实现的差异（在远程云端运行）

# Setup the cloud resources and then form a baseline
> git checkout main # checkout original implementation
> cargo windsock cloud-setup db=kafka # setup the cloud resources required to run all kafka benchmarks
> cargo windsock cloud-run db=kafka # run all the kafka benchmarks in the cloud
> cargo windsock baseline-set # set the last benchmark run as the baseline

# Make a change and and measure the effect
> vim src/main.rs # modify implementation
> cargo windsock cloud-run db=kafka # run all benchmarks, every result is compared against the baseline
> cargo windsock results # view those results in a nice table, compared against the baseline

# And again
> vim src/main.rs # modify implementation again
> cargo windsock cloud-run db=kafka # run all benchmarks, every result is compared against the baseline

# And finally...
> cargo windsock cloud-cleanup # Terminate all the cloud resources now that we are done

生成图形网页

待办事项：计划中，但尚未实现

> cargo windsock local-run # run all benches
> cargo windsock generate-webpage # generate a webpage from the results