ssubmit

无需创建提交脚本即可提交 sbatch 作业

• 动机
• 安装
• 用法

动机

本项目受以下动机驱动：我希望能够只提交命令作为作业，而不想花时间编写提交脚本。

ssubmit 包装了整个过程，让你过上最好的生活 #blessed。

安装

总结

curl -sSL install.ssubmit.mbh.sh | sh
# or with wget
wget -nv -O - install.ssubmit.mbh.sh | sh

你可以这样传递给脚本的选项

$ curl -sSL install.ssubmit.mbh.sh | sh -s -- --help
install.sh [option]

Fetch and install the latest version of ssubmit, if ssubmit is already
installed it will be updated to the latest version.

Options
        -V, --verbose
                Enable verbose output for the installer

        -f, -y, --force, --yes
                Skip the confirmation prompt during installation

        -p, --platform
                Override the platform identified by the installer [default: apple-darwin]

        -b, --bin-dir
                Override the bin installation directory [default: /usr/local/bin]

        -a, --arch
                Override the architecture identified by the installer [default: x86_64]

        -B, --base-url
                Override the base URL used for downloading releases [default: https://github.com/mbhall88/ssubmit/releases]

        -h, --help
                Display this help message

Cargo

$ cargo install ssubmit

从源代码构建

$ git clone https://github.com/mbhall88/ssubmit.git
$ cd ssubmit
$ cargo build --release
$ target/release/ssubmit -h

用法

提交一个名为 "foo" 的 rsync 作业，并请求 350MB 的内存和一周的时间限制

$ ssubmit -m 350m -t 1w foo "rsync -az src/ dest/"

提交需要 8 个 CPU 的作业

$ ssubmit -m 16g -t 1d align "minimap2 -t 8 ref.fa query.fq > out.paf" -- -c 8

$ ssubmit -h
Submit sbatch jobs without having to create a submission script

Usage: ssubmit [OPTIONS] <NAME> <COMMAND> [-- <REMAINDER>...]

Arguments:
  <NAME>          Name of the job
  <COMMAND>       Command to be executed by the job
  [REMAINDER]...  Options to be passed on to sbatch

Options:
  -o, --output <OUTPUT>    File to write job stdout to. (See `man sbatch | grep -A 3 'output='`) [default: %x.out]
  -e, --error <ERROR>      File to write job stderr to. (See `man sbatch | grep -A 3 'error='`) [default: %x.err]
  -m, --mem <size[unit]>   Specify the real memory required per node. e.g., 4.3kb, 7 Gb, 9000, 4.1MB become 5KB, 7000M, 9000M, and 5M, respectively [env: SSUBMIT_MEMORY=] [default: 1G]
  -t, --time <TIME>        Time limit for the job. e.g. 5d, 10h, 45m21s (case-insensitive) [env: SSUBMIT_TIME=] [default: 1d]
  -S, --shebang <SHEBANG>  The shell shebang for the submission script [env: SSUBMIT_SHEBANG=] [default: "#!/usr/bin/env bash"]
  -s, --set <SET>          Options for the set command in the shell script [env: SSUBMIT_SET=] [default: "euxo pipefail"]
  -n, --dry-run            Print the sbatch command and submission script that would be executed, but do not execute them
  -T, --test-only          Return an estimate of when the job would be scheduled to run given the current queue. No job is actually submitted. [sbatch --test-only]
  -h, --help               Print help (see more with '--help')
  -V, --version            Print version

ssubmit 调用的基本结构如下

ssubmit [OPTIONS] <NAME> <COMMAND> [-- <REMAINDER>...]

NAME 是作业的名称（在 sbatch 中的 --job-name 参数）。

COMMAND 是你想要由作业执行的内容。它 必须 被引用（单引号或双引号）。

REMAINDER 是任何（可选的）你想要传递给 sbatch 的特定选项。这些必须跟在 COMMAND 后面，并使用 -- 分隔。

内存

内存（-m,--mem）比 sbatch --mem 选项更易于使用。例如，你可以传递 -m 0.5g，而 ssubmit 会将其解释并转换为 500M。单位不区分大小写。超过 1M 的内存值将向上取整到最接近的整数。例如，1.1M 将向上取整到 2M。如果你想要使用集群的默认内存限制，则只需传递 -m 0。

为了简化，所有超过一兆字节的值都以兆字节的形式传递给 sbatch，例如，1.1G 将传递为 1100M。

环境变量 SSUBMIT_MEM 可以设置为默认内存限制。这可以通过传递 -m 来覆盖。

时间

与内存类似，时间（-t,--time）的目的是简单。如果你想设置三天的限制，只需传递 -t 3d。想设置两个半小时？那么 -t 2h30m 就可以。如果你想只使用集群的默认限制，只需传递 -t 0。你也可以直接传递 sbatch 使用的 sbatch 时间格式，这将无缝传递。要查看支持的时间单位完整列表，请查看 duration-str 仓库。需要注意的是，传递单个数字，不带单位，slurm 会将其解释为分钟。然而，在 5m3 的示例中不提供单位会被解释为 5 分钟和 3 秒。

环境变量 SSUBMIT_TIME 可以设置为默认时间限制。这可以通过传递 -t 来覆盖。

干燥运行

您可以使用干燥运行（-n,--dry-run）来查看 ssubmit 会做什么，而无需实际提交作业。这将打印 sbatch 命令以及将要提供的提交脚本。

$ ssubmit -n -m 4g -t 1d dry "rsync -az src/ dest/" -- -c 8
[2022-01-19T08:58:58Z INFO  ssubmit] Dry run requested. Nothing submitted
sbatch -c 8 <script>
=====<script>=====
#!/usr/bin/env bash
#SBATCH --job-name=dry
#SBATCH --mem=4000M
#SBATCH --time=24:0:0
#SBATCH --error=%x.err
#SBATCH --output=%x.out
set -euxo pipefail

rsync -az src/ dest/
=====<script>=====

脚本设置

脚本的默认 shebang 为 #!/usr/bin/env bash。但是，如果您想使用其他内容，请使用 -S,--shebang 传递或设置环境变量 SSUBMIT_SHEBANG。

此外，我们默认使用 set -euxo pipefail，这将当命令以非零退出代码退出时退出（e）、当尝试使用未设置的变量时出错（u）、将所有运行的命令打印到 stderr（x），以及当管道中的命令失败时退出（-o pipefail）。您可以使用 -s,--set 或环境变量 SSUBMIT_SET 来更改这些设置。您可以通过传递 -s '' 来关闭此功能。

日志文件

默认情况下，作业的 stderr 和 stdout 分别发送到 %x.err 和 %x.out。其中 %x 是作业名称的文件名模式。因此，如果作业名称为 foo，stderr 文件将是 foo.err。您可以在 文档 中查看所有可用的模式。当然，您不必使用模式。

完整用法

$ ssubmit --help
ssubmit 0.2.0
Michael Hall <[email protected]>
Submit sbatch jobs without having to create a submission script

-----------
# EXAMPLES
-----------

Submit a simple rsync command with a 600MB memory limit.

$ ssubmit -m 600m rsync_my_data "rsync -az src/ dest/"

Submit a command that involves piping the output into another command. sbatch options
are passed after a `--`.

$ ssubmit -m 4G align "minimap2 -t 8 ref.fa reads.fq | samtools sort -o sorted.bam" -- -c 8
Submit sbatch jobs without having to create a submission script

-----------
# EXAMPLES
-----------

Submit a simple rsync command with a 600MB memory limit.

$ ssubmit -m 600m rsync_my_data "rsync -az src/ dest/"

Submit a command that involves piping the output into another command. sbatch options
are passed after a `--`.

$ ssubmit -m 4G align "minimap2 -t 8 ref.fa reads.fq | samtools sort -o sorted.bam" -- -c 8

Usage: ssubmit [OPTIONS] <NAME> <COMMAND> [-- <REMAINDER>...]

Arguments:
  <NAME>
          Name of the job

          See `man sbatch | grep -A 2 'job-name='` for more details.

  <COMMAND>
          Command to be executed by the job

  [REMAINDER]...
          Options to be passed on to sbatch

Options:
  -o, --output <OUTPUT>
          File to write job stdout to. (See `man sbatch | grep -A 3 'output='`)

          Run `man sbatch | grep -A 37 '^filename pattern'` to see available patterns.

          [default: %x.out]

  -e, --error <ERROR>
          File to write job stderr to. (See `man sbatch | grep -A 3 'error='`)

          Run `man sbatch | grep -A 37 '^filename pattern'` to see available patterns.

          [default: %x.err]

  -m, --mem <size[unit]>
          Specify the real memory required per node. e.g., 4.3kb, 7 Gb, 9000, 4.1MB become 5KB, 7000M, 9000M, and 5M, respectively.

          If no unit is specified, megabytes will be used, as per the sbatch default. The value will be rounded up to the nearest megabyte. If the value is less than 1M, it will be rounded up to the nearest kilobyte. See `man sbatch | grep -A 4 'mem='` for the full details.

          [env: SSUBMIT_MEMORY=]
          [default: 1G]

  -t, --time <TIME>
          Time limit for the job. e.g. 5d, 10h, 45m21s (case-insensitive)

          Run `man sbatch | grep -A 7 'time=<'` for more details. If a single digit is passed, it will be passed straight to sbatch (i.e. minutes). However, 5m5 will be considered 5 minutes and 5 seconds.

          [env: SSUBMIT_TIME=]
          [default: 1d]

  -S, --shebang <SHEBANG>
          The shell shebang for the submission script

          [env: SSUBMIT_SHEBANG=]
          [default: "#!/usr/bin/env bash"]

  -s, --set <SET>
          Options for the set command in the shell script

          For example, to exit when the command exits with a non-zero code and to treat unset variables as an error during substitution, pass 'eu'. Pass '' or "" to set nothing

          [env: SSUBMIT_SET=]
          [default: "euxo pipefail"]

  -n, --dry-run
          Print the sbatch command and submission script that would be executed, but do not execute them

  -T, --test-only
          Return an estimate of when the job would be scheduled to run given the current queue. No job is actually submitted. [sbatch --test-only]

  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version