CESU-8 编码/解码器，用于 Rust

在普通 UTF-8 和 CESU-8 编码（维基百科，Unicode 联合会）之间进行转换。

CESU-8 将基本多语言平面的字符编码为两个 UTF-16 代理字符，然后进一步重新编码为无效的、3 字节 UTF-8 字符。这意味着 4 字节 UTF-8 序列成为 6 字节 CESU-8 序列。

请注意，CESU-8 仅适用于紧密耦合系统内部使用，不适用于数据交换。

在处理 Java、Oracle 或 MySQL、尝试在补充多语言平面或补充汉字平面上存储表情符号、象形文字或其他字符时，有时需要此编码。

我们还支持 Java 的 修改过的 UTF-8 编码，该编码还使用多字节 UTF-8 序列编码 \0。

命令行工具

安装：cargo install cesu8str --features=build-binary

cesu8str

MUTF8/CESU8 与 UTF8 之间的编码/解码

$ cesu8str --help
Converts files or standard IO streams between standard UTF8 and CESU8, or the JVM's modified UTF-8.
Note that the default Windows' console does not support non-UTF8 sequences - attempting to type/print them will result in 
an error.

This tool will immediately exit upon finding an invalid character sequence.

EXIT CODES:
0 - if completed normally
1 - if an IO error has occured
2 - if an encoding error has occured (invalid/incomplete character sequences/etc)


Usage: cesu8str [OPTIONS] [INPUT] [OUTPUT]

Arguments:
  [INPUT]   The input file. Defaults to stdin if '-' or not set
  [OUTPUT]  The output file. Defaults to stdout if '-' or not set

Options:
  -j, --java     Toggles the use of the JVM's modified UTF8. In effect, it encodes nul-bytes as 0xC0,0x80 while nul-bytes 
are left alone in normal mode
  -d, --decode   Decodes CESU8 text into standard UTF8. By default, this tool encodes UTF8 to CESU8
  -h, --help     Print help information
  -V, --version  Print version information

count_codepoints - 计算 UTF8/16 及其朋友的 Unicode 代码点长度

更多是开发期间验证功能的一种工具。

$ count_codepoints --help
USAGE: count_codepoints [--json] [file] [KB_chunk_size=4]

Reads a file and emits the number of occurences differing lengths of UTF-8 codepoints on stdout.

If `file` is not supplied, stdin will be read.

Exit Codes:
0       Success
1       Command-line error
2       IO Error

示例输出

$ count_codepoints ./test_files/random.txt      
len4: 1001
len3: 1013
len2: 966
ascii: 1095
nul: 0
total (chars/bytes): 4075/10070

$ count_codepoints ./test_files/random.cesu8.txt
len4: 0
len3: 3015
len2: 966
ascii: 1095
nul: 0
total (chars/bytes): 5076/1207

许可证

部分代码改编自 Rust 的 src/libcore/str.rs 文件。此代码受 LICENSE-RUST.txt 许可并由 Rust 项目开发者和 Rust 贡献者拥有版权，如该文件所述。

本项目中大部分代码是从Eric Kidd的cesu8crate分叉并改编的。改编的代码按照分叉时的原始项目条款进行分发。

本项目中新增的代码也按照相同的条款进行分发。