Convert DefaultMessageAction from bincode to speedy serialization

[fia0]

Extracted from Till's notes:

Currently uses bincode, unstable format, and slower/larger than speedy

This still seems like a good change, we currently use partially speedy and partially bincode. Let's unify this.

[fia0]

I made some comparison to already existing benchmarks (https://github.com/djkoloski/rust_serialization_benchmark) on the whole ordeal of speedy vs bincode vs others. The reason why is that we have to resolve a conflict between multiple aspects. This comparison basically breaks down into the three categories:

1. Standardization (How is the format specified in which we serialize to? Can multiple crates interact with this format?)
2. Maintenance (How likely is it that crate x is supported in 5-10 years?)
3. Performance (Okay that's just performance)

From the 1. point most options are rather similar they specify more or less their own format and read it again later, more often than not without any information about the version of format this data has been serialized as (less important to us right now). Only notable exceptions are something like capnproto, flatbuffers, cbor, or protobuf. Although the focus of these is just a little bit different from our preferences as they are often network bound they try to radically minimize size while speed is for our purposes essential. This brings us to the "high-performing" crates such as rkyv, speedy, bincode, and others. These are a collection of crates trying to minimize size while preserving fast serialization and deserialization. For any purposes concerning us they have an equal feature set, only differing in their state of maintenance and specification formality. From these rkyv and bincode seem to be the most robust in terms of maintenance. speedy is a bit mixed but seems to be still active. Performance-wise speedy seems to be the fasted while still retaining actual resolved types, rkyv uses "archived" types which allows it to skip deserialization in these cases, which is very interesting considering we could replace the packed leafs with such mechanic. Lastly is bincode which is much slower depending on the serde traits, although this is in the process of changing with version 2.0 https://docs.rs/bincode/2.0.0-rc.2/bincode/index.html containing there own serialization/deserialization traits to push performance further. The performance comparison between these libraries (and including many others) can be seen below:

These benchmarks utilize the most recent versions of speedy, rkyv and bincode with each of those using their own traits to serialize data.

Last updated: 2022-12-21

log

This data set is composed of HTTP request logs that are small and contain many strings.

Raw Data

For operations, time per iteration; for size, bytes. Lower is better.

Serialize / deserialize speed and size

Format / Lib	Serialize	Deserialize	Size	Zlib	Zstd
abomonation	187.77 µs	2.0146 ms*	1705800	506924	403407
bare	675.82 µs	2.6768 ms	765778	312739	264630
bincode	495.40 µs	3.4650 ms	741295	305030	257153
borsh	422.22 µs	2.5783 ms	885780	363280	286514
bson	2.3295 ms	8.6115 ms	1924682	537661	376270
capnp	668.47 µs	†	1443216	509618	428649
cbor	1.7745 ms	5.1908 ms	1407835	407372	324081
flatbuffers	1.6054 ms	†	1276368	469962	388832
nachricht	5.5145 ms	4.6747 ms	818669	334639	285514
postcard	354.69 µs	2.8369 ms	724953	303462	253747
prost	2.6816 ms* 494.19 µs*	3.2238 ms	764951	269811	227947
rkyv	290.34 µs	2.0744 ms* 2.7254 ms*	1011488	392809	331932
rmp	1.3921 ms	3.6198 ms	784997	326654	278219
ron	14.545 ms	14.137 ms	1607459	452648	349713
scale	536.17 µs	2.7496 ms	765778	312771	264518
serde_json	3.3429 ms	7.2958 ms	1827461	474358	361090
simd-json	3.5495 ms	5.3092 ms	1827461	474358	361090
speedy	209.88 µs	2.2695 ms	885780	363280	286514
alkahest	213.19 µs	†	1045784	454748	389424
dlhn	610.98 µs	3.0040 ms	724953	302512	253629

Zero-copy deserialization speed

Format / Lib	Access	Read	Update
abomonation	33.784 µs*	55.041 µs*	‡
capnp	70.568 ns*	303.37 µs*	‡
flatbuffers	2.6519 ns* 1.6572 ms*	86.437 µs* 1.7323 ms*	‡
rkyv	1.2381 ns* 650.73 µs*	15.212 µs* 667.44 µs*	12.501 µs
alkahest	2.1851 ns*	32.904 µs*	‡

Comparison

Relative to best. Higher is better.

Serialize / deserialize speed and size

Format / Lib	Serialize	Deserialize	Size	Zlib	Zstd
abomonation	100.00%	100.00%*	42.50%	53.23%	56.51%
bare	27.78%	75.26%	94.67%	86.27%	86.14%
bincode	37.90%	58.14%	97.80%	88.45%	88.64%
borsh	44.47%	78.14%	81.84%	74.27%	79.56%
bson	8.06%	23.39%	37.67%	50.18%	60.58%
capnp	28.09%	†	50.23%	52.94%	53.18%
cbor	10.58%	38.81%	51.49%	66.23%	70.34%
flatbuffers	11.70%	†	56.80%	57.41%	58.62%
nachricht	3.41%	43.10%	88.55%	80.63%	79.84%
postcard	52.94%	71.01%	100.00%	88.91%	89.83%
prost	7.00%* 38.00%*	62.49%	94.77%	100.00%	100.00%
rkyv	64.67%	97.12%* 73.92%*	71.67%	68.69%	68.67%
rmp	13.49%	55.66%	92.35%	82.60%	81.93%
ron	1.29%	14.25%	45.10%	59.61%	65.18%
scale	35.02%	73.27%	94.67%	86.26%	86.17%
serde_json	5.62%	27.61%	39.67%	56.88%	63.13%
simd-json	5.29%	37.95%	39.67%	56.88%	63.13%
speedy	89.47%	88.77%	81.84%	74.27%	79.56%
alkahest	88.08%	†	69.32%	59.33%	58.53%
dlhn	30.73%	67.06%	100.00%	89.19%	89.87%

Zero-copy deserialization speed

Format / Lib	Access	Read	Update
abomonation	0.00%*	27.64%*	‡
capnp	1.75%*	5.01%*	‡
flatbuffers	46.69%* 0.00%*	17.60%* 0.88%*	‡
rkyv	100.00%* 0.00%*	100.00%* 2.28%*	100.00%
alkahest	56.66%*	46.23%*	‡

mesh

This data set is a single mesh. The mesh contains an array of triangles, each of which has three vertices and a normal vector.

Raw Data

For operations, time per iteration; for size, bytes. Lower is better.

Serialize / deserialize speed and size

Format / Lib	Serialize	Deserialize	Size	Zlib	Zstd
abomonation	594.47 µs	607.33 µs*	6000024	5380836	5345890
bare	5.0923 ms	2.3208 ms	6000003	5380817	5345900
bincode	908.57 µs	4.2226 ms	6000005	5380818	5345897
borsh	4.5040 ms	5.1952 ms	6000004	5380818	5345889
bson	44.820 ms	97.449 ms	23013911	9211138	7497811
capnp	9.6079 ms	†	14000088	6729881	6051062
cbor	36.266 ms	43.007 ms	13122324	7527423	6759658
flatbuffers	1.1999 ms	†	6000024	5380800	5345910
nachricht	120.62 ms	31.332 ms	8125037	6495174	6386940
postcard	934.24 µs	2.1900 ms	6000003	5380817	5345900
prost	11.300 ms* 9.1391 ms*	16.579 ms	8750000	6683814	6421871
rkyv	1.0120 ms	907.69 µs* 867.19 µs*	6000008	5380822	5345892
rmp	14.913 ms	16.801 ms	8125006	6496879	6391037
ron	176.25 ms	229.46 ms	22192885	9009575	8138755
scale	4.0471 ms	4.0424 ms	6000004	5380818	5345889
serde_json	83.944 ms	77.683 ms	26192883	9612105	8586741
simd-json	85.686 ms	180.96 ms	39152823	16587283	14549214
speedy	591.73 µs	632.35 µs	6000004	5380818	5345889
alkahest	1.0034 ms	†	6000008	5380823	5345890
dlhn	5.1863 ms	8.1562 ms	6000003	5380817	5345900

Zero-copy deserialization speed

Format / Lib	Access	Read	Update
abomonation	2.1446 ns*	262.73 µs*	‡
capnp	126.16 ns*	4.4321 ms*	‡
flatbuffers	2.6178 ns* 38.147 ns*	66.951 µs* 66.160 µs*	‡
rkyv	1.2448 ns* 13.044 ns*	67.285 µs* 67.288 µs*	273.31 µs
alkahest	2.1931 ns*	68.231 µs*	‡

Comparison

Relative to best. Higher is better.

Serialize / deserialize speed and size

Format / Lib	Serialize	Deserialize	Size	Zlib	Zstd
abomonation	99.54%	100.00%*	100.00%	100.00%	100.00%
bare	11.62%	26.17%	100.00%	100.00%	100.00%
bincode	65.13%	14.38%	100.00%	100.00%	100.00%
borsh	13.14%	11.69%	100.00%	100.00%	100.00%
bson	1.32%	0.62%	26.07%	58.42%	71.30%
capnp	6.16%	†	42.86%	79.95%	88.35%
cbor	1.63%	1.41%	45.72%	71.48%	79.09%
flatbuffers	49.31%	†	100.00%	100.00%	100.00%
nachricht	0.49%	1.94%	73.85%	82.84%	83.70%
postcard	63.34%	27.73%	100.00%	100.00%	100.00%
prost	5.24%* 6.47%*	3.66%	68.57%	80.50%	83.25%
rkyv	58.47%	66.91%* 70.03%*	100.00%	100.00%	100.00%
rmp	3.97%	3.61%	73.85%	82.82%	83.65%
ron	0.34%	0.26%	27.04%	59.72%	65.68%
scale	14.62%	15.02%	100.00%	100.00%	100.00%
serde_json	0.70%	0.78%	22.91%	55.98%	62.26%
simd-json	0.69%	0.34%	15.32%	32.44%	36.74%
speedy	100.00%	96.04%	100.00%	100.00%	100.00%
alkahest	58.97%	†	100.00%	100.00%	100.00%
dlhn	11.41%	7.45%	100.00%	100.00%	100.00%

Zero-copy deserialization speed

Format / Lib	Access	Read	Update
abomonation	58.04%*	25.18%*	‡
capnp	0.99%*	1.49%*	‡
flatbuffers	47.55%* 3.26%*	98.82%* 100.00%*	‡
rkyv	100.00%* 9.54%*	98.33%* 98.32%*	100.00%
alkahest	56.76%*	96.96%*	‡

minecraft_savedata

This data set is composed of Minecraft player saves that contain highly structured data.

Raw Data

For operations, time per iteration; for size, bytes. Lower is better.

Serialize / deserialize speed and size

Format / Lib	Serialize	Deserialize	Size	Zlib	Zstd
abomonation	201.38 µs	1.5505 ms*	1290592	391124	330250
bare	735.16 µs	2.4497 ms	356311	213270	198488
bincode	504.35 µs	2.1922 ms	368392	222323	207063
borsh	475.79 µs	2.0620 ms	446595	234395	210008
bson	3.6433 ms	10.493 ms	1619653	506953	328399
capnp	557.42 µs	†	803896	336655	280851
cbor	1.7022 ms	5.0966 ms	1109821	347812	274526
flatbuffers	2.9606 ms	†	844168	346957	294015
nachricht	5.0234 ms	4.1721 ms	449745	252743	231110
postcard	378.83 µs	2.1879 ms	367489	222144	207344
prost	2.9704 ms* 1.1190 ms*	3.6056 ms	596811	306728	269310
rkyv	334.54 µs	1.4825 ms* 2.0167 ms*	596952	254571	219976
rmp	1.5170 ms	3.1263 ms	424533	245594	226188
ron	7.8674 ms	14.958 ms	1465223	439761	343338
scale	556.45 µs	2.1867 ms	356311	213188	198524
serde_json	3.5431 ms	8.4326 ms	1623191	472275	359623
simd-json	3.5183 ms	5.3939 ms	1663769	496401	383682
speedy	318.64 µs	1.7411 ms	449595	235136	210361
alkahest	248.16 µs	†	667570	325536	320452

Zero-copy deserialization speed

Format / Lib	Access	Read	Update
abomonation	46.189 µs*	46.948 µs*	‡
capnp	69.088 ns*	489.05 ns*	‡
flatbuffers	2.5879 ns* 1.7906 ms*	2.2220 µs* 1.7898 ms*	‡
rkyv	1.2177 ns* 530.36 µs*	281.39 ns* 536.46 µs*	1.4894 µs
alkahest	2.1986 ns*	22.691 µs*	‡

Comparison

Relative to best. Higher is better.

Serialize / deserialize speed and size

Format / Lib	Serialize	Deserialize	Size	Zlib	Zstd
abomonation	100.00%	95.61%*	27.61%	54.51%	60.10%
bare	27.39%	60.52%	100.00%	99.96%	100.00%
bincode	39.93%	67.63%	96.72%	95.89%	95.86%
borsh	42.33%	71.90%	79.78%	90.95%	94.51%
bson	5.53%	14.13%	22.00%	42.05%	60.44%
capnp	36.13%	†	44.32%	63.33%	70.67%
cbor	11.83%	29.09%	32.11%	61.29%	72.30%
flatbuffers	6.80%	†	42.21%	61.45%	67.51%
nachricht	4.01%	35.53%	79.23%	84.35%	85.88%
postcard	53.16%	67.76%	96.96%	95.97%	95.73%
prost	6.78%* 18.00%*	41.12%	59.70%	69.50%	73.70%
rkyv	60.20%	100.00%* 73.51%*	59.69%	83.74%	90.23%
rmp	13.27%	47.42%	83.93%	86.81%	87.75%
ron	2.56%	9.91%	24.32%	48.48%	57.81%
scale	36.19%	67.80%	100.00%	100.00%	99.98%
serde_json	5.68%	17.58%	21.95%	45.14%	55.19%
simd-json	5.72%	27.48%	21.42%	42.95%	51.73%
speedy	63.20%	85.15%	79.25%	90.67%	94.36%
alkahest	81.15%	†	53.37%	65.49%	61.94%

Zero-copy deserialization speed

Format / Lib	Access	Read	Update
abomonation	0.00%*	0.60%*	‡
capnp	1.76%*	57.54%*	‡
flatbuffers	47.05%* 0.00%*	12.66%* 0.02%*	‡
rkyv	100.00%* 0.00%*	100.00%* 0.05%*	100.00%
alkahest	55.39%*	1.24%*	‡

Footnotes:

* mouse over for situational details

† do not provide deserialization capabilities, but the user can write their own

‡ do not support buffer mutation (capnp and flatbuffers may but not for rust)

[fia0]

Since the usage of serde seems to be for all implementations decremental to performance we likely need to implement large parts of the serialization anyway anew, even considering we will remain with bincode, therefore we have at the moment the optimal moment to reconsider our decision and choose a good serialization candidate for the future.

[fia0]

Considering the effort, with the 2.0 bincode patches (which begin to settle on a proper release atm) we should be able to relatively easy move from bincode to speedy and reverse. Have a look at https://docs.rs/bincode/2.0.0-rc.2/bincode/enc/trait.Encode.html and https://docs.rs/speedy/latest/speedy/trait.Writer.html. Atleast the structure could remain largely the same, so we could maybe even perform some comparison benchmarks.