2B for Vec size instead of 1B as in bincode

[mentalblood0]

    #[derive(bitcode::Encode, bitcode::Decode, bincode::Encode, Debug)]
    struct TestStruct {
        key: Vec<u8>,
        value: Vec<u8>,
    }

    #[test]
    fn test_encoding() {
        let input = TestStruct {
            key: "key".as_bytes().to_vec(),
            value: "value".as_bytes().to_vec(),
        };
        {
            let encoded = bitcode::encode(&input);
            assert_eq!(encoded.len(), 12);
        }
        {
            let encoded = bincode::encode_to_vec(&input, bincode::config::standard()).unwrap();
            assert_eq!(encoded.len(), 10);
        }
    }

[finnbear]

Can you please elaborate?

[mentalblood0]

Where result got using bincode contains one-byte encoded lengths (0x03 and 0x05 in above-given example), result got using bitcode have 0x0300 and 0x0500 correspondingly. It is unexpected as it is claimed to be compact and bitwise

[finnbear]

Ok, thanks! Note that bitcode optimizations are focused on larger messages. Simply by encoding random instances of your struct multiple times, bitcode wins.

#[derive(Clone, bitcode::Encode, bitcode::Decode, bincode::Encode, Debug)]
struct TestStruct {
    key: Vec<u8>,
    value: Vec<u8>,
}

fn main() {
    use rand::SeedableRng;
    let mut rng = rand::rngs::StdRng::from_seed([0; 32]);
    let mut random_bytes = |len: std::ops::Range<usize>| -> Vec<u8> {
        use rand::Rng;
        let len = rng.random_range(len);
        std::iter::repeat_with(|| rng.random::<u8>())
            .take(len)
            .collect()
    };

    for n in 1..=100 {
        let inputs = std::iter::repeat_with(|| TestStruct {
            key: random_bytes(2..5),
            value: random_bytes(4..7),
        })
        .take(n)
        .collect::<Vec<_>>();
        let bitcode = {
            let encoded = bitcode::encode(&inputs);
            encoded.len()
        };
        let bincode = {
            let encoded = bincode::encode_to_vec(&inputs, bincode::config::standard()).unwrap();
            encoded.len()
        };

        println!(
            "with {n} TestStruct, bitcode saves {}B",
            (bincode as isize) - (bitcode as isize)
        )
    }
}

with 1 TestStruct, bitcode saves -2B
with 2 TestStruct, bitcode saves -2B
with 3 TestStruct, bitcode saves -1B
with 4 TestStruct, bitcode saves 1B
with 5 TestStruct, bitcode saves 1B
with 6 TestStruct, bitcode saves 2B
with 7 TestStruct, bitcode saves 4B
with 8 TestStruct, bitcode saves 6B
with 9 TestStruct, bitcode saves 8B
with 10 TestStruct, bitcode saves 10B
...lines omitted...
with 17 TestStruct, bitcode saves 20B
...lines omitted...
with 67 TestStruct, bitcode saves 100B
...lines omitted...
with 98 TestStruct, bitcode saves 150B
with 99 TestStruct, bitcode saves 152B
with 100 TestStruct, bitcode saves 154B

Graph of bytes saved vs. number of TestStruct:

See also: https://github.com/SoftbearStudios/bitcode/wiki/Optimization-principles#overhead

I'll let @caibear comments on whether it makes sense to optimize for your original case, given the relevant trade-offs that it would involve.

Finally, note that bitcode@0.5 may conform better to your expectations. It treats "bitwise" as a strict rule.

[caibear]

The 2nd byte is not part of the length, it represents the type of integer compression (called packing in bitcode's codebase) applied to the u8 in the Vec<u8>. For example a vec![0u8, 1, 1, 0, 1, 0, 1, 1] would take 3 bytes [8 (len), ? (some byte that represents 1 bit integers), 0b01101011 (the bit packed u8s)]. When this compression isn't useful it adds 1 byte of overhead to each unique integer field encoded by bitcode, which is only significant for extremely small messages.