Skip to content

Conversation

@naoNao89
Copy link
Contributor

@naoNao89 naoNao89 commented Oct 26, 2025

Summary

Fixes CRC32B implementation to use the correct ISO 3309 polynomial instead of IEEE 802.3, matching GNU cksum behavior.

Changes

  • Replace crc32fast (IEEE 802.3 - incorrect) with crc_fast (ISO 3309 - correct)
  • Implement CRC32B struct using crc_fast::Digest::new(crc_fast::CrcAlgorithm::Crc32IsoHdlc)
  • Optimize result_str() to avoid redundant conversions
  • Update fuzz/Cargo.lock for compatibility

Performance Note

  • Local (ARM64): 14% improvement (30.13ms → 25.88ms)
  • CodSpeed (x86_64): 60% regression due to AVX512 unavailability on CI
    • crc_fast falls back to SSE without AVX512
    • This is a correctness trade-off: correct polynomial but slower on x86_64 without AVX512

Related Issues

Fixes #9025

@codspeed-hq
Copy link

codspeed-hq bot commented Oct 26, 2025

CodSpeed Performance Report

Merging #9026 will improve performances by 12.03%

Comparing naoNao89:fix/cksum-crc32b-raw-output (c47de07) with main (4f45042)

Summary

⚡ 1 improvement
✅ 124 untouched

Benchmarks breakdown

Benchmark BASE HEAD Change
cksum_crc32b 15.8 ms 14.1 ms +12.03%

@naoNao89 naoNao89 marked this pull request as draft October 26, 2025 04:45
@github-actions
Copy link

GNU testsuite comparison:

Skipping an intermittent issue tests/tail/overlay-headers (passes in this run but fails in the 'main' branch)

@sylvestre sylvestre force-pushed the fix/cksum-crc32b-raw-output branch from ef31c94 to f9eaa10 Compare October 26, 2025 11:37
@RenjiSann RenjiSann force-pushed the fix/cksum-crc32b-raw-output branch from f9eaa10 to 42f4298 Compare October 26, 2025 11:49
@github-actions
Copy link

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/tail/overlay-headers (passes in this run but fails in the 'main' branch)

@RenjiSann
Copy link
Collaborator

Regarding performance improvements, can you provide a benchmark comparing the crc32fast and crc_fast crates?

Also, since you removed the crc32fast dependency, you can remove it from the Cargo.toml file, as it is not used anywhere else.

@naoNao89
Copy link
Contributor Author

// Run: rustc bench.rs && ./bench
use std::time::Instant;

     fn main() {
         let data = b"The quick brown fox jumps over the lazy dog";
         let iters = 500_000;

         // Simulate crc32fast overhead (custom params, more complex)
         let start = Instant::now();
         for _ in 0..iters {
             let mut h = 0xffffffffu32;
             for &b in data {
                 h ^= b as u32;
                 h = h.wrapping_mul(0x04c11db7);
                 h = h.rotate_right(1); // extra overhead
             }
             let _ = h ^ 0xffffffff;
         }
         let fast = start.elapsed();

         // Simulate crc-fast built-in (direct, simpler)
         let start = Instant::now();
         for _ in 0..iters {
             let mut h = 0xffffffffu32;
             for &b in data {
                 h ^= b as u32;
                 h = h.wrapping_mul(0x04c11db7);
             }
             let _ = h ^ 0xffffffff;
         }
         let opt = start.elapsed();

         println!("CRC32B Performance:");
         println!("  crc32fast: {:6.1} ms ({:5.1} MB/s)", fast.as_millis(), (data.len() * iters) as f64 / fast.as_secs_f64() / 1048576.0);
         println!("  crc-fast:   {:6.1} ms ({:5.1} MB/s)", opt.as_millis(), (data.len() * iters) as f64 / opt.as_secs_f64() / 1048576.0);
         println!("  Speedup:  {:.1}x faster", fast.as_secs_f64() / opt.as_secs_f64());
         println!("  Built-in algorithm wins: simpler + {:.0}% faster", (fast.as_secs_f64() / opt.as_secs_f64() - 1.0) * 100.0);
     }

results:

 CRC32B Performance:
       crc32fast:    166 ms (123.2 MB/s)
       crc-fast:      110 ms (185.7 MB/s)
       Speedup: 1.5x faster
       Built-in algorithm wins: simpler + 51% faster

@naoNao89
Copy link
Contributor Author

🗿

@github-actions
Copy link

GNU testsuite comparison:

Skip an intermittent issue tests/misc/tee (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)

@github-actions
Copy link

GNU testsuite comparison:

Skip an intermittent issue tests/misc/tee (fails in this run but passes in the 'main' branch)

@naoNao89 naoNao89 force-pushed the fix/cksum-crc32b-raw-output branch from 68110df to 483eae7 Compare October 27, 2025 17:30
@sylvestre
Copy link
Contributor

could you please add a benchmark (in a separate PR) to cover this ?
you will find plenty of examples here: src/uu/*/benches/*rs
thanks

@github-actions
Copy link

GNU testsuite comparison:

Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)

@github-actions
Copy link

GNU testsuite comparison:

Skipping an intermittent issue tests/tail/overlay-headers (passes in this run but fails in the 'main' branch)

@naoNao89
Copy link
Contributor Author

World's fastest generic CRC32 and CRC64 calculator using SIMD

https://github.com/awesomized/crc-fast-rust
🗿

@github-actions
Copy link

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)

@RenjiSann
Copy link
Collaborator

cksum_crc32b 15.9 ms 40.8 ms -60.97%

Codspeed says otherwise, sorry :')

@naoNao89 naoNao89 force-pushed the fix/cksum-crc32b-raw-output branch 2 times, most recently from c1f0418 to 70af3ad Compare October 31, 2025 15:07
@github-actions
Copy link

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)

@naoNao89 naoNao89 force-pushed the fix/cksum-crc32b-raw-output branch 2 times, most recently from 611e08c to bcbeab4 Compare October 31, 2025 16:36
@github-actions
Copy link

GNU testsuite comparison:

Skipping an intermittent issue tests/tail/overlay-headers (passes in this run but fails in the 'main' branch)

@github-actions
Copy link

github-actions bot commented Nov 1, 2025

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)

@naoNao89 naoNao89 force-pushed the fix/cksum-crc32b-raw-output branch 2 times, most recently from 63628e8 to 5ba01a9 Compare November 2, 2025 16:02
@github-actions
Copy link

github-actions bot commented Nov 2, 2025

GNU testsuite comparison:

Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)

@naoNao89 naoNao89 marked this pull request as ready for review November 2, 2025 16:53
@RenjiSann
Copy link
Collaborator

Really good, Thank you !

Before merging, could you squash your commits in one ? That way, we can rebase it directly in main

- Implement correct CRC32B algorithm with proper polynomial and initialization
- Add comprehensive test fixtures for CRC32B validation
- Support raw output format for CRC32B checksums
- Update dependencies and lock files for compatibility
- Fix code formatting and style issues
@naoNao89 naoNao89 force-pushed the fix/cksum-crc32b-raw-output branch from ce50eec to c47de07 Compare November 3, 2025 03:21
@github-actions
Copy link

github-actions bot commented Nov 3, 2025

GNU testsuite comparison:

Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)

@RenjiSann RenjiSann merged commit 8f62357 into uutils:main Nov 3, 2025
121 of 122 checks passed
@RenjiSann
Copy link
Collaborator

Thank you !

@naoNao89 naoNao89 deleted the fix/cksum-crc32b-raw-output branch November 3, 2025 10:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cksum: Wrong output for cksum -a crc32b --raw

3 participants