benchmarking v0.6

[bluegenes]

Built gtdb-rs226 from scratch (732,475 genomes) to get a large microbial database benchmark

at 30 simultaneous downloads, 30 cores, with a batch size of 100k:

• 5.35G RAM
• 11 hours 14 minutes
• % cpu time: 1069%

Failed Accessions:

• 441 failed due to missing download links (could not get fetch link from NCBI),
• 0 download failures
• 0 checksum (crc32) failures.

I spot checked 10 random failed accessions and all were suppressed.

v0.6.2

Command:

sourmash scripts gbsketch gtdb-rs226.gbsketch.csv -o gtdb-rs226.sig.zip -n 30 -r 15 -g -c 30 \
                          --batch-size 100_000 -p dna,k=21,k=31,k=51,scaled=1000 \
                          --write-urlsketch-csv --verbose 2> gtdb-rs226.build-genomic.log

I provided my api key via the $NCBI_API_KEY variable.

Tool output:

== This is sourmash version 4.8.14. ==
== Please cite Irber et. al (2024), doi:10.21105/joss.06830. ==

=> sourmash_plugin_directsketch 0.6.2
params: ['dna,k=21,k=31,k=51,scaled=1000']
Batch size is set, enabling --no-overwrite-fasta by default.
Downloading and sketching all accessions in 'gtdb-rs226.gbsketch.csv using 30 simultaneous downloads, 15 retries, and 30 threads.
No valid existing signature batches found; building all signatures.
No protein signature templates provided, and --keep-fasta is not set.
Downloading and sketching genomes only.
Successfully downloaded and parsed dehydrated zipfile. Now processing accessions.
Skipped 441 download(s) due to missing download URLs.
Wrote urlsketch csv with download links to 'gtdb-rs226.gbsketch.csv.urlsketch.csv'
finished batch 1: wrote to 'gtdb-rs226.1.sig.zip'
finished batch 2: wrote to 'gtdb-rs226.2.sig.zip'
finished batch 3: wrote to 'gtdb-rs226.3.sig.zip'
finished batch 4: wrote to 'gtdb-rs226.4.sig.zip'
finished batch 5: wrote to 'gtdb-rs226.5.sig.zip'
finished batch 6: wrote to 'gtdb-rs226.6.sig.zip'
finished batch 7: wrote to 'gtdb-rs226.7.sig.zip'
finished batch 8: wrote to 'gtdb-rs226.8.sig.zip'
Wrote list of all batches to 'gtdb-rs226.sig.zip.batches.txt'
...gbsketch is done!
Sigs in 'gtdb-rs226.1.sig.zip', etc

Output from /usr/bin/time -v:

User time (seconds): 415849.58
        System time (seconds): 16567.65
        Percent of CPU this job got: 1069%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 11:13:56
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 5353936
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 18
        Minor (reclaiming a frame) page faults: 61622149
        Voluntary context switches: 652535474
        Involuntary context switches: 58011180
        Swaps: 0
        File system inputs: 84496
        File system outputs: 129107288
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

[ctb]

nice work!

[bluegenes]

repo used to build: https://github.com/bluegenes/2025-gtdb-rs226

[bluegenes]

~1.46 million bacterial genomes from GenBank (to update GB db): 17 hours 15min, 10GB RAM

== This is sourmash version 4.8.14. ==
== Please cite Irber et. al (2024), doi:10.21105/joss.06830. ==

=> sourmash_plugin_directsketch 0.6.3
params: ['dna,k=21,k=31,k=51,scaled=1000']
Downloading and sketching all accessions in 'missing-genomes.20250408-bacteria.csv using 30 simultaneous downloads, 15 retries, and 30 threads.
No valid existing signature batches found; building all signatures.
No protein signature templates provided, and --keep-fasta is not set.
Downloading and sketching genomes only.
WARNING: extra column 'ftp_path' in CSV file. Ignoring.
Successfully downloaded and parsed dehydrated zipfile. Now processing accessions.
Skipped 1315 download(s) due to missing download URLs.
Wrote urlsketch csv with download links to 'missing-genomes.20250408-bacteria.csv.urlsketch.csv'
finished batch 1: wrote to 'genbank.20250408.missing.1.sig.zip'
finished batch 2: wrote to 'genbank.20250408.missing.2.sig.zip'
finished batch 3: wrote to 'genbank.20250408.missing.3.sig.zip'
finished batch 4: wrote to 'genbank.20250408.missing.4.sig.zip'
finished batch 5: wrote to 'genbank.20250408.missing.5.sig.zip'
finished batch 6: wrote to 'genbank.20250408.missing.6.sig.zip'
finished batch 7: wrote to 'genbank.20250408.missing.7.sig.zip'
finished batch 8: wrote to 'genbank.20250408.missing.8.sig.zip'
finished batch 9: wrote to 'genbank.20250408.missing.9.sig.zip'
finished batch 10: wrote to 'genbank.20250408.missing.10.sig.zip'
finished batch 11: wrote to 'genbank.20250408.missing.11.sig.zip'
Error while processing accession 'GCA_000321065.1' - moltype 'DNA': HTTP error 404 Not Found from https://api.ncbi.nlm.nih.gov/datasets/fetch_h/R2V0UmVtb3RlRGF0YWZpbGU/eNqTyuXKzktOytROKymw0tdPT83Lz00t1k_MydF3d3bUNzAw0Dc2MtQ3MDMF8eOBfCAXyNMzjHcM9gWzywzxSMWDTcxM1kvLS9RLrzJgtGAEADGnHxk
finished batch 12: wrote to 'genbank.20250408.missing.12.sig.zip'
finished batch 13: wrote to 'genbank.20250408.missing.13.sig.zip'
Error while processing accession 'GCA_030213925.1' - moltype 'DNA': HTTP error 404 Not Found from https://api.ncbi.nlm.nih.gov/datasets/fetch_h/R2V0UmVtb3RlRGF0YWZpbGU/eNqTKuTKz0tOytROKymw0tdPT83Lz00t1k_MydF3d3bUNzA20DcyNNa3NDIF8eOBfCAXyNMzjHcM9oXyygzxSsaDTc1M1kvLS9RLrzJgtGAEAK3hIAk
Error while processing accession 'GCA_030226095.1' - moltype 'DNA': HTTP error 404 Not Found from https://api.ncbi.nlm.nih.gov/datasets/fetch_h/R2V0UmVtb3RlRGF0YWZpbGU/eNqTKuTKz0tOytROKymw0tdPT83Lz00t1k_MydF3d3bUNzA20DcyMtM3sDQF8eOBfCAXyNMzjHcM9oXyygzxSsaDTc1M1kvLS9RLrzJgtGAEAK_fIBM
finished batch 14: wrote to 'genbank.20250408.missing.14.sig.zip'
Error while processing accession 'GCA_905332505.2' - moltype 'DNA': HTTP error 404 Not Found from https://api.ncbi.nlm.nih.gov/datasets/fetch_h/R2V0UmVtb3RlRGF0YWZpbGU/eNqTWsLItZAxLzkpUzutpMBKXz89NS8_N7VYPzEnR9_d2VHf0sBU39jYSN8USAP58UA-kAvk6RnFuwHV5uQkFmUmxhcX5BflpyUWl2SmZOYXlxbHu_oGeTg7xxuZkKcrHuyOzGS9tLxEvfQqA0YLRgDjiDbV
finished batch 15: wrote to 'genbank.20250408.missing.15.sig.zip'
Wrote list of all batches to 'genbank.20250408.missing.sig.zip.batchlist.txt'
...gbsketch is done!

Command being timed: "sourmash scripts gbsketch missing-genomes.20250408-bacteria.csv -o genbank.20250408.missing.sig.zip -n 30 -r 15 -g -c 30 --batch-size 100_000 -p dna,k=21,k=31,k=51,scaled=1000 --write-urlsketch-csv --verbose"
        User time (seconds): 913583.70
        System time (seconds): 29306.74
        Percent of CPU this job got: 1517%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 17:15:36
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 9780288
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 65271403
        Voluntary context switches: 1194313135
        Involuntary context switches: 65485300
        Swaps: 0
        File system inputs: 0
        File system outputs: 264812824
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0