Set up garbage collection

[fricklerhandwerk]

In order to limit the system's ever-growing resource consumption:

• Policy: Define a cut-off date and freeze issues that are associated with data older than that

  • While we can retain everything, this will simply produce cognitive load for volunteers who already have a well-established policy of not touching stale channels.
  • If needed and provided with resources, it's still possible to retrofit a feature for back-filling old channels.

• Mechanism: Remove all related objects from the database older than the cut-off date

  There is an edge case that needs to be guarded against:

  • There may be packages belonging to a published issue that only live in a stale channel
  • In this case we should set a tombstone for that package which states the stale channel
  • In the UI this would display as an explanation or link to an explanation
  • Possibly we'd want to also close the associated GitHub issue

• Remove derivations from /nix/store related to outdated objects

  see feat: make eval store configurable #451 (comment) for details

[Erethon]

This is somewhat relevant. I'm runinng a nix store --gc in the production sectracker instance and it's been running for more than an hour now. Chasing this around a bit lead me to NixOS/nix#9581 and in particular to this comment, as I'm seeing a similar behaviour (single CPU bound to 100% usage, no IO wait after some time (the data on prometheus will be deleted in a couple of weeks, adding screenshots at the end of this comment to properly log this)).

It might be interesting to test the patches mentioned in NixOS/nix#9581 (comment).

[Erethon]

Linking this here to leave a paper trail on GH issues besides just documenting it in our repo.

While trying to optimize the nix store we ran into this issue. Enabling large_dir on the filesystem side resolved this. For the record, we have 6.0622e+06 paths to be optimized.

[Erethon]

Ever since we started running evals again (25/7/2025), the database size has kept increasing as one would expect. However, it feels like it's increasing too fast. We've gone from 15.4 million to 22 million derivations now and the database has grown from ~80GB to ~118GB. I'm experimenting with #606 (the small drop in the graph) to reclaim some space, but I'm trying to figure out why we're seeing such a big increase.

The two biggest tables are shared_nixderivationoutput (which contains 334761202 derivations and is 41GB currently!) and shared_nixderivation_dependencies which again is 41GB.

[Erethon]

Another thing that keeps escalating in terms of disk space is the nix database. Before and after running a manual vacuum on it:

[nix-shell:~/nix-security-tracker]# ls -laht /nix/var/nix/db/db.sqlite
-rw-r--r-- 1 root root 9.8G Aug 16 23:19 /nix/var/nix/db/db.sqlite

[nix-shell:~/nix-security-tracker]# sqlite3 /nix/var/nix/db/db.sqlite VACUUM

[nix-shell:~/nix-security-tracker]# ls -laht /nix/var/nix/db/db.sqlite
-rw-r--r-- 1 root root 3.7G Aug 16 23:26 /nix/var/nix/db/db.sqlite