Document "hash derivation quotiented", resolution, and build trace

Progress on #13405, which asks for an explicit characterisation of the
equivalence relation like the one given here.

Also progress on #11895, because we're using the term "build trace
entry" instead of "realisation".

Mention #9259, a future work item.

Co-authored-by: Robert Hensing <[email protected]>

Author: Ericson2314

doc/manual/book.toml.in

@@ -7,6 +7,7 @@ additional-css = ["custom.css"]
 additional-js = ["redirects.js"]
 edit-url-template = "https://github.com/NixOS/nix/tree/master/doc/manual/{path}"
 git-repository-url = "https://github.com/NixOS/nix"
+mathjax-support = true
 
 # Handles replacing @docroot@ with a path to ./source relative to that markdown file,
 # {{#include handlebars}}, and the @generated@ syntax used within these. it mostly

doc/manual/meson.build

@@ -92,6 +92,8 @@ manual = custom_target(
         (cd @2@; RUST_LOG=warn @1@ build -d @2@ 3>&2 2>&1 1>&3) | { grep -Fv "because fragment resolution isn't implemented" || :; } 3>&2 2>&1 1>&3
         rm -rf @2@/manual
         mv @2@/html @2@/manual
+        # Remove Mathjax 2.7, because we will actually use MathJax 3.x
+        find @2@/manual | grep .html | xargs sed -i -e '/2.7.1.MathJax.js/d'
         find @2@/manual -iname meson.build -delete
     '''.format(
       python.full_path(),

doc/manual/source/SUMMARY.md.in

@@ -26,9 +26,12 @@
     - [Derivation Outputs and Types of Derivations](store/derivation/outputs/index.md)
       - [Content-addressing derivation outputs](store/derivation/outputs/content-address.md)
       - [Input-addressing derivation outputs](store/derivation/outputs/input-address.md)
+  - [Build Trace](store/build-trace.md)
+  - [Derivation Resolution](store/resolution.md)
   - [Building](store/building.md)
   - [Store Types](store/types/index.md)
 {{#include ./store/types/SUMMARY.md}}
+  - [Appendix: Math notation](store/math-notation.md)
 - [Nix Language](language/index.md)
   - [Data Types](language/types.md)
     - [String context](language/string-context.md)

doc/manual/source/protocols/derivation-aterm.md

@@ -1,6 +1,8 @@
 # Derivation "ATerm" file format
 
-For historical reasons, [store derivations][store derivation] are stored on-disk in [ATerm](https://homepages.cwi.nl/~daybuild/daily-books/technology/aterm-guide/aterm-guide.html) format.
+For historical reasons, [store derivations][store derivation] are stored on-disk in "Annotated Term" (ATerm) format
+([guide](https://homepages.cwi.nl/~daybuild/daily-books/technology/aterm-guide/aterm-guide.html),
+[paper](https://doi.org/10.1002/(SICI)1097-024X(200003)30:3%3C259::AID-SPE298%3E3.0.CO;2-Y)).
 
 ## The ATerm format used
 

doc/manual/source/protocols/json/schema/derivation-v3.yaml

@@ -39,9 +39,9 @@ properties:
       This is a guard that allows us to continue evolving this format.
       The choice of `3` is fairly arbitrary, but corresponds to this informal version:
 
-      - Version 0: A-Term format
+      - Version 0: ATerm format
 
-      - Version 1: Original JSON format, with ugly `"r:sha256"` inherited from A-Term format.
+      - Version 1: Original JSON format, with ugly `"r:sha256"` inherited from ATerm format.
 
       - Version 2: Separate `method` and `hashAlgo` fields in output specs
 

doc/manual/source/store/build-trace.md

@@ -0,0 +1,53 @@
+# Build Trace
+
+> **Warning**
+>
+> This entire concept is currently
+> [**experimental**](@docroot@/development/experimental-features.md#xp-feature-ca-derivations)
+> and subject to change.
+
+The *build trace* is a [memoization table](https://en.wikipedia.org/wiki/Memoization) for builds.
+It maps the the inputs of builds to the outputs of builds.
+Concretely, that means it maps [derivations][derivation] to maps of [output] names to [store objects][store object].
+
+In general the derivations used as a key should be be [*resolved*](./resolution.md).
+A build trace with all-resolved-derivation keys is also called a *base build trace* for extra clarity.
+If all the resolved inputs of a derivation are content-addressed, that means the inputs will be fully determined, leaving no ambiguity for what build was performed.
+(Input-addressed inputs however are still ambiguous. They too should be locked down, but this is left as future work.)
+
+Accordingly, to look up an unresolved derivation, one must first resolve it to get a resolved derivation.
+Resolving itself involves looking up entries in the build trace, so this is a mutually recursive process that will end up inspecting possibly many entries.
+
+Except for the issue with input-addressed paths called out above, base build traces are trivially *coherent* -- incoherence is not possible.
+That means that the claims that each key-value base build try entry makes are independent, and no mapping invalidates another mapping.
+
+Whether the mappings are *true*, i.e. the faithful recording of actual builds performed, is another matter.
+Coherence is about the multiple claims of the build trace being mutually consistent, not about whether the claims are individually true or false.
+
+In general, there is no way to audit a build trace entry except for by performing the build again from scratch.
+And even in that case, a different result doesn't mean the original entry was a "lie", because the derivation being built may be non-deterministic.
+As such, the decision of whether to trust a counterparty's build trace is a fundamentally subject policy choice.
+Build trace entries are typically *signed* in order to enable arbitrary public-key-based trust polices.
+
+## Derived build traces
+
+Implementations that wish to memoize the above may also keep additional *derived* build trace entries that do map unresolved derivations.
+But if they do so, they *must* also keep the underlying base entries with resolved derivation keys around.
+Firstly, this ensures that the derived entries are merely cache, which could be recomputed from scratch.
+Secondly, this ensures the coherence of the derived build trace.
+
+Unlike with base build traces, incoherence with derived build traces is possible.
+The key ingredient is that derivation resolution is only deterministic with respect to a fixed base build trace.
+Without fixing the base build trace, it inherits the subjectivity of base build traces themselves.
+
+Concretely, suppose there are three derivations \\(a\\), \\(b\\), and \((c\\).
+Let \\(a\\) be a resolved derivation, but let \\(b\\) and \((c\\) be unresolved and both take as an input an output of \\(a\\).
+Now suppose that derived entries are made for \\(b\\) and \((c\\) based on two different entries of \\(a\\).
+(This could happen if \\(a\\) is non-deterministic, \\(a\\) and \\(b\\) are built in one store, \\(a\\) and \\(c\\) are built in another store, and then a third store substitutes from both of the first two stores.)
+
+If trusting the derived build trace entries for \\(b\\) and \((c\\) requires that each's underlying entry for \\(a\\) be also trusted, the two different mappings for \\(a\\) will be caught.
+However, if \\(b\\) and \((c\\)'s entries can be combined in isolation, there will be nothing to catch the contradiction in their hidden assumptions about \\(a\\)'s output.
+
+[derivation]: ./derivation/index.md
+[output]: ./derivation/outputs/index.md
+[store object]: @docroot@/store/store-object.md

doc/manual/source/store/derivation/index.md

@@ -245,7 +245,7 @@ If those other derivations *also* abide by this common case (and likewise for tr
   >                                                           note the ".drv"
   > ```
 
-## Extending the model to be higher-order
+## Extending the model to be higher-order {#dynamic}
 
 **Experimental feature**: [`dynamic-derivations`](@docroot@/development/experimental-features.md#xp-feature-dynamic-derivations)
 

doc/manual/source/store/derivation/outputs/content-address.md

@@ -167,10 +167,10 @@ It is only in the potential for that check to fail that they are different.
 >
 > In a future world where floating content-addressing is also stable, we in principle no longer need separate [fixed](#fixed) content-addressing.
 > Instead, we could always use floating content-addressing, and separately assert the precise value content address of a given store object to be used as an input (of another derivation).
-> A stand-alone assertion object of this sort is not yet implemented, but its possible creation is tracked in [Issue #11955](https://github.com/NixOS/nix/issues/11955).
+> A stand-alone assertion object of this sort is not yet implemented, but its possible creation is tracked in [issue #11955](https://github.com/NixOS/nix/issues/11955).
 >
 > In the current version of Nix, fixed outputs which fail their hash check are still registered as valid store objects, just not registered as outputs of the derivation which produced them.
-> This is an optimization that means if the wrong output hash is specified in a derivation, and then the derivation is recreated with the right output hash, derivation does not need to be rebuilt --- avoiding downloading potentially large amounts of data twice.
+> This is an optimization that means if the wrong output hash is specified in a derivation, and then the derivation is recreated with the right output hash, derivation does not need to be rebuilt — avoiding downloading potentially large amounts of data twice.
 > This optimisation prefigures the design above:
 > If the output hash assertion was removed outside the derivation itself, Nix could additionally not only register that outputted store object like today, but could also make note that derivation did in fact successfully download some data.
 For example, for the "fetch URL" example above, making such a note is tantamount to recording what data is available at the time of download at the given URL.