Merge branch 'master' into fix/prove/request-parents-once

Author: schomatis

.circleci/config.yml

@@ -78,7 +78,7 @@ jobs:
           name: Test (stable) in release profile
           command: |
             cargo +stable test --verbose --release --frozen --all
-            cargo +stable build --examples --release --frozen --all
+            RUSTFLAGS="-D warnings" cargo +stable build --examples --release --frozen --all
 
   ffi_regression:
     docker:

CONTRIBUTING.md

@@ -0,0 +1,47 @@
+# Contributing
+
+Welcome, it is great that you found your way here. In order to make the best of all our time, we have gathered some notes 
+below which we think can be helpful when contributing to this project.
+
+## Getting Started
+
+Please start by reviewing this file.
+
+## Coding Standards
+
+- No compiler warnings.
+- No [clippy](https://github.com/rust-lang/rust-clippy) warnings.
+- Minimize use of `unsafe` and justify usage in comments.
+- Prefer `expect` with a good description to `unwrap`.
+- Write unit tests in the same file.
+- Format your code with `rustfmt`
+- Code should compile on `stable` and `nightly`. If adding `nightly` only features they should be behind a flag.
+- Write benchmarks for performance sensitive areas. We use [criterion.rs](https://github.com/japaric/criterion.rs).
+
+
+## General Guidelines
+- PRs require code owner approval to merge.
+- Please scope PRs to areas in which you have expertise. This code is still close to research.
+- Welcome contribution areas might include:
+  - SNARKs
+  - Proof-of-replication
+  - Rust improvements
+  - Optimizations
+  - Documentation (expertise would require careful reading of the code)
+
+## Resources for learning Rust
+
+- Beginners 
+  - [The Rust Book](https://doc.rust-lang.org/book/)
+  - [Rust Playground](https://play.rust-lang.org/)
+  - [Rust Docs](https://doc.rust-lang.org/)
+  - [Clippy](https://github.com/rust-lang/rust-clippy)
+  - [Rustfmt](https://github.com/rust-lang/rustfmt)
+- Advanced
+  - What does the Rust compiler do with my code? [Godbolt compiler explorer](https://rust.godbolt.org/)
+  - How to safely write unsafe Rust: [The Rustonomicon](https://doc.rust-lang.org/nomicon/)
+  - Did someone say macros? [The Little Book of Rust Macros](https://danielkeep.github.io/tlborm/book/index.html)
+
+## Licensing
+
+As mentioned in the [readme](README.md) all contributions are dual licensed under Apache 2 and MIT. 

README.md

@@ -1,6 +1,6 @@
 # Filecoin Proving Subsystem (FPS)
 
-The **Filecoin Proving Subsystem** provides the storage proofs required by the Filecoin protocol.  It is implemented entirely in Rust, as a series of partially inter-dependent crates – some of which export C bindings to the supported API. This decomposition into distinct crates/modules is relatively recent, and in some cases current code has not been fully refactored to reflect the intended eventual organization.
+The **Filecoin Proving Subsystem** provides the storage proofs required by the Filecoin protocol. It is implemented entirely in Rust, as a series of partially inter-dependent crates – some of which export C bindings to the supported API. This decomposition into distinct crates/modules is relatively recent, and in some cases current code has not been fully refactored to reflect the intended eventual organization.
 
 There are currently four different crates:
 
@@ -23,23 +23,23 @@ There are currently four different crates:
   A sector database abstracting away underlying storage considerations. This abstraction will allow for alternate implementations mapping logical sectors to physical storage – facilitating both support for miner specialization, and configurable adaptation to a given miner’s physical hardware and preferences.
 
 - [**Storage Backend (`storage-backend`)**](./storage-backend)
-  The `storage-backend` crate is intended to contain abstractions and implementations of non-filecoin-specific storage mechanisms require by `storage-proofs`. However, for the sake of simplicity, it is currently an empty placeholder.
+  The `storage-backend` crate is intended to contain abstractions and implementations of non-Filecoin-specific storage mechanisms require by `storage-proofs`. However, for the sake of simplicity, it is currently an empty placeholder.
 
     ![FPS crate dependencies](/img/fps-dependencies.png?raw=true)
 
 ## Design Notes
 
-Earlier in the design process, we considered implementing what has become the **FPS** in Go – as a wrapper around potentially multiple SNARK circuit libraries. We eventually decided to use [bellman](https://github.com/zkcrypto/bellman) – a library developed by ZCash, which supports efficient pedersen hashing inside of SNARKs. Having made that decision, it was natural and efficient to implement the entire subsystem in Rust. We considered the benefits (self-contained code base, ability to rely on static typing across layers) and costs (developer ramp-up, sometimes unwieldiness of borrow-checker) as part of that larger decision and determined that the overall project benefits (in particular ability to build on Zcash’s work) outweighed the costs.
+Earlier in the design process, we considered implementing what has become the **FPS** in Go – as a wrapper around potentially multiple SNARK circuit libraries. We eventually decided to use [bellman](https://github.com/zkcrypto/bellman) – a library developed by Zcash, which supports efficient pedersen hashing inside of SNARKs. Having made that decision, it was natural and efficient to implement the entire subsystem in Rust. We considered the benefits (self-contained codebase, ability to rely on static typing across layers) and costs (developer ramp-up, sometimes unwieldiness of borrow-checker) as part of that larger decision and determined that the overall project benefits (in particular ability to build on Zcash’s work) outweighed the costs.
 
-We also considered whether the **FPS** should be implemented as a standalone binary accessed from [**`go-filecoin`**](https://github.com/filecoin-project/go-filecoin) either as a single-invocation CLI or as a long-running daemon process. We chose to bundle the **FPS** as an FFI dependency for both the simplicity of having a Filecoin node be deliverable as a single monolithic binary, and for the (perceived) relative development simplicity of the API implementation.
+We also considered whether the **FPS** should be implemented as a standalone binary accessed from [**`go-filecoin`**](https://github.com/filecoin-project/go-filecoin) either as a single-invocation CLI or as a long-running daemon process. Bundling the **FPS** as an FFI dependency was chosen for both the simplicity of having a Filecoin node deliverable as a single monolithic binary, and for the (perceived) relative development simplicity of the API implementation.
 
-If at any point it were to become clear that the FFI approach is irredeemably problematic, the option of moving to a standalone **FPS** remains. However, we have now already solved the majority of technical problems associated with calling from Go into Rust even while allowing for a high degree of runtime configurability. Therefore, it seems most likely that we continue down the path in which we have already invested and begin to reap reward.
+If at any point it were to become clear that the FFI approach is irredeemably problematic, the option of moving to a standalone **FPS** remains. However, the majority of technical problems associated with calling from Go into Rust are now solved, even while allowing for a high degree of runtime configurability. Therefore, continuing down the same path we have already invested in, and have begun to reap rewards from, seems likely.
 
 ## Install and configure Rust
 
-**NOTE:** If you have installed `rust-proofs` incidentally, as a submodule of `go-filecoin`, then you may already have installed Rust.
+**NOTE:** If you have installed `rust-fil-proofs` incidentally, as a submodule of `go-filecoin`, then you may already have installed Rust.
 
-The instructions below assume you have independently installed `rust-proofs` in order to test, develop, or experiment with it.
+The instructions below assume you have independently installed `rust-fil-proofs` in order to test, develop, or experiment with it.
 
 [Install Rust.](https://www.rust-lang.org/en-US/install.html)
 
@@ -93,13 +93,13 @@ To benchmark the examples you can [bencher](src/bin/bencher.rs).
 
 The results are written into the `.bencher` directory, as JSON files. The benchmarks are controlled through the [bench.config.toml](bench.config.toml) file.
 
-Note: on macOS you need `gtime` (`brew install gnu-time`), as the built in `time` command is not enough.
+Note: On macOS you need `gtime` (`brew install gnu-time`), as the built in `time` command is not enough.
 
 ## Logging
 
 For better logging with backtraces on errors, developers should use `expects` rather than `expect` on `Result<T, E>` and `Option<T>`.
 
-Developers can control `rust-proofs` logging through environment variables:
+Developers can control `rust-fil-proofs` logging through environment variables:
 
 -
   `RUST_PROOFS_LOG_JSON`
@@ -138,7 +138,7 @@ can run the following command:
 RUSTFLAGS="-Z sanitizer=leak" cargo run --release --package filecoin-proofs --example ffi --target x86_64-unknown-linux-gnu
 ```
 
-If using OSX, you'll have to run the leak detection from within a Docker
+If using mac OS, you'll have to run the leak detection from within a Docker
 container. After installing Docker, run the following commands to build and run
 the proper Docker image and then the leak detector itself:
 
@@ -156,25 +156,25 @@ docker build -t foo -f ./Dockerfile-ci . && \
 
 ## Generate Documentation
 
-First, navigate to the `rust-proofs` directory.
-- If you installed `rust-proofs` automatically as a submodule of `go-filecoin`:
+First, navigate to the `rust-fil-proofs` directory.
+- If you installed `rust-fil-proofs` automatically as a submodule of `go-filecoin`:
 ```
-> cd <go-filecoin-install-path>/go-filecoin/proofs/rust-proofs
+> cd <go-filecoin-install-path>/go-filecoin/proofs/rust-fil-proofs
 ```
 
-- If you cloned `rust-proofs` manually, it will be wherever you cloned it:
+- If you cloned `rust-fil-proofs` manually, it will be wherever you cloned it:
 ```
-> cd <install-path>/rust-proofs
+> cd <install-path>/rust-fil-proofs
 ```
 
-[Note that the version of `rust-proofs` included in `go-filecoin` as a submodule is not always the current head of `rust-proofs/master`. For documentation corresponding to the latest source, you should clone `rust-proofs` yourself.]
+[Note that the version of `rust-fil-proofs` included in `go-filecoin` as a submodule is not always the current head of `rust-fil-proofs/master`. For documentation corresponding to the latest source, you should clone `rust-fil-proofs` yourself.]
 
 Now, generate the documentation:
 ```
 > cargo doc --all --no-deps
 ```
 
-View the docs by pointing your browser at: `…/rust-proofs/target/doc/proofs/index.html`.
+View the docs by pointing your browser at: `…/rust-fil-proofs/target/doc/proofs/index.html`.
 
 ---
 
@@ -184,20 +184,24 @@ The **FPS** is accessed from [**go-filecoin**](https://github.com/filecoin-proje
 
  The Rust source code serves as the source of truth defining the **FPS** APIs. View the source directly:
 
-- [**filecoin-proofs**](https://github.com/filecoin-project/rust-proofs/blob/master/filecoin-proofs/src/api/mod.rs)
-- [**sector-base**](https://github.com/filecoin-project/rust-proofs/blob/master/sector-base/README.md#api-reference).
+- [**filecoin-proofs**](https://github.com/filecoin-project/rust-fil-proofs/blob/master/filecoin-proofs/src/api/mod.rs)
+- [**sector-base**](https://github.com/filecoin-project/rust-fil-proofs/blob/master/sector-base/README.md#api-reference).
 
 
 Or better, generate the documentation locally (until repository is public). Follow the instructions to generate documentation above. Then navigate to:
-- **Sector Base API:** `…/rust-proofs/target/doc/sector_base/api/index.html`
-- **Filecoin Proofs API:** `…/rust-proofs/target/doc/filecoin_proofs/api/index.html`
+- **Sector Base API:** `…/rust-fil-proofs/target/doc/sector_base/api/index.html`
+- **Filecoin Proofs API:** `…/rust-fil-proofs/target/doc/filecoin_proofs/api/index.html`
 
 - [Go implementation of filecoin-proofs API](https://github.com/filecoin-project/go-filecoin/blob/master/proofs/rustprover.go) and [associated interface structures](https://github.com/filecoin-project/go-filecoin/blob/master/proofs/interface.go).
 - [Go implementation of sector-base API](https://github.com/filecoin-project/go-filecoin/blob/master/proofs/disk_backed_sector_store.go).
 
+## Contributing
+
+See [Contributing](CONTRIBUTING.md)
+
 ## License
 
 The Filecoin Project is dual-licensed under Apache 2.0 and MIT terms:
 
-- Apache License, Version 2.0, ([LICENSE-APACHE](https://github.com/filecoin-project/rust-proofs/blob/master/LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
-- MIT license ([LICENSE-MIT](https://github.com/filecoin-project/rust-proofs/blob/master/LICENSE-MIT) or http://opensource.org/licenses/MIT)
+- Apache License, Version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
+- MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)

filecoin-proofs/Cargo.toml

@@ -32,8 +32,10 @@ itertools = "0.7.3"
 serde_cbor = "0.9.0"
 serde = { version = "1", features = ["rc"] }
 serde_derive = "1.0"
+serde_json = "1.0"
 blake2 = "0.8"
 slog = { version = "2.4.1", features = ["max_level_trace", "release_max_level_trace"] }
+regex = "1"
 
 [dev-dependencies]
 gperftools = { git = "https://github.com/dignifiedquire/rust-gperftools" }

filecoin-proofs/build.rs

@@ -75,7 +75,7 @@ fn main() {
         pc_file,
         "Name: libfilecoin_proofs
 Version: {version}
-Description: rust-proofs library
+Description: rust-fil-proofs library
 Libs: {libs}
 ",
         version = git_hash.trim(),

filecoin-proofs/examples/ffi/main.rs

@@ -16,6 +16,7 @@ use ffi_toolkit::c_str_to_rust_str;
 use ffi_toolkit::free_c_str;
 use ffi_toolkit::rust_str_to_c_str;
 use rand::{thread_rng, Rng};
+use std::env;
 use std::error::Error;
 use std::ptr;
 use std::sync::atomic::AtomicPtr;
@@ -64,9 +65,9 @@ unsafe fn create_sector_builder(
     sealed_dir: &TempDir,
     prover_id: [u8; 31],
     last_committed_sector_id: u64,
-) -> (*mut SectorBuilder, u64) {
+    sector_store_config: ConfiguredStore,
+) -> (*mut SectorBuilder, usize) {
     let mut prover_id: [u8; 31] = prover_id;
-    let sector_store_config: ConfiguredStore = ConfiguredStore_ProofTest;
 
     let c_metadata_dir = rust_str_to_c_str(metadata_dir.path().to_str().unwrap());
     let c_sealed_dir = rust_str_to_c_str(sealed_dir.path().to_str().unwrap());
@@ -100,24 +101,56 @@ unsafe fn create_sector_builder(
 
     (
         (*resp).sector_builder,
-        (*resp_2).max_staged_bytes_per_sector,
+        (*resp_2).max_staged_bytes_per_sector as usize,
     )
 }
 
-unsafe fn sector_builder_lifecycle() -> Result<(), Box<Error>> {
+struct ConfigurableSizes {
+    store: ConfiguredStore,
+    max_bytes: usize,
+    first_piece_bytes: usize,
+    second_piece_bytes: usize,
+    third_piece_bytes: usize,
+}
+
+unsafe fn sector_builder_lifecycle(use_live_store: bool) -> Result<(), Box<Error>> {
     let metadata_dir = tempfile::tempdir().unwrap();
     let staging_dir = tempfile::tempdir().unwrap();
     let sealed_dir = tempfile::tempdir().unwrap();
 
-    let (sector_builder_a, max_bytes) =
-        create_sector_builder(&metadata_dir, &staging_dir, &sealed_dir, [0; 31], 123);
+    let sizes = if use_live_store {
+        ConfigurableSizes {
+            store: ConfiguredStore_Live,
+            max_bytes: 266338304,
+            first_piece_bytes: 26214400,
+            second_piece_bytes: 131072000,
+            third_piece_bytes: 157286400,
+        }
+    } else {
+        ConfigurableSizes {
+            store: ConfiguredStore_Test,
+            max_bytes: 1016,
+            first_piece_bytes: 100,
+            second_piece_bytes: 500,
+            third_piece_bytes: 600,
+        }
+    };
+
+    let (sector_builder_a, max_bytes) = create_sector_builder(
+        &metadata_dir,
+        &staging_dir,
+        &sealed_dir,
+        [0; 31],
+        123,
+        sizes.store,
+    );
 
     // TODO: Replace the hard-coded byte amounts with values computed
     // from whatever was retrieved from the SectorBuilder.
-    if max_bytes != 127 {
+    if max_bytes != sizes.max_bytes {
         panic!(
             "test assumes the wrong number of bytes (expected: {}, actual: {})",
-            127, max_bytes
+            sizes.max_bytes, max_bytes
         );
     }
 
@@ -144,7 +177,7 @@ unsafe fn sector_builder_lifecycle() -> Result<(), Box<Error>> {
 
     // add first piece, which lazily provisions a new staged sector
     {
-        let (_, _, resp) = create_and_add_piece(sector_builder_a, 10);
+        let (_, _, resp) = create_and_add_piece(sector_builder_a, sizes.first_piece_bytes);
         defer!(destroy_add_piece_response(resp));
 
         if (*resp).status_code != 0 {
@@ -156,7 +189,7 @@ unsafe fn sector_builder_lifecycle() -> Result<(), Box<Error>> {
 
     // add second piece, which fits into existing staged sector
     {
-        let (_, _, resp) = create_and_add_piece(sector_builder_a, 50);
+        let (_, _, resp) = create_and_add_piece(sector_builder_a, sizes.second_piece_bytes);
         defer!(destroy_add_piece_response(resp));
 
         if (*resp).status_code != 0 {
@@ -168,7 +201,7 @@ unsafe fn sector_builder_lifecycle() -> Result<(), Box<Error>> {
 
     // add third piece, which won't fit into existing staging sector
     {
-        let (_, _, resp) = create_and_add_piece(sector_builder_a, 100);
+        let (_, _, resp) = create_and_add_piece(sector_builder_a, sizes.third_piece_bytes);
         defer!(destroy_add_piece_response(resp));
 
         if (*resp).status_code != 0 {
@@ -197,13 +230,20 @@ unsafe fn sector_builder_lifecycle() -> Result<(), Box<Error>> {
 
     // create a new sector builder using same prover id, which should
     // initialize with metadata persisted by previous sector builder
-    let (sector_builder_b, _) =
-        create_sector_builder(&metadata_dir, &staging_dir, &sealed_dir, [0; 31], 123);
+    let (sector_builder_b, _) = create_sector_builder(
+        &metadata_dir,
+        &staging_dir,
+        &sealed_dir,
+        [0; 31],
+        123,
+        sizes.store,
+    );
     defer!(destroy_sector_builder(sector_builder_b));
 
     // add fourth piece, where size(piece) == max (will trigger sealing)
     let (bytes_in, piece_key) = {
-        let (piece_bytes, piece_key, resp) = create_and_add_piece(sector_builder_b, 127);
+        let (piece_bytes, piece_key, resp) =
+            create_and_add_piece(sector_builder_b, sizes.max_bytes);
         defer!(destroy_add_piece_response(resp));
 
         if (*resp).status_code != 0 {
@@ -251,7 +291,11 @@ unsafe fn sector_builder_lifecycle() -> Result<(), Box<Error>> {
         });
 
         // wait up to 5 minutes for sealing to complete
-        let now_sealed_sector_id = result_rx.recv_timeout(Duration::from_secs(300)).unwrap();
+        let now_sealed_sector_id = if use_live_store {
+            result_rx.recv().unwrap()
+        } else {
+            result_rx.recv_timeout(Duration::from_secs(300)).unwrap()
+        };
 
         assert_eq!(now_sealed_sector_id, 126);
     }
@@ -294,5 +338,12 @@ unsafe fn sector_builder_lifecycle() -> Result<(), Box<Error>> {
 }
 
 fn main() {
-    unsafe { sector_builder_lifecycle().unwrap() };
+    // If TEST_LIVE_SEAL is set, use the Live configuration, and don't unseal
+    // — so process running time will closely approximate sealing time.
+    let use_live_store = match env::var("TEST_LIVE_SEAL") {
+        Ok(_) => true,
+        Err(_) => false,
+    };
+
+    unsafe { sector_builder_lifecycle(use_live_store).unwrap() };
 }