Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: split up window post API into separate calls #1278

Merged
merged 4 commits into from
Sep 14, 2020
+579 −12
Merged

feat: split up window post API into separate calls #1278

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
dfd4235
feat: split up window post API into separate calls
cryptonemo Sep 10, 2020
a561213
feat: verify partition proofs before returning them
cryptonemo Sep 10, 2020
328c04f
feat: fix winning post api split
cryptonemo Sep 10, 2020
4139697
fix: apply review feedback
cryptonemo Sep 11, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
208 changes: 110 additions & 98 deletions filecoin-proofs/src/api/post.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ use crate::caches::{get_post_params, get_post_verifying_key};
use crate::constants::*;
use crate::parameters::{window_post_setup_params, winning_post_setup_params};
use crate::types::{
ChallengeSeed, Commitment, FallbackPoStVanillaProof, PersistentAux, PoStConfig, ProverId,
ChallengeSeed, Commitment, FallbackPoStSectorProof, PersistentAux, PoStConfig, ProverId,
SectorSize, SnarkProof, TemporaryAux, VanillaProof,
};
use crate::PoStType;
Expand Down Expand Up @@ -244,12 +244,12 @@ pub fn clear_caches<Tree: MerkleTreeTrait>(
Ok(())
}

/// Generates a Window proof-of-spacetime with provided vanilla proofs.
/// Generates a Winning proof-of-spacetime with provided vanilla proofs.
pub fn generate_winning_post_with_vanilla<Tree: 'static + MerkleTreeTrait>(
post_config: &PoStConfig,
randomness: &ChallengeSeed,
prover_id: ProverId,
vanilla_proofs: Vec<FallbackPoStVanillaProof<Tree>>,
vanilla_proofs: Vec<FallbackPoStSectorProof<Tree>>,
) -> Result<SnarkProof> {
info!("generate_winning_post_with_vanilla:start");
ensure!(
Expand Down Expand Up @@ -293,7 +293,10 @@ pub fn generate_winning_post_with_vanilla<Tree: 'static + MerkleTreeTrait>(
k: None,
};

let partitions = 1;
let partitions = match pub_params.partitions {
Some(x) => x,
None => 1,
};
let partitioned_proofs = partition_vanilla_proofs(
&post_config,
&pub_params.vanilla_params,
Expand Down Expand Up @@ -460,15 +463,16 @@ pub fn generate_fallback_sector_challenges<Tree: 'static + MerkleTreeTrait>(
let mut sector_challenges: BTreeMap<SectorId, Vec<u64>> = BTreeMap::new();

let num_sectors_per_chunk = post_config.sector_count;
porcuquine marked this conversation as resolved.
Show resolved Hide resolved
let partitions = if post_config.typ == PoStType::Window {
match get_partitions_for_window_post(pub_sectors.len(), &post_config) {
let partitions = match post_config.typ {
PoStType::Window => match get_partitions_for_window_post(pub_sectors.len(), &post_config) {
Some(x) => x,
None => 1,
}
} else {
assert_eq!(post_config.typ, PoStType::Winning);
},
PoStType::Winning => {
assert_eq!(post_config.typ, PoStType::Winning);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assertion is quite redundant here. The previous line literally guarantees this will never fail, so I think it can be removed. I would instead add a comment justifying the decision. Since I don't think the number of partitions is structurally guaranteed to be one, I would probably add a comment justifying this decision. That comment should make clear what change to the code base (even if it can't be conveniently checked here) would necessitate a change here.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, that looks like a hold-over from the previous if/else branch change. Rather than update and push, I'm opting to merge this and update in a future PR.


1
1
}
};

for partition_index in 0..partitions {
Expand All @@ -484,13 +488,13 @@ pub fn generate_fallback_sector_challenges<Tree: 'static + MerkleTreeTrait>(
let challenge_index = ((partition_index * post_config.sector_count + i)
* post_config.challenge_count
+ n) as u64;
let challenged_leaf_start = fallback::generate_leaf_challenge(
let challenged_leaf = fallback::generate_leaf_challenge(
&public_params,
randomness_safe,
u64::from(*sector),
challenge_index,
)?;
challenges.push(challenged_leaf_start);
challenges.push(challenged_leaf);
}

sector_challenges.insert(*sector, challenges);
Expand All @@ -502,27 +506,31 @@ pub fn generate_fallback_sector_challenges<Tree: 'static + MerkleTreeTrait>(
Ok(sector_challenges)
}

/// Generates a single vanilla proof required for Window proof-of-spacetime.
/// Generates a single vanilla proof required for either Window proof-of-spacetime
/// or Winning proof-of-spacetime.
pub fn generate_single_vanilla_proof<Tree: 'static + MerkleTreeTrait>(
post_config: &PoStConfig,
sector_id: SectorId,
replica: &PrivateReplicaInfo<Tree>,
challenges: &[u64],
) -> Result<FallbackPoStVanillaProof<Tree>> {
let sector_count = 1;

) -> Result<FallbackPoStSectorProof<Tree>> {
info!("generate_single_vanilla_proof:start");
ensure!(
post_config.sector_count == sector_count,
"invalid post config sector size ({} required)",
sector_count
);

let tree = &replica.merkle_tree(post_config.sector_size)?;
let comm_r = replica.safe_comm_r()?;
let comm_c = replica.safe_comm_c()?;
let comm_r_last = replica.safe_comm_r_last()?;

// There is only enough information in the arguments to generate a
// single vanilla proof, so the behaviour is unexpected if the
// sector_count provided is > 1.
let sector_count = post_config.sector_count;
ensure!(
post_config.sector_count == 1,
"invalid post config sector size ({} required)",
sector_count
);

let mut priv_sectors = Vec::with_capacity(sector_count);
priv_sectors.push(fallback::PrivateSector {
tree,
Expand All @@ -534,11 +542,11 @@ pub fn generate_single_vanilla_proof<Tree: 'static + MerkleTreeTrait>(
sectors: &priv_sectors,
};

let vanilla_proof = fallback::vanilla_proofs(sector_id, &priv_inputs, challenges)?;
let vanilla_proof = fallback::vanilla_proof(sector_id, &priv_inputs, challenges)?;

info!("generate_single_vanilla_proof:finish");

Ok(FallbackPoStVanillaProof {
Ok(FallbackPoStSectorProof {
sector_id,
comm_r,
vanilla_proof,
Expand Down Expand Up @@ -644,15 +652,15 @@ pub fn verify_winning_post<Tree: 'static + MerkleTreeTrait>(
Ok(true)
}

// Partition a flat vector of vanilla proofs -- the post_config type
// is required in order to determine the proper shape of the returned
// partitioned proofs.
// Partition a flat vector of vanilla sector proofs. The post_config
// (PoSt) type is required in order to determine the proper shape of
// the returned partitioned proofs.
pub fn partition_vanilla_proofs<Tree: MerkleTreeTrait>(
post_config: &PoStConfig,
pub_params: &fallback::PublicParams,
pub_inputs: &fallback::PublicInputs<<Tree::Hasher as Hasher>::Domain>,
partition_count: usize,
vanilla_proofs: &[FallbackPoStVanillaProof<Tree>],
vanilla_proofs: &[FallbackPoStSectorProof<Tree>],
) -> Result<Vec<VanillaProof<Tree>>> {
info!("partition_vanilla_proofs:start");
ensure!(
Expand All @@ -675,80 +683,84 @@ pub fn partition_vanilla_proofs<Tree: MerkleTreeTrait>(

// Note that the partition proofs returned are shaped differently
// based on which type of PoSt is being considered.
if post_config.typ == PoStType::Window {
for (j, sectors_chunk) in pub_inputs.sectors.chunks(num_sectors_per_chunk).enumerate() {
trace!("processing partition {}", j);

let mut proofs = Vec::with_capacity(num_sectors_per_chunk);

for pub_sector in sectors_chunk.iter() {
let cur_proof = vanilla_proofs
.iter()
.find(|&proof| proof.sector_id == pub_sector.id)
.expect("failed to locate sector proof");

// Note: Window post requires all inclusion proofs (based on the challenge
// count per sector) per sector proof.
proofs.extend(cur_proof.vanilla_proof.sectors.clone());
}

// If there were less than the required number of sectors provided, we duplicate the last one
// to pad the proof out, such that it works in the circuit part.
while proofs.len() < num_sectors_per_chunk {
proofs.push(proofs[proofs.len() - 1].clone());
match post_config.typ {
PoStType::Window => {
for (j, sectors_chunk) in pub_inputs.sectors.chunks(num_sectors_per_chunk).enumerate() {
trace!("processing partition {}", j);

let mut sector_proofs = Vec::with_capacity(num_sectors_per_chunk);

for pub_sector in sectors_chunk.iter() {
let cur_proof = vanilla_proofs
.iter()
.find(|&proof| proof.sector_id == pub_sector.id)
.expect("failed to locate sector proof");

// Note: Window post requires all inclusion proofs (based on the challenge
// count per sector) per sector proof.
sector_proofs.extend(cur_proof.vanilla_proof.sectors.clone());
}

// If there were less than the required number of sectors provided, we duplicate the last one
// to pad the proof out, such that it works in the circuit part.
while sector_proofs.len() < num_sectors_per_chunk {
sector_proofs.push(sector_proofs[sector_proofs.len() - 1].clone());
}

partition_proofs.push(fallback::Proof::<<Tree as MerkleTreeTrait>::Proof> {
sectors: sector_proofs,
});
}

partition_proofs
.push(fallback::Proof::<<Tree as MerkleTreeTrait>::Proof> { sectors: proofs });
}
} else {
for (j, sectors_chunk) in vanilla_proofs.chunks(num_sectors_per_chunk).enumerate() {
trace!("processing partition {}", j);

// Winning post sector_count is winning post challenges per sector
ensure!(
post_config.sector_count == vanilla_proofs[j].vanilla_proof.sectors.len(),
"invalid number of sector proofs for Winning PoSt"
);

let mut proofs = Vec::with_capacity(post_config.challenge_count);

// Sanity check incoming structure
ensure!(
sectors_chunk.len() == 1,
"Invalid sector chunk for Winning PoSt"
);
ensure!(
sectors_chunk[0].vanilla_proof.sectors.len() == 1,
"Invalid sector count for Winning PoSt chunk"
);

let cur_sector_proof = &sectors_chunk[0].vanilla_proof.sectors[0];

// Unroll inclusions proofs from the single provided sector_proof (per partition)
// into individual sector proofs, required for winning post.
for cur_inclusion_proof in cur_sector_proof.inclusion_proofs() {
proofs.push(SectorProof {
inclusion_proofs: vec![cur_inclusion_proof.clone()],
comm_c: cur_sector_proof.comm_c,
comm_r_last: cur_sector_proof.comm_r_last,
PoStType::Winning => {
for (j, sectors_chunk) in vanilla_proofs.chunks(num_sectors_per_chunk).enumerate() {
trace!("processing partition {}", j);

// Sanity check incoming structure
ensure!(
sectors_chunk.len() == 1,
"Invalid sector chunk for Winning PoSt"
);
ensure!(
sectors_chunk[0].vanilla_proof.sectors.len() == 1,
"Invalid sector count for Winning PoSt chunk"
);

// Winning post sector_count is winning post challenges per sector
ensure!(
post_config.sector_count == sectors_chunk[j].vanilla_proof.sectors.len(),
"invalid number of sector proofs for Winning PoSt"
);

let mut sector_proofs = Vec::with_capacity(post_config.challenge_count);
let cur_sector_proof = &sectors_chunk[0].vanilla_proof.sectors[0];

// Unroll inclusions proofs from the single provided sector_proof (per partition)
// into individual sector proofs, required for winning post.
for cur_inclusion_proof in cur_sector_proof.inclusion_proofs() {
sector_proofs.push(SectorProof {
inclusion_proofs: vec![cur_inclusion_proof.clone()],
comm_c: cur_sector_proof.comm_c,
comm_r_last: cur_sector_proof.comm_r_last,
});
}

// If there were less than the required number of sectors provided, we duplicate the last one
// to pad the proof out, such that it works in the circuit part.
while sector_proofs.len() < num_sectors_per_chunk {
sector_proofs.push(sector_proofs[sector_proofs.len() - 1].clone());
}

// Winning post Challenge count is the total winning post challenges
ensure!(
sector_proofs.len() == post_config.challenge_count,
"invalid number of partition proofs based on Winning PoSt challenges"
);

partition_proofs.push(fallback::Proof::<<Tree as MerkleTreeTrait>::Proof> {
sectors: sector_proofs,
});
}

// If there were less than the required number of sectors provided, we duplicate the last one
// to pad the proof out, such that it works in the circuit part.
while proofs.len() < num_sectors_per_chunk {
proofs.push(proofs[proofs.len() - 1].clone());
}

// Winning post Challenge count is the total winning post challenges
ensure!(
proofs.len() == post_config.challenge_count,
"invalid number of partition proofs based on Winning PoSt challenges"
);

partition_proofs
.push(fallback::Proof::<<Tree as MerkleTreeTrait>::Proof> { sectors: proofs });
}
}

Expand All @@ -771,7 +783,7 @@ pub fn generate_window_post_with_vanilla<Tree: 'static + MerkleTreeTrait>(
post_config: &PoStConfig,
randomness: &ChallengeSeed,
prover_id: ProverId,
vanilla_proofs: Vec<FallbackPoStVanillaProof<Tree>>,
vanilla_proofs: Vec<FallbackPoStSectorProof<Tree>>,
) -> Result<SnarkProof> {
info!("generate_window_post_with_vanilla:start");
ensure!(
Expand Down
5 changes: 4 additions & 1 deletion filecoin-proofs/src/types/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,8 +89,11 @@ pub struct SealPreCommitPhase1Output<Tree: MerkleTreeTrait> {
pub type SnarkProof = Vec<u8>;
pub type VanillaProof<Tree> = Proof<<Tree as MerkleTreeTrait>::Proof>;

// This FallbackPoStSectorProof is used during Fallback PoSt, but
// contains only Vanilla proof information and is not a full Fallback
// PoSt proof.
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct FallbackPoStVanillaProof<Tree: MerkleTreeTrait> {
pub struct FallbackPoStSectorProof<Tree: MerkleTreeTrait> {
pub sector_id: SectorId,
pub comm_r: <Tree::Hasher as Hasher>::Domain,
#[serde(bound(
Expand Down
4 changes: 2 additions & 2 deletions storage-proofs/core/src/compound_proof.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,15 +78,15 @@ where
// This will always run at least once, since there cannot be zero partitions.
ensure!(partition_count > 0, "There must be partitions");

info!("vanilla_proof:start");
info!("vanilla_proofs:start");
let vanilla_proofs = S::prove_all_partitions(
&pub_params.vanilla_params,
&pub_in,
priv_in,
partition_count,
)?;

info!("vanilla_proof:finish");
info!("vanilla_proofs:finish");

let sanity_check =
S::verify_all_partitions(&pub_params.vanilla_params, &pub_in, &vanilla_proofs)?;
Expand Down
12 changes: 5 additions & 7 deletions storage-proofs/post/src/fallback/vanilla.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -244,14 +244,14 @@ enum ProofOrFault<T> {
}

// Generates a single vanilla proof, given the private inputs and sector challenges.
pub fn vanilla_proofs<Tree: MerkleTreeTrait>(
pub fn vanilla_proof<Tree: MerkleTreeTrait>(
sector_id: SectorId,
priv_inputs: &PrivateInputs<Tree>,
challenges: &[u64],
) -> Result<Proof<Tree::Proof>> {
ensure!(
priv_inputs.sectors.len() == 1,
"vanilla_proofs called with multiple sector proofs"
"vanilla_proof called with multiple sector proofs"
);

let priv_sector = &priv_inputs.sectors[0];
Expand All @@ -271,13 +271,11 @@ pub fn vanilla_proofs<Tree: MerkleTreeTrait>(
let inclusion_proofs = (0..challenges.len())
.into_par_iter()
.map(|challenged_leaf_index| {
let challenged_leaf_start = challenges[challenged_leaf_index];
let proof =
tree.gen_cached_proof(challenged_leaf_start as usize, Some(rows_to_discard))?;
let challenged_leaf = challenges[challenged_leaf_index];
let proof = tree.gen_cached_proof(challenged_leaf as usize, Some(rows_to_discard))?;

ensure!(
proof.validate(challenged_leaf_start as usize)
&& proof.root() == priv_sector.comm_r_last,
proof.validate(challenged_leaf as usize) && proof.root() == priv_sector.comm_r_last,
"Generated vanilla proof for sector {} is invalid",
sector_id
);
Expand Down