Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update rand, proc-macro and zstd dependencies #488

Merged
merged 9 commits into from
Jul 6, 2021
Merged

Update rand, proc-macro and zstd dependencies #488

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
cd5bb8b
update rand
alamb Jun 21, 2021
578f770
Update code for new rand interface
alamb Jun 22, 2021
4dd8090
update zstd
alamb Jun 21, 2021
24801d7
update proc-macro2
alamb Jun 21, 2021
efec71c
another
alamb Jun 22, 2021
1d228cc
Specify js feature of getrandom
alamb Jun 22, 2021
e07ca05
enable rng std stuff
alamb Jun 22, 2021
e349b96
checkpoint
alamb Jul 6, 2021
6ccb25a
cleanup
alamb Jul 6, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion arrow-flight/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ futures = { version = "0.3", default-features = false, features = ["alloc"]}
tonic-build = "0.4"
# Pin specific version of the tonic-build dependencies to avoid auto-generated
# (and checked in) arrow.flight.protocol.rs from changing
proc-macro2 = "=1.0.24"
proc-macro2 = "=1.0.27"

#[lib]
#name = "flight"
Expand Down
2 changes: 2 additions & 0 deletions arrow-pyarrow-integration-testing/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ name = "arrow_pyarrow_integration_testing"
crate-type = ["cdylib"]

[dependencies]
# ensure we get the std version of rand so arrow builds without default features
rand = { version = "0.8" }
arrow = { path = "../arrow", version = "5.0.0-SNAPSHOT" }
pyo3 = { version = "0.12.1", features = ["extension-module"] }

Expand Down
5 changes: 4 additions & 1 deletion arrow/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,10 @@ serde = { version = "1.0", features = ["rc"] }
serde_derive = "1.0"
serde_json = { version = "1.0", features = ["preserve_order"] }
indexmap = "1.6"
rand = "0.7"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, I removed the wacky std feature and found a different work around for arrow-pyarrow-integration-testing

If this passes CI I think it is good enough to merge

rand = { version = "0.8", default-features = false }
# getrandom is a dependency of rand, not (directly) of arrow
# need to specify `js` feature to build on wasm
getrandom = { version = "0.2", features = ["js"] }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we may be able to achieve the same using rand with feature getrandom/js?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will give it a try

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried this and sadly I am told:

alamb@ip-10-0-0-124:~/Software/arrow-rs/arrow$ cargo build --target wasm32-unknown-unknown
error: failed to parse manifest at `/Users/alamb/Software/arrow-rs/arrow/Cargo.toml`

Caused by:
  feature `getrandom/js` in dependency `rand` is not allowed to contain slashes
  If you want to enable features of a transitive dependency, the direct dependency needs to re-export those features from the `[features]` table.
alamb@ip-10-0-0-124:~/Software/arrow-rs/arrow$ git diff Cargo.toml
diff --git a/arrow/Cargo.toml b/arrow/Cargo.toml
index 478f33678..f6255a93d 100644
--- a/arrow/Cargo.toml
+++ b/arrow/Cargo.toml
@@ -40,10 +40,7 @@ serde = { version = "1.0", features = ["rc"] }
 serde_derive = "1.0"
 serde_json = { version = "1.0", features = ["preserve_order"] }
 indexmap = "1.6"
-rand = { version = "0.8", default-features = false }
-# getrandom is a dependency of rand, not (directly) of arrow
-# need to specify `js` feature to build on wasm
-getrandom = { version = "0.2", features = ["js"] }
+rand = { version = "0.8", default-features = false, features = ["getrandom/js"] }
 num = "0.4"
 csv_crate = { version = "1.1", optional = true, package="csv" }
 regex = "1.3"
alamb@ip-10-0-0-124:~/Software/arrow-rs/arrow$

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sadly, it appears this workaround is the recommended way from rand rate: https://crates.io/crates/rand

WASM support

The WASM target wasm32-unknown-unknown is not automatically supported by rand or getrandom. To solve this, either use a different target such as wasm32-wasi or add a direct dependancy on getrandom with the js feature (if the target supports JavaScript). See getrandom#WebAssembly support.

num = "0.4"
csv_crate = { version = "1.1", optional = true, package="csv" }
regex = "1.3"
Expand Down
4 changes: 2 additions & 2 deletions arrow/benches/mutable_array.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@ fn create_slices(size: usize) -> Vec<(usize, usize)> {

(0..size)
.map(|_| {
let start = rng.gen_range(0, size / 2);
let end = rng.gen_range(start + 1, size);
let start = rng.gen_range(0..size / 2);
let end = rng.gen_range(start + 1..size);
(start, end)
})
.collect()
Expand Down
2 changes: 1 addition & 1 deletion arrow/benches/take_kernels.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ fn create_random_index(size: usize, null_density: f32) -> UInt32Array {
if rng.gen::<f32>() < null_density {
builder.append_null().unwrap()
} else {
let value = rng.gen_range::<u32, _, _>(0u32, size as u32);
let value = rng.gen_range::<u32, _>(0u32..size as u32);
builder.append_value(value).unwrap();
}
}
Expand Down
5 changes: 3 additions & 2 deletions arrow/src/util/bench_util.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,8 @@ pub fn create_string_array<Offset: StringOffsetSizeTrait>(
if rng.gen::<f32>() < null_density {
None
} else {
let value = rng.sample_iter(&Alphanumeric).take(4).collect::<String>();
let value = rng.sample_iter(&Alphanumeric).take(4).collect();
let value = String::from_utf8(value).unwrap();
Some(value)
}
})
Expand All @@ -124,7 +125,7 @@ pub fn create_binary_array<Offset: BinaryOffsetSizeTrait>(
} else {
let value = rng
.sample_iter::<u8, _>(Standard)
.take(range_rng.gen_range(0, 8))
.take(range_rng.gen_range(0..8))
.collect::<Vec<u8>>();
Some(value)
}
Expand Down
2 changes: 1 addition & 1 deletion arrow/src/util/bit_util.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -271,7 +271,7 @@ mod tests {
let mut v = HashSet::new();
let mut rng = seedable_rng();
for _ in 0..NUM_SETS {
let offset = rng.gen_range(0, 8 * NUM_BYTES);
let offset = rng.gen_range(0..8 * NUM_BYTES);
v.insert(offset);
set_bit(&mut buffer[..], offset);
}
Expand Down
2 changes: 1 addition & 1 deletion arrow/src/util/data_gen.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,7 @@ fn create_random_offsets<T: OffsetSizeTrait + SampleUniform>(
offsets.push(current_offset);

(0..size).for_each(|_| {
current_offset += rng.gen_range(min, max);
current_offset += rng.gen_range(min..max);
offsets.push(current_offset);
});

Expand Down
2 changes: 1 addition & 1 deletion arrow/src/util/test_util.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ pub fn random_bytes(n: usize) -> Vec<u8> {
let mut result = vec![];
let mut rng = seedable_rng();
for _ in 0..n {
result.push(rng.gen_range(0, 255));
result.push(rng.gen_range(0..255));
}
result
}
Expand Down
3 changes: 1 addition & 2 deletions parquet/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ snap = { version = "1.0", optional = true }
brotli = { version = "3.3", optional = true }
flate2 = { version = "1.0", optional = true }
lz4 = { version = "1.23", optional = true }
zstd = { version = "0.8", optional = true }
zstd = { version = "0.9", optional = true }
chrono = "0.4"
num-bigint = "0.4"
arrow = { path = "../arrow", version = "5.0.0-SNAPSHOT", optional = true }
Expand All @@ -54,7 +54,6 @@ snap = "1.0"
brotli = "3.3"
flate2 = "1.0"
lz4 = "1.23"
zstd = "0.8"
arrow = { path = "../arrow", version = "5.0.0-SNAPSHOT" }
serde_json = { version = "1.0", features = ["preserve_order"] }

Expand Down