Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Miri failure on rustc 1.73.0-nightly (31395ec38 2023-07-24) #235

Closed
lopopolo opened this issue Jul 26, 2023 · 1 comment · Fixed by #236
Closed

Miri failure on rustc 1.73.0-nightly (31395ec38 2023-07-24) #235

lopopolo opened this issue Jul 26, 2023 · 1 comment · Fixed by #236
Labels
A-interner Area: String interners, data structures, and types. A-security Area: Security vulnerabilities and unsoundness issues. C-bug Category: This is a bug. E-help-wanted Call for participation: Help is requested to fix this issue.

Comments

@lopopolo
Copy link
Member

https://github.com/artichoke/intaglio/actions/runs/5663120563/job/15344308090

error: Undefined Behavior: trying to retag from <99387> for SharedReadOnly permission at alloc31495[0x0], but that tag does not exist in the borrow stack for this location
   --> /home/runner/work/intaglio/intaglio/src/bytes.rs:713:45
    |
713 |         debug_assert_eq!(self.get(id), Some(slice));
    |                                             ^^^^^
    |                                             |
    |                                             trying to retag from <99387> for SharedReadOnly permission at alloc31495[0x0], but that tag does not exist in the borrow stack for this location
    |                                             this error occurs as part of retag at alloc31495[0x0..0x64]
    |
    = help: this indicates a potential bug in the program: it performed an invalid operation, but the Stacked Borrows rules it violated are still experimental
    = help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md for further information
help: <99387> was created by a SharedReadOnly retag at offsets [0x0..0x64]
   --> /home/runner/work/intaglio/intaglio/src/bytes.rs:708:30
    |
708 |         let slice = unsafe { name.as_static_slice() };
    |                              ^^^^^^^^^^^^^^^^^^^^^^
help: <99387> was later invalidated at offsets [0x0..0x64] by a Unique retag (of a reference/box inside this compound value)
   --> /home/runner/work/intaglio/intaglio/src/bytes.rs:711:23
    |
711 |         self.vec.push(name);
    |                       ^^^^
    = note: BACKTRACE (of the first span):
    = note: inside `intaglio::bytes::SymbolTable::intern::<std::vec::Vec<u8>>` at /home/runner/work/intaglio/intaglio/src/bytes.rs:713:45: 713:50
note: inside `bytes::dealloc_owned_data`
   --> tests/leak_drop/bytes.rs:9:22
    |
9   |         let sym_id = table.intern(symbol.clone()).unwrap();
    |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside closure
   --> tests/leak_drop/bytes.rs:4:25
    |
3   | #[test]
    | ------- in this procedural macro expansion
4   | fn dealloc_owned_data() {
    |                         ^
    = note: this error originates in the attribute macro `test` (in Nightly builds, run with -Z macro-backtrace for more info)

note: some details are omitted, run with `MIRIFLAGS=-Zmiri-backtrace=full` for a verbose backtrace

error: aborting due to previous error

error: test failed, to rerun pass `--test leak_drop`
@lopopolo lopopolo added A-security Area: Security vulnerabilities and unsoundness issues. C-bug Category: This is a bug. labels Jul 26, 2023
@lopopolo
Copy link
Member Author

Implicated code:

intaglio/src/bytes.rs

Lines 687 to 717 in 70ccd8b

pub fn intern<T>(&mut self, contents: T) -> Result<Symbol, SymbolOverflowError>
where
T: Into<Cow<'static, [u8]>>,
{
let contents = contents.into();
if let Some(&id) = self.map.get(&*contents) {
return Ok(id);
}
let name = Interned::from(contents);
let id = self.map.len().try_into()?;
// SAFETY: This expression creates a reference with a `'static` lifetime
// from an owned and interned buffer, which is permissible because:
//
// - `Interned` is an internal implementation detail of `SymbolTable`.
// - `SymbolTable` never gives out `'static` references to underlying
// byte string byte contents.
// - All slice references given out by the `SymbolTable` have the same
// lifetime as the `SymbolTable`.
// - The `map` field of `SymbolTable`, which contains the `'static`
// references, is dropped before the owned buffers stored in this
// `Interned`.
let slice = unsafe { name.as_static_slice() };
self.map.insert(slice, id);
self.vec.push(name);
debug_assert_eq!(self.get(id), Some(slice));
debug_assert_eq!(self.intern(slice), Ok(id));
Ok(id)
}

@lopopolo lopopolo added E-help-wanted Call for participation: Help is requested to fix this issue. A-interner Area: String interners, data structures, and types. labels Jul 26, 2023
lopopolo added a commit that referenced this issue Jul 26, 2023
This stacked borrows violation impacts all `SymbolTable` variants in
this crate and has been present in all published versions.

The `SymbolTable::intern` function previously worked like this:

1. Convert the given contents to be interned into a `Cow<'static, T>`.
2. Check to see if this content is already interned by checking the
  internal bimap. If found, return the matching symbol.
3. Convert the `Cow` into an `Interned` which is an enum of either a
  &'static T` or a `Box<T>`.
4. Borrow a `&'static T` from the `Interned`.
5. Push the `Interned` into the vec.
6. Add the `&'static T` to the map.
7. Check that the insertion happened correctly with some debug
  assertions.
8. Return the computed symbol.

As of at least rustc 1.73.0-nightly (31395ec38 2023-07-24), Miri pops
all borrows from the `Interned` at step 5 when the `Interned` is moved
into the vec, which means the borrow made at 4 is untracked when Miri
executes the debug assertions.

The fix here is to make the assignment/move of the `Interned` happen
first so Miri doesn't retag the inner `Box<T>`. This is accomplished by
reordering the `self.vec.push` operation to immediately follow step 3.
An additional unchecked index into the vec is used to get a shared
access to that `Interned` object to later derive the `&'static T`.

This commit also adds an additional test to resolve a slice from the
symbol table to ensure Miri does not disagree as the `Interned` is moved
during growth of the `SymbolTable`.

Fixes #235.
lopopolo added a commit that referenced this issue Jul 26, 2023
This stacked borrows violation impacts all `SymbolTable` variants in
this crate and has been present in all published versions.

The `SymbolTable::intern` function previously worked like this:

1. Convert the given contents to be interned into a `Cow<'static, T>`.
2. Check to see if this content is already interned by checking the
  internal bimap. If found, return the matching symbol.
3. Convert the `Cow` into an `Interned` which is an enum of either a
  &'static T` or a `Box<T>`.
4. Borrow a `&'static T` from the `Interned`.
5. Push the `Interned` into the vec.
6. Add the `&'static T` to the map.
7. Check that the insertion happened correctly with some debug
  assertions.
8. Return the computed symbol.

As of at least rustc 1.73.0-nightly (31395ec38 2023-07-24), Miri pops
all borrows from the `Interned` at step 5 when the `Interned` is moved
into the vec, which means the borrow made at 4 is untracked when Miri
executes the debug assertions.

The fix here is to make the assignment/move of the `Interned` happen
first so Miri doesn't retag the inner `Box<T>`. This is accomplished by
reordering the `self.vec.push` operation to immediately follow step 3.
An additional unchecked index into the vec is used to get a shared
access to that `Interned` object to later derive the `&'static T`.

This commit also adds an additional test to resolve a slice from the
symbol table to ensure Miri does not disagree as the `Interned` is moved
during growth of the `SymbolTable`.

Fixes #235.
lopopolo added a commit that referenced this issue Jul 26, 2023
This stacked borrows violation impacts all `SymbolTable` variants in
this crate and has been present in all published versions.

The `SymbolTable::intern` function previously worked like this:

1. Convert the given contents to be interned into a `Cow<'static, T>`.
2. Check to see if this content is already interned by checking the
  internal bimap. If found, return the matching symbol.
3. Convert the `Cow` into an `Interned` which is an enum of either a
  &'static T` or a `Box<T>`.
4. Borrow a `&'static T` from the `Interned`.
5. Push the `Interned` into the vec.
6. Add the `&'static T` to the map.
7. Check that the insertion happened correctly with some debug
  assertions.
8. Return the computed symbol.

As of at least rustc 1.73.0-nightly (31395ec38 2023-07-24), Miri pops
all borrows from the `Interned` at step 5 when the `Interned` is moved
into the vec, which means the borrow made at 4 is untracked when Miri
executes the debug assertions.

The fix here is to make the assignment/move of the `Interned` happen
first so Miri doesn't retag the inner `Box<T>`. This is accomplished by
reordering the `self.vec.push` operation to immediately follow step 3.
An additional unchecked index into the vec is used to get a shared
access to that `Interned` object to later derive the `&'static T`.

This commit also adds an additional test to resolve a slice from the
symbol table to ensure Miri does not disagree as the `Interned` is moved
during growth of the `SymbolTable`.

Fixes #235.
lopopolo added a commit that referenced this issue Jul 26, 2023
This stacked borrows violation impacts all `SymbolTable` variants in
this crate and has been present in all published versions.

The `SymbolTable::intern` function previously worked like this:

1. Convert the given contents to be interned into a `Cow<'static, T>`.
2. Check to see if this content is already interned by checking the
  internal bimap. If found, return the matching symbol.
3. Convert the `Cow` into an `Interned` which is an enum of either a
  &'static T` or a `Box<T>`.
4. Borrow a `&'static T` from the `Interned`.
5. Push the `Interned` into the vec.
6. Add the `&'static T` to the map.
7. Check that the insertion happened correctly with some debug
  assertions.
8. Return the computed symbol.

As of at least rustc 1.73.0-nightly (31395ec38 2023-07-24), Miri pops
all borrows from the `Interned` at step 5 when the `Interned` is moved
into the vec, which means the borrow made at 4 is untracked when Miri
executes the debug assertions.

The fix here is to make the assignment/move of the `Interned` happen
first so Miri doesn't retag the inner `Box<T>`. This is accomplished by
reordering the `self.vec.push` operation to immediately follow step 3.
An additional unchecked index into the vec is used to get a shared
access to that `Interned` object to later derive the `&'static T`.

This commit also adds an additional test to resolve a slice from the
symbol table to ensure Miri does not disagree as the `Interned` is moved
during growth of the `SymbolTable`.

Fixes #235.
lopopolo added a commit that referenced this issue Jul 26, 2023
See #235, #236.
See #236 (comment).

Introduce a new `PinBox` type which stores a boxed slice as a `NonNull`
pointer. The pointer is converted back to a `Box` on drop.
lopopolo added a commit that referenced this issue Jul 26, 2023
See #235, #236.
See #236 (comment).

Introduce a new `PinBox` type which stores a boxed slice as a `NonNull`
pointer. The pointer is converted back to a `Box` on drop.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-interner Area: String interners, data structures, and types. A-security Area: Security vulnerabilities and unsoundness issues. C-bug Category: This is a bug. E-help-wanted Call for participation: Help is requested to fix this issue.
Development

Successfully merging a pull request may close this issue.

1 participant