Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rollup of 3 pull requests #67412

Closed

Conversation

Mark-Simulacrum
Copy link
Member

Successful merges:

Failed merges:

r? @ghost

pnkfelix and others added 16 commits December 4, 2019 11:15
however, make sure run-make test actually runs, too. (The `# only-target` thing
does not work, at least not for cross-compilation. See issue 67018.))
Instead of accumulating (and acting upon) LTO import information over an
unbounded number of prior compilations, just see if the curent import
set matches the previous import set. if they don't match, then you cannot
reuse the PostLTO build product for that module.
…=dtolnay

Implement `DebugStruct::non_exhaustive`.

This patch adds a function (finish_non_exhaustive) to add ellipsis before the closing brace when formatting using `DebugStruct`.

 ## Example

 ```rust
 #![feature(debug_non_exhaustive)]
 use std::fmt;

 struct Bar {
     bar: i32,
     hidden: f32,
 }

 impl fmt::Debug for Bar {
     fn fmt(&self, fmt: &mut fmt::Formatter<'_>) -> fmt::Result {
         fmt.debug_struct("Bar")
            .field("bar", &self.bar)
            .non_exhaustive(true) // Show that some other field(s) exist.
            .finish()
     }
 }

 assert_eq!(
     format!("{:?}", Bar { bar: 10, hidden: 1.0 }),
     "Bar { bar: 10, .. }",
 );
 ```
…t-lto-imports, r=michaelwoerister

save LTO import info and check it when trying to reuse build products

Fix rust-lang#59535

Previous runs of LTO optimization on the previous incremental build can import larger portions of the dependence graph into a codegen unit than the current compilation run is choosing to import. We need to take that into account when we choose to reuse PostLTO-optimization object files from previous compiler invocations.

This PR accomplishes that by serializing the LTO import information on each incremental build. We load up the previous LTO import data as well as the current LTO import data. Then as we decide whether to reuse previous PostLTO objects or redo LTO optimization, we check whether the LTO import data matches. After we finish with this decision process for every object, we write the LTO import data back to disk.

----

What is the scenario where comparing against past LTO import information is necessary?

I've tried to capture it in the comments in the regression test, but here's yet another attempt from me to summarize the situation:

 1. Consider a call-graph like `[A] -> [B -> D] <- [C]` (where the letters are functions and the modules are enclosed in `[]`)
 2. In our specific instance, the earlier compilations were inlining the call to`B` into `A`; thus `A` ended up with a external reference to the symbol `D` in its object code, to be resolved at subsequent link time. The LTO import information provided by LLVM for those runs reflected that information: it explicitly says during those runs, `B` definition and `D` declaration were imported into `[A]`.
 3. The change between incremental builds was that the call `D <- C` was removed.
 4. That change, coupled with other decisions within `rustc`, made the compiler decide to make `D` an internal symbol (since it was no longer accessed from other codegen units, this makes sense locally). And then the definition of `D` was inlined into `B` and `D` itself was eliminated entirely.
  5. The current LTO import information reported that `B` alone is imported into `[A]` for the *current compilation*. So when the Rust compiler surveyed the dependence graph, it determined that nothing `[A]` imports changed since the last build (and `[A]` itself has not changed either), so it chooses to reuse the object code generated during the previous compilation.
  6. But that previous object code has an unresolved reference to `D`, and that causes a link time failure!

----

The interesting thing is that its quite hard to actually observe the above scenario arising, which is probably why no one has noticed this bug in the year or so since incremental LTO support landed (PR rust-lang#53673).

I've literally spent days trying to observe the bug on my local machine, but haven't managed to find the magic combination of factors to get LLVM and `rustc` to do just the right set of the inlining and `internal`-reclassification choices that cause this particular problem to arise.

----

Also, I have tried to be careful about injecting new bugs with this PR. Specifically, I was/am worried that we could get into a scenario where overwriting the current LTO import data with past LTO import data would cause us to "forget" a current import. ~~To guard against this, the PR as currently written always asserts, at overwrite time, that the past LTO import-set is a *superset* of the current LTO import-set. This way, the overwriting process should always be safe to run.~~
 * The previous note was written based on the first version of this PR. It has since been revised to use a simpler strategy, where we never attempt to merge the past LTO import information into the current one. We just *compare* them, and act accordingly.
 * Also, as you can see from the comments on the PR itself, I was quite right to be worried about forgetting past imports; that scenario was observable via a trivial transformation of the regression test I had devised.
…=varkor

Use structured suggestion for disambiguating method calls

Fix rust-lang#65635.
@Mark-Simulacrum
Copy link
Member Author

@bors r+ p=100

@bors
Copy link
Contributor

bors commented Dec 19, 2019

📌 Commit dced4c2 has been approved by Mark-Simulacrum

@bors
Copy link
Contributor

bors commented Dec 19, 2019

🌲 The tree is currently closed for pull requests below priority 100, this pull request will be tested once the tree is reopened

@bors bors added the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label Dec 19, 2019
@JohnTitor JohnTitor added the rollup A PR which is a rollup label Dec 19, 2019
@bors
Copy link
Contributor

bors commented Dec 19, 2019

⌛ Testing commit dced4c2 with merge 45c7edc53117ae3185133abc07adf17cb2fb358d...

@rust-highfive
Copy link
Collaborator

The job dist-various-1 of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-12-19T04:46:43.1092230Z 
2019-12-19T04:46:43.1092269Z 
2019-12-19T04:46:43.1092331Z failures:
2019-12-19T04:46:43.1092415Z 
2019-12-19T04:46:43.1092768Z ---- [run-make] run-make/removing-code-and-incremental-lto stdout ----
2019-12-19T04:46:43.1092920Z error: make failed
2019-12-19T04:46:43.1093004Z status: exit code: 2
2019-12-19T04:46:43.1093071Z command: "make" "make"
2019-12-19T04:46:43.1093153Z stdout:
2019-12-19T04:46:43.1093153Z stdout:
2019-12-19T04:46:43.1093396Z ------------------------------------------
2019-12-19T04:46:43.1093772Z mkdir -p /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/removing-code-and-incremental-lto/removing-code-and-incremental-lto
2019-12-19T04:46:43.1094169Z mkdir -p /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/removing-code-and-incremental-lto/removing-code-and-incremental-lto/incr
2019-12-19T04:46:43.1094619Z mkdir -p /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/removing-code-and-incremental-lto/removing-code-and-incremental-lto/rubble1
2019-12-19T04:46:43.1095063Z mkdir -p /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/removing-code-and-incremental-lto/removing-code-and-incremental-lto/rubble2
2019-12-19T04:46:43.1095776Z /checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc --crate-name cortex_m_rt cortex-m-rt.rs --crate-type lib --emit=metadata,link -C opt-level=s --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/removing-code-and-incremental-lto/removing-code-and-incremental-lto --target thumbv6m-none-eabi -C link-arg=-Tlink.x.in -L .
2019-12-19T04:46:43.1096819Z /checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc --edition=2018 --crate-name nrf52810_hal nrf52810-hal.rs --crate-type lib --emit=metadata,link -C opt-level=s -C metadata=aa86958b67bf89f5 --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/removing-code-and-incremental-lto/removing-code-and-incremental-lto --target thumbv6m-none-eabi --extern cortex_m_rt=/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/removing-code-and-incremental-lto/removing-code-and-incremental-lto/libcortex_m_rt.rmeta -C link-arg=-Tlink.x.in -L .
2019-12-19T04:46:43.1097675Z cp rubble.rs.v1 /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/removing-code-and-incremental-lto/removing-code-and-incremental-lto/rubble.rs
2019-12-19T04:46:43.1099506Z /checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc --crate-name rubble /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/removing-code-and-incremental-lto/removing-code-and-incremental-lto/rubble.rs --crate-type bin --emit=link -C opt-level=s --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/removing-code-and-incremental-lto/removing-code-and-incremental-lto/rubble1 --target thumbv6m-none-eabi -C incremental=/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/removing-code-and-incremental-lto/removing-code-and-incremental-lto/incr -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/removing-code-and-incremental-lto/removing-code-and-incremental-lto --extern nrf52810_hal=/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/removing-code-and-incremental-lto/removing-code-and-incremental-lto/libnrf52810_hal.rlib -C link-arg=-Tlink.x.in -L . -C linker-flavor=ld.lld -C codegen-units=2
2019-12-19T04:46:43.1100186Z Makefile:69: recipe for target 'all' failed
2019-12-19T04:46:43.1100484Z ------------------------------------------
2019-12-19T04:46:43.1100573Z stderr:
2019-12-19T04:46:43.1100972Z ------------------------------------------
2019-12-19T04:46:43.1100972Z ------------------------------------------
2019-12-19T04:46:43.1101066Z warning: unused variable: `submilli_micros`
2019-12-19T04:46:43.1101492Z   --> /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/removing-code-and-incremental-lto/removing-code-and-incremental-lto/rubble.rs:25:30
2019-12-19T04:46:43.1101623Z    |
2019-12-19T04:46:43.1101875Z 25 |                 let (millis, submilli_micros) = (self.0 / 1000, self.0 % 1000);
2019-12-19T04:46:43.1102202Z    |                              ^^^^^^^^^^^^^^^ help: consider prefixing with an underscore: `_submilli_micros`
2019-12-19T04:46:43.1102389Z    = note: `#[warn(unused_variables)]` on by default
2019-12-19T04:46:43.1102438Z 
2019-12-19T04:46:43.1102438Z 
2019-12-19T04:46:43.1102516Z error: linker `lld` not found
2019-12-19T04:46:43.1102672Z   = note: No such file or directory (os error 2)
2019-12-19T04:46:43.1102719Z 
2019-12-19T04:46:43.1102795Z error: aborting due to previous error
2019-12-19T04:46:43.1102839Z 
2019-12-19T04:46:43.1102839Z 
2019-12-19T04:46:43.1102913Z make: *** [all] Error 1
2019-12-19T04:46:43.1103225Z ------------------------------------------
2019-12-19T04:46:43.1103275Z 
2019-12-19T04:46:43.1103309Z 
2019-12-19T04:46:43.1103342Z 
2019-12-19T04:46:43.1103342Z 
2019-12-19T04:46:43.1103418Z failures:
2019-12-19T04:46:43.1103672Z     [run-make] run-make/removing-code-and-incremental-lto
2019-12-19T04:46:43.1104208Z test result: FAILED. 3 passed; 1 failed; 9 ignored; 0 measured; 0 filtered out
2019-12-19T04:46:43.1104291Z 
2019-12-19T04:46:43.1111462Z 
2019-12-19T04:46:43.1111561Z 
2019-12-19T04:46:43.1111561Z 
2019-12-19T04:46:43.1113773Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/thumbv6m-none-eabi/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/run-make" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make" "--stage-id" "stage2-thumbv6m-none-eabi" "--mode" "run-make" "--target" "thumbv6m-none-eabi" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/checkout/obj/build/x86_64-unknown-linux-gnu/llvm/build/bin/FileCheck" "--linker" "arm-none-eabi-gcc" "--host-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/thumbv6m-none-eabi/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--llvm-version" "9.0.0-rust-1.42.0-nightly\n" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
2019-12-19T04:46:43.1114486Z 
2019-12-19T04:46:43.1114523Z 
2019-12-19T04:46:43.1121990Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --target thumbv6m-none-eabi,thumbv7m-none-eabi,thumbv7em-none-eabi,thumbv7em-none-eabihf src/test/run-make
2019-12-19T04:46:43.1122148Z Build completed unsuccessfully in 1:00:09
2019-12-19T04:46:43.1122148Z Build completed unsuccessfully in 1:00:09
2019-12-19T04:46:43.1176122Z == clock drift check ==
2019-12-19T04:46:43.1195207Z   local time: Thu Dec 19 04:46:43 UTC 2019
2019-12-19T04:46:43.6495719Z   network time: Thu, 19 Dec 2019 04:46:43 GMT
2019-12-19T04:46:43.6496120Z == end clock drift check ==
2019-12-19T04:46:45.3349515Z 
2019-12-19T04:46:45.3453661Z ##[error]Bash exited with code '1'.
2019-12-19T04:46:45.3519537Z ##[section]Starting: Checkout
2019-12-19T04:46:45.3521688Z ==============================================================================
2019-12-19T04:46:45.3521780Z Task         : Get sources
2019-12-19T04:46:45.3521878Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@bors
Copy link
Contributor

bors commented Dec 19, 2019

💔 Test failed - checks-azure

@bors bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Dec 19, 2019
@Centril
Copy link
Contributor

Centril commented Dec 19, 2019

I would strongly recommend against rolling up things that touch run-make, run-*-fulldeps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
rollup A PR which is a rollup S-waiting-on-review Status: Awaiting review from the assignee but also interested parties.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants