Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rustdoc: refactor: centralize all command-line argument parsing #55515

Merged
merged 7 commits into from
Nov 5, 2018

Conversation

QuietMisdreavus
Copy link
Member

This is something i've wanted to do for a while, since we keep having to add new arguments to places like rust_input or core::run_core whenever we add a new CLI flag or the like. Those functions have inflated up to 11-19, and in some cases hiding away the locations where some CLI flags were being parsed, obscuring their use. Now, we have a central place where all command-line configuration occurs, including argument validation.

One note about the design: i grouped together all the arguments that html::render::run needed, so that i could pass them on from compilation in one lump instead of trying to thread through individual items or clone the entire blob ahead of time.

One other thing this adds is that rustdoc also now recognizes all the -Z options that rustc does, since we were manually grabbing a few previously. Now we parse a full DebuggingOptions struct and hand it directly to rustc when scraping docs.

@QuietMisdreavus
Copy link
Member Author

r? @GuillaumeGomez

@rust-highfive

This comment has been minimized.

@GuillaumeGomez
Copy link
Member

That simplifies the code a lot!

r=me once CI passed.

src/librustdoc/config.rs Outdated Show resolved Hide resolved
@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:199bc3a2:start=1540931077754171562,finish=1540931148860112008,duration=71105940446
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
---
[00:51:17] .................................................................................................... 100/4983
[00:51:20] .................................................................................................... 200/4983
[00:51:23] ...........................................................................................ii....... 300/4983
[00:51:26] .........................................................................................iii........ 400/4983
[00:51:29] iiiiiiii.iii...........................iii...........................................i...........i.. 500/4983
[00:51:36] .................................................................................................... 700/4983
[00:51:42] ..................................................................i...........i..................... 800/4983
[00:51:45] ....................................................................................iiiii........... 900/4983
[00:51:49] .................................................................................................... 1000/4983
---
[00:52:28] .................................................................................................... 2200/4983
[00:52:32] .................................................................................................... 2300/4983
[00:52:36] .................................................................................................... 2400/4983
[00:52:40] .................................................................................................... 2500/4983
[00:52:44] ...................................................................iiiiiiiii........................ 2600/4983
[00:52:51] ..................ii................................................................................ 2800/4983
[00:52:54] .................................................................................................... 2900/4983
[00:52:58] .................................................................................................... 3000/4983
[00:53:01] .............i...................................................................................... 3100/4983
---
travis_time:start:test_codegen
Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:06:30] 
[01:06:30] running 112 tests
[01:06:33] i..ii...iii.......i...i.........i..iii...........i.....i.....ii...i..i.ii..............i...ii..ii.i. 100/112
[01:06:34] test result: ok. 82 passed; 0 failed; 30 ignored; 0 measured; 0 filtered out
[01:06:34] 
[01:06:34]  finished in 3.612
[01:06:34] travis_fold:end:test_codegen
---
[01:46:33] 
[01:46:33] 12 3 | no
[01:46:33] 13   | ^^ not found in this scope
[01:46:33] 14 
[01:46:33] - thread '$DIR/failed-doctest-output.rs - OtherStruct (line 27)' panicked at 'couldn't compile the test', librustdoc/test.rs:332:13
[01:46:33] + thread '$DIR/failed-doctest-output.rs - OtherStruct (line 27)' panicked at 'couldn't compile the test', librustdoc/test.rs:323:13
[01:46:33] 17 
[01:46:33] 17 
[01:46:33] 18 ---- $DIR/failed-doctest-output.rs - SomeStruct (line 21) stdout ----
[01:46:33] 
[01:46:33] 21 thread 'main' panicked at 'oh no', $DIR/failed-doctest-output.rs:3:1
[01:46:33] 23 
[01:46:33] - ', librustdoc/test.rs:367:17
[01:46:33] + ', librustdoc/test.rs:358:17
[01:46:33] 25 
[01:46:33] 25 
[01:46:33] 26 
[01:46:33] 27 failures:
[01:46:33] 
[01:46:33] 
[01:46:33] The actual stdout differed from the expected stdout.
[01:46:33] Actual stdout saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/failed-doctest-output/failed-doctest-output.stdout
[01:46:33] To update references, rerun the tests and pass the `--bless` flag
[01:46:33] To only update this specific test, also pass `--test-args failed-doctest-output.rs`
[01:46:33] error: 1 errors occurred comparing output.
[01:46:33] status: exit code: 101
[01:46:33] status: exit code: 101
[01:46:33] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustdoc" "/checkout/src/test/rustdoc-ui/failed-doctest-output.rs" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/failed-doctest-output/a" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--test" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/failed-doctest-output/auxiliary"
[01:46:33] ------------------------------------------
[01:46:33] 
[01:46:33] running 2 tests
[01:46:33] test /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 27) ... FAILED
[01:46:33] test /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 27) ... FAILED
[01:46:33] test /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - SomeStruct (line 21) ... FAILED
[01:46:33] failures:
[01:46:33] 
[01:46:33] ---- /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 27) stdout ----
[01:46:33] ---- /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 27) stdout ----
[01:46:33] error[E0425]: cannot find value `no` in this scope
[01:46:33]   |
[01:46:33] 3 | no
[01:46:33]   | ^^ not found in this scope
[01:46:33] 
[01:46:33] 
[01:46:33] thread '/checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 27)' panicked at 'couldn't compile the test', librustdoc/test.rs:323:13
[01:46:33] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[01:46:33] 
[01:46:33] ---- /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - SomeStruct (line 21) stdout ----
[01:46:33] thread '/checkout/src/test/rustdoc-ui/failed-doctest-output.rs - SomeStruct (line 21)' panicked at 'test executable failed:
[01:46:33] 
[01:46:33] thread 'main' panicked at 'oh no', /checkout/src/test/rustdoc-ui/failed-doctest-output.rs:3:1
[01:46:33] 
[01:46:33] ', librustdoc/test.rs:358:17
[01:46:33] 
[01:46:33] 
[01:46:33] 
[01:46:33] failures:
[01:46:33]     /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 27)
[01:46:33]     /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - SomeStruct (line 21)
[01:46:33] test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out
[01:46:33] 
[01:46:33] 
[01:46:33] ------------------------------------------
---
[01:46:33] test result: FAILED. 7 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
[01:46:33] 
[01:46:33] 
[01:46:33] 
[01:46:33] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--rustdoc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustdoc" "--src-base" "/checkout/src/test/rustdoc-ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "ui" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-5.0/bin/FileCheck" "--host-rustcflags" "-Zunstable-options " "--target-rustcflags" "-Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "5.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
[01:46:33] 
[01:46:33] 
[01:46:33] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:46:33] Build completed unsuccessfully in 0:59:06
[01:46:33] Build completed unsuccessfully in 0:59:06
[01:46:33] Makefile:58: recipe for target 'check' failed
[01:46:33] make: *** [check] Error 1

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:15961758
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
---
travis_time:end:152e5fbc:start=1540937563897119088,finish=1540937563903988632,duration=6869544
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:086a87cc
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@QuietMisdreavus
Copy link
Member Author

@bors r=GuillaumeGomez

@bors
Copy link
Contributor

bors commented Oct 31, 2018

📌 Commit bdb8cb4553f914134e57c73197f3b34aa8ac14a6 has been approved by GuillaumeGomez

@bors bors added the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label Oct 31, 2018
@bors
Copy link
Contributor

bors commented Nov 2, 2018

☔ The latest upstream changes (presumably #54543) made this pull request unmergeable. Please resolve the merge conflicts.

@bors bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Nov 2, 2018
@QuietMisdreavus
Copy link
Member Author

Well that was an intense rebase. @GuillaumeGomez can you take another look? I had to change around html::render::{run, write_shared} a little more, to make sure that markdown::render could be called properly. I moved (enable_)index_page out of the Context, since i had to put them in RenderOptions anyway, and i changed markdown::render to take the input file separately so it could take a RenderOptions instead. To make that work, i moved the markdown-specific items into RenderOptions, too. I ran tests locally and everything seems to still work.

@QuietMisdreavus QuietMisdreavus added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Nov 2, 2018
for &flag in removed_flags.iter() {
if matches.opt_present(flag) {
diag.struct_warn(&format!("the '{}' flag no longer functions", flag))
.warn("see CVE-2018-1000622")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All removed flags are linked to this CVE?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both of these flags are, yes. They showed a similar message that was printed with eprintln!(), which was being checked in rust_input.

@QuietMisdreavus
Copy link
Member Author

Updated!

@GuillaumeGomez
Copy link
Member

Thanks!

@bors: r+

@bors
Copy link
Contributor

bors commented Nov 5, 2018

📌 Commit 560a01c has been approved by GuillaumeGomez

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 5, 2018
@bors
Copy link
Contributor

bors commented Nov 5, 2018

⌛ Testing commit 560a01c with merge 6cfc603...

bors added a commit that referenced this pull request Nov 5, 2018
rustdoc: refactor: centralize all command-line argument parsing

This is something i've wanted to do for a while, since we keep having to add new arguments to places like `rust_input` or `core::run_core` whenever we add a new CLI flag or the like. Those functions have inflated up to 11-19, and in some cases hiding away the locations where some CLI flags were being parsed, obscuring their use. Now, we have a central place where all command-line configuration occurs, including argument validation.

One note about the design: i grouped together all the arguments that `html::render::run` needed, so that i could pass them on from compilation in one lump instead of trying to thread through individual items or clone the entire blob ahead of time.

One other thing this adds is that rustdoc also now recognizes all the `-Z` options that rustc does, since we were manually grabbing a few previously. Now we parse a full `DebuggingOptions` struct and hand it directly to rustc when scraping docs.
@bors
Copy link
Contributor

bors commented Nov 5, 2018

☀️ Test successful - status-appveyor, status-travis
Approved by: GuillaumeGomez
Pushing 6cfc603 to master...

@bors bors merged commit 560a01c into rust-lang:master Nov 5, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants