Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

misc(runner): add assertion for devtoolsLog as requiredArtifact #9290

Merged
merged 2 commits into from
Jun 27, 2019
+24 −4
Merged

misc(runner): add assertion for devtoolsLog as requiredArtifact #9290

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f235a32
core(runner): add assertion for devtoolsLog as requiredArtifact
brendankenny Jun 27, 2019
6f58f43
-TODO
brendankenny Jun 27, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions lighthouse-core/runner.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -289,11 +289,13 @@ class Runner {
for (const artifactName of audit.meta.requiredArtifacts) {
const noArtifact = artifacts[artifactName] === undefined;

// If trace required, check that DEFAULT_PASS trace exists.
// TODO: need pass-specific check of networkRecords and traces.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

networkRecords! This is old :)

const noTrace = artifactName === 'traces' && !artifacts.traces[Audit.DEFAULT_PASS];
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm I wasn't even aware we did this, I guess it's impossible to eliminate the defaultPass then even with a custom config if you want any network or trace data?

// If trace/devtoolsLog required, check that DEFAULT_PASS trace/devtoolsLog exists.
// TODO: need pass-specific check of traces and devtoolsLogs.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would personally just get rid of this TODO, are we really going to move to requiredArtifacts: ['traces.mySpecialPass'] at any point in the near future?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would personally just get rid of this TODO, are we really going to move to requiredArtifacts: ['traces.mySpecialPass'] at any point in the near future?

I think we're going to run into this sooner or later for the custom config/plugin reason you mention above. I think there are solutions, e.g. we could use a tracePassName (or whatever) convention in the audit options that defaults to defaultPass and check based on that here. That has its own pretty serious issues (we need to check requiredArtifacts and audit options now? what if an audit doesn't want to/can't follow the convention? etc), but I'd like to keep thinking about it.

Main thing is that if/when that situation does come up, this is where the error will occur, so it would be good to have a comment on it. I can definitely make it not a TODO, though.

const noRequiredTrace = artifactName === 'traces' && !artifacts.traces[Audit.DEFAULT_PASS];
const noRequiredDevtoolsLog = artifactName === 'devtoolsLogs' &&
!artifacts.devtoolsLogs[Audit.DEFAULT_PASS];

if (noArtifact || noTrace) {
if (noArtifact || noRequiredTrace || noRequiredDevtoolsLog) {
log.warn('Runner',
`${artifactName} gatherer, required by audit ${audit.meta.id}, did not run.`);
throw new Error(`Required ${artifactName} gatherer did not run.`);
Expand Down
18 changes: 18 additions & 0 deletions lighthouse-core/test/runner-test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,24 @@ describe('Runner', () => {
});
});

it('outputs an error audit result when devtoolsLog required but not provided', async () => {
const config = new Config({
settings: {
auditMode: __dirname + '/fixtures/artifacts/empty-artifacts/',
},
audits: [
// requires devtoolsLogs[Audit.DEFAULT_PASS]
'is-on-https',
],
});

const results = await Runner.run({}, {config});
const auditResult = results.lhr.audits['is-on-https'];
assert.strictEqual(auditResult.score, null);
assert.strictEqual(auditResult.scoreDisplayMode, 'error');
assert.strictEqual(auditResult.errorMessage, 'Required devtoolsLogs gatherer did not run.');
});

it('outputs an error audit result when missing a required artifact', () => {
const config = new Config({
settings: {
Expand Down