fileViewer: Create a directory view to navigate a report

[marco-c]

No description provided.

[marco-c]

Depends on mozilla/release-services#1161.

[klahnakoski]

There was a previous project that worked on this: https://github.com/mozilla/moz-coco/blob/master/src/queries/10-DirectoryDrillDown-Cov.js

It used the coverage-summary table (currently not updated) to get the coverage stats by directory. A similar query could run off of of coverage directly once the TUIDs are in place: Requesting the number of unique TUIDs covered, and uncovered, for each child-directory. Much like the ActiveDataRecipies do now, but with source files:
https://github.com/ahal/active-data-recipes/blob/d0c7350bfbfcbf2ebf84701bb55ca182cd089f39/adr/queries/code_coverage.query#L12

[jankeromnes]

On the front-end, this could be achieved by hooking into these lines:

firefox-code-coverage-frontend/src/containers/fileViewer.jsx

Lines 88 to 89 in 2695903

	{ !appErr && (parsedFile) &&
	<FileViewer {...this.state} onLineClick={this.setSelectedLine} /> }

and implementing a new DirectoryViewer component a bit similar to the FileViewer component:

firefox-code-coverage-frontend/src/containers/fileViewer.jsx

Lines 100 to 121 in 2695903

	// This component renders each line of the file with its line number
	const FileViewer = ({
	parsedFile, coverage, selectedLine, onLineClick,
	}) => (
	<table className="file-view-table">
	<tbody>
	{parsedFile.map((text, lineNumber) => {
	const uniqueId = hash(text) + lineNumber;
	return (
	<Line
	key={uniqueId}
	lineNumber={lineNumber + 1}
	text={text}
	coverage={coverage}
	selectedLine={selectedLine}
	onLineClick={onLineClick}
	/>
	);
	})}
	</tbody>
	</table>
	);

with table headers like, e.g.:

<tr>
  <th>Sub-Directory/File</th>
  <th>Coverage</th>
</tr>

[jankeromnes]

And the back-end can provide a flat view of coverage for all files under a certain path, like so:

https://activedata.allizom.org/tools/query.html#query_id=R3XCygfE

Maybe the front-end can calculate per-immediate-subdirectory coverage by aggregating the coverage of all files under a given sub-directory.

Or, if that's too compute-intensive on the client-side, maybe we could build another query that already groups by sub-directory instead of returning all the files recursively.

[klahnakoski]

There may be easier ways to do this:

{
	"from":"coverage",
	"groupby":[{
		"name":"subdir",
		"value":{
			"when":{"find":{"source.file.name":"/"},"start":5},
			"then":{"between":{"source.file.name":[5,"/"]}},
			"else":{"not_left":{"source.file.name":5}}
		}
	}],
	"where":{"and":[
		{"prefix":{"source.file.name":"mfbt/"}},
		{"eq":{"repo.changeset.id12":"96f7c7e518ce"}}
	]}
}

https://activedata.allizom.org/tools/query.html#query_id=6e8tvR7z

The groupby clause is parsing the file names to get the immediate children. The 5 is the length of "mfbt/". This query returns nothing if the "directory" is a file. The new cluster will make the query simpler because between can accept expressions on all parameters, not just constants.

[marco-c]

The backend could also retrieve the list of directories from hgmo if it's not easy/clean to do it via a query.

[jankeromnes]

https://activedata.allizom.org/tools/query.html#query_id=6e8tvR7z

Thank you for this query!

So I see that it returns:

    "data":[
        ["double-conversion",37],
        ["decimal",12],
        ["GuardObjects.h",6],
        ["MathAlgorithms.h",6],
        ["RefPtr.h",6],
        ["ThreadLocal.h",6],
        ["Vector.h",6],
        ["AlreadyAddRefed.h",5],
        ["Casting.h",5],
        ["DoublyLinkedList.h",5],
        [null,310]
    ]

How should I interpret this data to get a code coverage percentage?

Additionally, codecov.io seems to have a lot more coverage data for mfbt/. Why are we missing so many files in ActiveData? Could the [null,310] entry be a symptom of the underlying problem?

[klahnakoski]

Sorry, that query was not meant to provide all the data you want, just to show we can do some string manipulation to get "directories" and files. Furthermore, it looks like it is incomplete; I missed the [null,310] line; and you are correct to assume it is showing all the records my groupby logic is not catching. Doh!

[klahnakoski]

We must add a select clause to sum the total_covered and total_uncovered to calculate the overall percent.

[jankeromnes]

@klahnakoski Thanks! I'll try that (but please feel free to post the amended query here if you have it mind, in case I haven't managed to guess it yet).

[jankeromnes]

Ok, I've gotten to this:

	"select":[
		"subdir",
		{
			"name":"total_covered",
			"value":"source.file.total_covered",
			"aggregate":"sum"
		},
		{
			"name":"total_uncovered",
			"value":"source.file.total_uncovered",
			"aggregate":"sum"
		}
	]

https://activedata.allizom.org/tools/query.html#query_id=_dYkLjYt

The result looks like this:

    "data":[
        ["double-conversion",null,83621,110451],
        ["decimal",null,15961,53534],
        ["RefPtr.h",null,25156,6746],
        ["ThreadLocal.h",null,2228,490],
        ["UniquePtrExtensions.h",null,559,38],
        ["AllocPolicy.h",null,1972,365],
        ["ArrayUtils.h",null,1889,40],
        ["BinarySearch.h",null,3992,782],
        ["GuardObjects.h",null,2316,0],
        ["Variant.h",null,17227,5582],
        [null,null,558819,393875]
    ]

Not sure where the second column comes from (it's all null), and it looks like our bogus null subdir from last time is still there.

[klahnakoski]

The groupby clause performs grouping, but also adds a column to the resultset. The "select": <list> works like SQL; it adds a column for each item found in list. Neither clause can refer to the columns of another; nested queries are need if you want to do that.

{"select":"subdir"}

is asking for the "subdir" column in the coverage table, which there is none, so it is null.

[klahnakoski]

This query adds a is_dir column

https://activedata.allizom.org/tools/query.html#query_id=SAVvrgte

[klahnakoski]

Always add a limit clause to be explicit about the number of records you want, otherwise you get 10.

https://activedata.allizom.org/tools/query.html#query_id=teAuwVKw

[jankeromnes]

Awesome, thanks @klahnakoski!

[klahnakoski]

The one thing that concerns me is the last row, the "null row", which catches everything not not covered by the groupby, but matches the where clause; I am looking into what those records are now; I use the etl.id to find examples and figure out how this query is not reporting them correctly.

https://activedata.allizom.org/tools/query.html#query_id=KoJZ8osX

[klahnakoski]

... but the last row is simply the sum of the others, which suggests the ES query responsible for counting the null row is wrong. Ignore it for now, and I will push a fix to Activedata

[klahnakoski]

mozilla/ActiveData#63

[jankeromnes]

Note: Since the example revision 752465b44c79 no longer returns any data, here is a query that gives a list of revisions for which the per-directory query should have some data:

https://activedata.allizom.org/tools/query.html#query_id=2lq0AhNB

[La0]

I integrated a new endpoint into the code coverage backend, using the ES cluster.
You can browse files/directories at https://coverage.staging.moz.tools/v2/path (staging only until next services release)

To view a sub-directory/file: https://coverage.staging.moz.tools/v2/path?path=js/src/

You can also specify another changeset by using the changeset parameter

[marco-c]

@jankeromnes could you make the frontend use the staging endpoint just for this for now?

[jankeromnes]

Awesome, thanks! I've cleaned up my prototype and will try making it use the new staging endpoint.