Update project load tracking to include experimental parser info

[kwigley]

resolves #3438

Description

I reset this branch several times while trying to add this data, this is what I landed on that doesn't involve refactoring pieces of partial parsing. Will pull logic in to 0.20.latest branch after approval.

• parsed_path_count -> path_count can represent the size of the project and amount of files eligible to be parsed while parsed_path_count will be the actual amount of files parsed
• is_static_analysis_enabled -> set if flag for experimental parser is turned on
• static_analysis_path_count -> number of files eligible to be parsed by experimental parser
• static_analysis_parsed_path_count -> number of files actually parsed by experimental parser

Checklist

• I have signed the CLA
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have updated the CHANGELOG.md and added information about my change to the "dbt next" section.

[jtcohen6]

• Event looks good, I confirmed the data flows all the way through as expected
• The updates to path_count + parsed_path_count are right on
• We're undercounting for static_analysis_path_count + static_analysis_parsed_path_count because of how we're using the node.meta property right now. Let's find another way to do this

[jtcohen6]

Could we add is_static_analysis_enabled to the output of dbt parse (target/perf_info.json) as well?

[jtcohen6]

I don't think we can use meta for this. That's a property that users can set in yaml files (docs). If the user sets any yaml properties for the node, even if they don't set any meta properties, the node drops {'is_statically_extractable': True} during yaml parsing here. The upshot is that we significantly undercount the number of statically extractable + extracted paths.

Is there a different preexisting, internal-use node property we could set here? Or another way to tag the node (without adding having to adjust the manifest schema)?

I'd also be open to adjusting the v2 manifest schema so that, instead of just adding a standalone created_at property, we added a parsing dictionary (which could include created_at, statically_extractable, statically_extracted, total parse time, etc).

[gshank]

One option might be to store the information in an attribute/dictionary that is filtered out in serialization so that we don't need to update the manifest schema. I couldn't do that with 'created_at' because it had to be in the partial_parse.msgpack file, but I think the information we're discussion here doesn't need to be stored.

I suspect that we're going to keep running into this "new attribute vs updating schema" issue, so we might want to think about something like the 'parsing' dictionary Jeremy proposes, maybe 'internal' or something, where we can put data that probably is not relevant except internally.

[gshank]

Oh, I see Kyle already is filtering out that info. Nevermind :)

[gshank]

Did we decide not to track the path counts on a project/parser basis? I see the total path_count is saved from the number of files.

Also we had discussed storing whether partial parsing is actually running ('partially_parsing'). Is that something we want to save?

[jtcohen6]

Also we had discussed storing whether partial parsing is actually running ('partially_parsing'). Is that something we want to save?

@gshank I think we can infer this from whether path_count != parsed_path_count

[jtcohen6]

Nice work! Numbers are looking right on.