Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CT 1644 node cleanup #6427

Merged
merged 17 commits into from
Dec 13, 2022
Merged

CT 1644 node cleanup #6427

merged 17 commits into from
Dec 13, 2022

Conversation

gshank
Copy link
Contributor

@gshank gshank commented Dec 12, 2022

resolves #6426

Description

This pull request contains a lot of cleanup of the manifest nodes.

  • Refactor the class hierarchies in core/dbt/contracts/graph/nodes.py so that it's easier to read and understand the hierarchy. Give all nodes the same base class.
  • Remove unneeded compilation and SQL node attributes from SeedNode by separating out the attributes that are needed for all ManifestNodes from those that are needed for compilable nodes.
  • Regularize the Unions that are used to refer to various categories of Nodes (and node-like objects)
  • Move "relation_name" from compile time to parse time.
  • Reduce the reliance on base classes defined in unparsed.py
  • Remove "tags" attributes from macros (unused and unneeded)
  • Add "resource_type" to Documentation nodes for consistency
  • Use consistent unique_ids for Documentation nodes (i.e. starting with "doc."
  • Make setting the resource_type more consistent

Note: core/dbt/contracts/graph/nodes.py has changed so much that I don't recommend trying to read the diff. Reviewers should read the changed file instead.

Checklist

@gshank gshank requested a review from a team as a code owner December 12, 2022 02:41
@gshank gshank requested a review from a team December 12, 2022 02:41
@gshank gshank requested review from a team as code owners December 12, 2022 02:41
@cla-bot cla-bot bot added the cla:yes label Dec 12, 2022
@github-actions
Copy link
Contributor

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

@gshank gshank requested review from peterallenwebb and removed request for iknox-fa and McKnight-42 December 12, 2022 02:48
@gshank gshank force-pushed the ct-1644-node_cleanup branch from 4580704 to a27f2d2 Compare December 12, 2022 14:58
Copy link
Member

@emmyoop emmyoop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than the one comment, looks good.

Comment on lines 258 to +260
if not isinstance(node, (ParsedNode)):
raise InternalException(f"type mismatch, expected ParsedNode but got {type(node)}")
raise InternalException(
f"type mismatch, expected ManifestNode but got {type(node)}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This error and the check don't match.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They're actually the same thing. Mypy wouldn't let me use ManifestNode in an isinstance check. I'd rather not use ParsedNode in the message, because I think using the superclasses is more confusing...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe just a comment in the code to that effect then?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In any case, it's really not a user-facing exception. It should only show up when a developer has made a mistake :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I just want to avoid spending 5 minutes wondering why these don't match next time I'm in there 😄

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment "# Can't use ManifestNode here because of parameterized generics" wasn't enough?

Comment on lines +62 to +77
# =====================================================================
# This contains the classes for all of the nodes and node-like objects
# in the manifest. In the "nodes" dictionary of the manifest we find
# all of the objects in the ManifestNode union below. In addition the
# manifest contains "macros", "sources", "metrics", "exposures", "docs",
# and "disabled" dictionaries.
#
# The SeedNode is a ManifestNode, but can't be compiled because it has
# no SQL.
#
# All objects defined in this file should have BaseNode as a parent
# class.
#
# The two objects which do not show up in the DAG are Macro and
# Documentation.
# =====================================================================
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[CT-1644] Cleanup node classes as part of protobuf node generation
3 participants