Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][transform-v2] Support transform metrics #8173

Open
wants to merge 71 commits into
base: dev
Choose a base branch
from

Conversation

CosmosNi
Copy link
Contributor

@CosmosNi CosmosNi commented Nov 29, 2024

close #8172


env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  # This is a example source plugin **only for test and demonstrate the feature source plugin**
  FakeSource {
      plugin_output = "fake1"
      schema = {
        table = "fake.table1"
        fields {
          id = bigint
          name = string
          score = int
        }
      }
      rows = [
        {
          kind = INSERT
          fields = [1, "A", 100]
        },
        {
          kind = INSERT
          fields = [2, "B", 100]
        },
        {
          kind = INSERT
          fields = [3, "C", 100]
        },
        {
          kind = INSERT
          fields = [3, "C", 100]
        },
        {
          kind = INSERT
          fields = [3, "C", 100]
        },
        {
          kind = INSERT
          fields = [3, "C", 100]
        }
        {
          kind = UPDATE_BEFORE
          fields = [1, "A", 100]
        },
        {
          kind = UPDATE_AFTER
          fields = [1, "A", 300]
        },
        {
          kind = DELETE
          fields = [2, "B", 100]
        },
                 {
                   kind = INSERT
                   fields = [2, "B", 100]
                 }
      ]
    }

    FakeSource {
        plugin_output = "fake2"
        schema = {
          table = "fake.public.table2"
          fields {
            id = bigint
            name = string
            score = int
          }
        }
        rows = [
          {
            kind = INSERT
            fields = [1, "A", 100]
          },
          {
            kind = INSERT
            fields = [2, "B", 100]
          },
          {
            kind = DELETE
            fields = [2, "B", 100]
          },
          {
            kind = INSERT
            fields = [3, "C", 100]
          },
          {
            kind = INSERT
            fields = [3, "C", 100]
          }
        ]
      }

}

transform {
  Sql {
    plugin_input = "fake1"
    plugin_output = "fake3"
    query = "select * from fake1"
  }
    Sql {
      plugin_input = "fake3"
      plugin_output = "fake4"
      query = "select * from fake3"
    }
    Sql {
      plugin_input = "fake4"
      plugin_output = "fake5"
      query = "select * from fake4"
    }
  Sql {
    plugin_input = "fake2"
    plugin_output = "fake6"
    query = "select * from fake2"
  }
    Sql {
      plugin_input = "fake6"
      plugin_output = "fake7"
      query = "select * from fake6"
    }
    Sql {
      plugin_input = "fake7"
      plugin_output = "fake8"
      query = "select * from fake7"
    }

}

sink {
  console {
    plugin_input = "fake5"
  }
  console {
    plugin_input = "fake8"
  }
}

flink/zeta log results:
image

rest:
image

multi tables:
image

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

@github-actions github-actions bot added core SeaTunnel core module Zeta transform-v2 api labels Nov 29, 2024
@CosmosNi
Copy link
Contributor Author

@hailin0 @Hisoka-X Temporarily implemented the metric of count.PTAL,see see if there is any problem

Copy link
Contributor

@corgy-w corgy-w left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've had a quick look, and I have a few minor questions

I think that is the process now.
image

ps: If you have time please make sure the http api returns new metrics as well. and add test cases

…_transform_metrics

# Conflicts:
#	seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/fieldmapper/FieldMapperTransform.java
@corgy-w
Copy link
Contributor

corgy-w commented Dec 3, 2024

@CosmosNi Looks good, Wait ci let @Hisoka-X take a look

@github-actions github-actions bot removed the document label Dec 5, 2024
@github-actions github-actions bot added the CI&CD label Dec 5, 2024
Copy link
Contributor

@corgy-w corgy-w left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

format

@github-actions github-actions bot removed the document label Jan 22, 2025
@github-actions github-actions bot removed the document label Jan 22, 2025
@CosmosNi
Copy link
Contributor Author

@Hisoka-X PTAL

@CosmosNi CosmosNi requested a review from Hisoka-X January 24, 2025 01:08
@CosmosNi CosmosNi force-pushed the feature_transform_metrics branch from 78e032f to 48c6e7d Compare January 24, 2025 08:00
@CosmosNi CosmosNi force-pushed the feature_transform_metrics branch from 48c6e7d to af224b1 Compare January 24, 2025 08:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature][transform] support transform metrics
3 participants