Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Subtask] Complete the metadata and interface design for messaging type catalog #2371

Closed
Tracked by #2369
mchades opened this issue Feb 27, 2024 · 4 comments · Fixed by #2444
Closed
Tracked by #2369

[Subtask] Complete the metadata and interface design for messaging type catalog #2371

mchades opened this issue Feb 27, 2024 · 4 comments · Fixed by #2444
Assignees
Labels
subtask Subtasks of umbrella issue

Comments

@mchades
Copy link
Contributor

mchades commented Feb 27, 2024

Describe the subtask

This issue tracks the design of the messaging catalog

Parent issue

#2369

@mchades mchades added the subtask Subtasks of umbrella issue label Feb 27, 2024
@mchades mchades added this to the Gravitino 0.5.0 milestone Feb 27, 2024
@mchades mchades self-assigned this Feb 27, 2024
@mchades
Copy link
Contributor Author

mchades commented Feb 27, 2024

Here is the initial design document:

https://docs.google.com/document/d/1NZKbM4e8iEMB0BOYD4nyzNhb3znGhxBXOE7OBx6_0zY/edit?usp=sharing

@mchades mchades changed the title [Subtask] Complete the metadata and interface design for file type catalog [Subtask] Complete the metadata and interface design for stream type catalog Feb 27, 2024
@YxAc
Copy link
Contributor

YxAc commented Feb 28, 2024

Here is the initial design document:

https://docs.google.com/document/d/1NZKbM4e8iEMB0BOYD4nyzNhb3znGhxBXOE7OBx6_0zY/edit?usp=sharing

I have some comments and pls take a look when you are free. @mchades

Can you spare some time to join the discussion? @SteNicholas , thx.

@mchades
Copy link
Contributor Author

mchades commented Mar 5, 2024

Given the complexity of message schema management, supporting message schemas in Gravitino is a substantial task.

Therefore, we do not intend to support schema management in the 0.5.0 version of the stream catalog.

If you have any requirements or scenarios regarding schema management through the Gravitino stream catalog, feel free to leave comments and engage in discussions on the relevant issue.

Below is the design proposal for the 0.5.0 Gravitino stream catalog.
https://docs.google.com/document/d/1B1DTf_CR_wIqvEBLLfytUZ27-VGsdv-pZqsRkA4iANQ/edit?usp=sharing

@mchades mchades changed the title [Subtask] Complete the metadata and interface design for stream type catalog [Subtask] Complete the metadata and interface design for messaging type catalog Mar 7, 2024
@mchades
Copy link
Contributor Author

mchades commented Mar 7, 2024

Regarding "stream", users have provided feedback that they more easily associate it with batch processing, which is primarily used for data handling.

After discussion, we have decided to change the catalog type to "Messaging", referring to the type of data managed by this catalog.

jerryshao pushed a commit that referenced this issue Mar 8, 2024
…ing catalog (#2444)

### What changes were proposed in this pull request?

This PR proposes to add the `Topic` metadata support in Gravitino.

Include the API and metadata module of `Topic`.

### Why are the changes needed?

This is the first work of support MQ management in Gravitino.

Fix: #2371 

### Does this PR introduce _any_ user-facing change?

Yes

### How was this patch tested?

Add some UTs to test metadata, API test will be added when the
implementation is achieved.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
subtask Subtasks of umbrella issue
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants