feat(vdp): upload raw inputs for run log #904
Conversation
| @@ -1064,36 +1103,44 @@ func (s *service) preTriggerPipeline(ctx context.Context, ns resource.Namespace, | |||
| if err != nil { | |||
| return err | |||
| } | |||
| uploadingPipelineData[idx][k] = v.GetStringValue() | |||
There was a problem hiding this comment.
@chuang8511
The URL might still be a dataURI (base64). If the user continues using dataURI or base64 in the input, we should save it to our blob storage and replace it with the corresponding blob URL. Could you check if this change can be implemented quickly? I guess we can modify lines L868–L907 to handle this.
There was a problem hiding this comment.
The URL might still be a dataURI (base64). If the user continues using dataURI or base64 in the input
Do you mean when the users call API to trigger pipeline but not from Console?
There was a problem hiding this comment.
Yes, that’s one of the possibilities.
There was a problem hiding this comment.
fixed in this commit.
| variable[k] = data.NewNull() | ||
| uploadingPipelineData[idx][k] = nil |
There was a problem hiding this comment.
Small question: what will be the difference between variable and uploadingPipelineData[idx] at the end of each iteration (besides the type data.Map and map[string]any, I mean in terms of data)? If it's the same, could we transform one to the other (I think map[string]any -> data.Map would be easiest but I haven't checked the details) at the end of the iteration and handle a single variable along the way?
There was a problem hiding this comment.
When the format is unstructured data, the data will be different.
e.g. format image
- variable -> will be blob data
- uploadingPipelineData[idx] -> will be the public URL that the console / users can access to.
Originally, I plan to write this part of code by
- Loop to set
uploadingPipelineData. - Upload data
- Loop to set
variable
But, I think the current way is easier, so I did not choose my original plan.
There was a problem hiding this comment.
Yep my suggestion was only if the data was the same. The function is quite long so I missed these small details 😅 Thanks for the clarification, in that case the code makes sense as it is (we can work on it later to make it more readable and compact).
There was a problem hiding this comment.
recorded in ins-7071
🤖 I have created a release *beep* *boop* --- ## [0.48.4-beta](v0.48.3-beta...v0.48.4-beta) (2024-12-09) ### Features * **pinecone:** Add rerank task for Pinecone component ([#773](#773)) ([e1fd611](e1fd611)) * **vdp:** upload raw inputs for run log ([#904](#904)) ([960f4c2](960f4c2)) ### Bug Fixes * **component,http:** fix the request body marshalling ([#928](#928)) ([b47cc71](b47cc71)) ### Miscellaneous Chores * release v0.48.4-beta ([b08878e](b08878e)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
🤖 I have created a release *beep* *boop* --- ## [0.48.4-beta](v0.48.3-beta...v0.48.4-beta) (2024-12-09) ### Features * **pinecone:** Add rerank task for Pinecone component ([#773](#773)) ([e1fd611](e1fd611)) * **vdp:** upload raw inputs for run log ([#904](#904)) ([960f4c2](960f4c2)) ### Bug Fixes * **component,http:** fix the request body marshalling ([#928](#928)) ([b47cc71](b47cc71)) ### Miscellaneous Chores * release v0.48.4-beta ([b08878e](b08878e)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Because
This commit