ISSUE-91:ML processors #92
Conversation
ML processors use a IIIF URL, so no need to download the file at all
Abstract bc it can't be used directly. Because each ML model is quite opinionated. Full of bugs, lacks of checks but it works. Will keep refining and trying to make it elegant before merging
Generates the proper embeddings and those end properly in Solr
But it will. Basically i can't pass an Image URL and a bounding box via GET (damn Drupal 10.2) but! i can gzip and base64 encode. I will that next. The idea here is that this filter can be fed via "ajax" by another view/formatter/plugin with a JSON structure that i will then decode here. I will check, if no bounding box was given, if i have a Vector for the same type configured in Solr and use that one, if not, i will call the corresponding API on the NLP container and the generate a vector on the fly (well like 3 seconds at least) and then alter the query to do a KNN... cool stuff here
Basically preparing for re-use inside a Views Filter. That way we can ensure we search with the same ML model/vector size that generated the "against what we are searching for"
and expand the interface public methods of the runnerUtility Service (just bc it aids in autocompleting/validating/overriding if needed/ever in the future)
…gins that are not "top level"... because we might end needing deeper level listings for external access of the method
Because vectors are so specific. And we don't want white screens of sorts like e.g "solr died" because it felt lonely, tired who knows.
+ match the description users would read, I think nobody is into vectors so better so
Simpler than multiple if/then/what/else/who-cares situation
So, now that we are actually - Fetching an image - Allowing a region - calling the processor on the backend to generate vector (and object detection... will be useful in the future too) We need to start thinking of query/alter via tags. Similar to what we do with the OCR one. But there are many things here we will have to figure out... like how does KNN interact with other ones? The prequery v/s the post query and how we will deal with facets. But baby steps first. We need to add a new query option, alter/act on it and see if !KNN works first
(strawberryfield level, since that module provides the knn fields) the query and join/etc the weird magic
Plus the filter processing via exposed filters (which is indeed a different plugin and the one i really need for interactive filtering)
… similar! @alliomeria (*since i did not pinged you before) I think that for exposed ML arguments instead of passing a JSON i will only allow a UUID of an existing file + a fragment selector so UUID#xywh=percent:{$left},{$top},{$width},{$height}" Why? The argument is shorter/easier do decode... also will basically remove the creepness of passing JSON/and s3:// Adds though an extra file load, but that adds security (ask me if you want tomorrow why an Contextual filter, and i will answer)
This filter could also allow a "hide" option... so basically only to be set via URL arguments ... right? @alliomeria right?
pack("*c" and stuff)
Processors are getting cleaner (not there yet) and Image Argument filter less beef-fy
One annoying thing is that because how i built Runners a runner can not generate (without a pager) multiple Flavors. Insightface so far is the one where embeddings have a closer alignment with the logic of detecting a face. The main features of a faces are encoded in the Vector which in test allowed me to even get family members (tested with myself and mom!) but also means that if a single image has multiple detections i need multiple Vectors, and Solr allows on vector per field/per document. Will explore the one to many option in our code @alliomeria
to the output so we can chain
Todo ask @alliomeria and the community about "empty completeness" v/s "absence of data" (first implies normalization, e.g doc count. == processed count), the second slimmer archipelago
We want people to know this will be possible, but for now it is easier to treat all filters are pre-queries (as we do)
Also don't state it is YOLOv8, in the future (near) it might be yolov10, yolov11, etc.
Also: Need to document this \Drupal\strawberry_runners\Plugin\views\filter\StrawberryRunnersMLImagefilter::IMAGEML_INPUT_SCHEMA so people testing know what the expected SHAPE of the JSON is. Again. This filter is just a way of exposing values. Programmatically (user/dev needs to know what needs to be done) one would submit a JSON with that structure @alliomeria
| @@ -298,19 +318,91 @@ public function invokeProcessorForAdo(ContentEntityInterface $entity, array $sbf | |||
| } | |||
| } | |||
| } | |||
| // JSON/Metadata level plugins coming from ADO JSON directly , $plugin_definition['input_type'] == entity:node | |||
There was a problem hiding this comment.
@alliomeria this is code for the future. This here will allow Post processors to also act on pure NODES and their metadata. I have no plugin that needs this but once we have a few ideas this can used directly.
|
@alliomeria I would like this is mentioned in the new release but documentation is something that we actually need to work internally and generate/decide on the demo implementations. The least harmful ML method here is mobileNet, so maybe enable as an admin that one? Also. I need to know which permission/users should be able to use ML driven Views for testing. |
|
Ok @DiegoPino, understood about your preferences for this. Admin only is what you've been saying for awhile and sounds good to me too. |
|
@alliomeria the only issue with Admin Only is if we have a group of people working/testing (this is query) part that the institution does not trust on full admin access, but still provides Authenticated/User access. |
|
Would defer to you on this @DiegoPino, all depends on how widely you think this tool should be used. Having the option for any type of Authenticated User with the checked permissions, but never anonymous/non-authenticated users also works & enables non-admistrator level users to work/test as you noted. |
|
@alliomeria I will try to work tonight on generating a very simple documentation piece. Added the permissions and restrictions. This requires a new NLP processor container which I will build and publish tomorrow AM (takes me 3 hours of docker built at least). Thanks for reading all this, this is at least 3K lines of code |
See #91
Fully depends on esmero/strawberryfield#326