Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UBI goes distrib #11

Merged
merged 6 commits into from
Dec 10, 2024
Merged

UBI goes distrib #11

merged 6 commits into from
Dec 10, 2024

Conversation

mkhludnev
Copy link

This what I understand about distributed Solr.

Pardon, I barely understand what's going on there.

// -------------------

CollectionAdminRequest.createCollection("ubi_queries", "_default", 1, 1)
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@epugh I noticed that UBI comp yields a kind of stream into ubi_queries. Should we create such collection before issuing queries?

@mkhludnev
Copy link
Author

mkhludnev commented Dec 7, 2024

UBIComponentDistrQueriesTest fails with

12544 WARN  (qtp1249319859-57) [n: c: s: r: x: t:] o.e.j.s.HttpChannel /solr/collection1_shard2_replica_n1/select
          => java.lang.AssertionError
	at org.apache.solr.handler.component.RealTimeGetComponent.addDocListToResponse(RealTimeGetComponent.java:1180)
java.lang.AssertionError: null
	at org.apache.solr.handler.component.RealTimeGetComponent.addDocListToResponse(RealTimeGetComponent.java:1180) ~[main/:?]

and it doesn't seem specific to UBI. I'm stuck. Don't know how to get through it.

okay. I've figured it out https://github.com/apache/solr/pull/2452/files#r1875035115

assertTrue(qid.length()>10);
Thread.sleep(10000); // I know what you think of
// TODO check that ids were recorded
QueryResponse queryCheck = cluster.getSolrClient("ubi_queries").query(new MapSolrParams(
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@epugh do you have an idea how to wait till update stream have landed to ubi_queries?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hummm.... so, I think that in the UBIComponent when we call getTuples, there is a chance that it waits till we finish that step? You are using the configset that wraps the update() with an explicit commit(), which you woudn't do in a prod setup, just so we can see the new docs:

https://github.com/apache/solr/pull/2452/files#diff-cdf9aaba8c6979df48bf217e3f81fa7be42af135abd382c4fba28d79778b4945

I looked at some tests like https://github.com/apache/solr/blob/main/solr/solrj-streaming/src/test/org/apache/solr/client/solrj/io/stream/StreamDecoratorTest.java#L364 and they are why I believe it is actually a synchornous process, not a back ground process...

@epugh
Copy link
Owner

epugh commented Dec 10, 2024

Would it be useful to add you to the epugh/solr repo?

@mkhludnev
Copy link
Author

Would it be useful to add you to the epugh/solr repo?

nope. I can push into apache#2452 as a maintainer.
Just merge this once please, since it's a kind of ground level for distributed UBI.
And then I'll continue in apache#2452 if you don't mind.

@epugh epugh merged commit fce0334 into epugh:ubi Dec 10, 2024
1 check passed
@epugh
Copy link
Owner

epugh commented Dec 10, 2024

Merged! and please do continue! this is super great to have a collaborator!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants