-
Notifications
You must be signed in to change notification settings - Fork 25k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Transform] improve placement of transform task targetting a remote index #50033
Comments
Pinging @elastic/ml-core (:ml/Transform) |
From my understanding we require the remote cluster to be accessible from any ML node. That's explicit because all |
The problem described in #46025 is as far as I understand about having more than 1 ML node while some are not allowed to connect to a remote cluster. This is for sure an exotic case and probably even a misconfiguration, however that's what we allow today and we saw such a setup at least once in the wild. Transform can run into the same problem, as it potentially runs on more nodes the problem is more likely to happen (but still exotic). The 2nd concern is usability: We can not reliable check when a user previews, updates or creates a transform.
Transform runs only on data nodes. A dedicated node type for background tasks sounds like a good idea, but is not solving the problem, because you would still be able to configure some of those with remote access and some not. Again usability is involved: it's not possible to check whether all "background task nodes" have remote access or not. TL/DR Making the ml/transform/background node setting mutually exclusive to disabling remote connection(meaning the node refuses to start if you enable ml but disable remote) sounds wrong to me. I guess nobody wants that, just mentioning this possibility. Maybe the discussion should be whether |
I think that that would be reasonable to require users to configure nodes eligible for transform with remote access. Today it is required on all data nodes since that's where Transform can run but it would be more flexible to tag the node eligible to run these tasks like we do for ml. I don't really like the automatic assignment based on the node setting, this opens the door to more specialization and makes it difficult to achieve good repartition even when you have a lot of nodes.
I think it's ok to allow specialization here but the requirement should be propagated to all features that use remote search. Today clients targets specific machine to coordinate the search, they sometimes exclude data nodes and master eligible nodes, we should have the same for transform; |
@hendrikmuhs Hi Hendrik, is there any plans to allow CCS indices in transforms in a nearest future? We are looking for this kind of functionality to support KPI processing for Cloud ECE and ESS environments. |
@nestermn test out the latest 7.x. @hendrikmuhs recently made changes that should be in 7.6 that allows transforms to supportt CCS (reading from remote clusters) |
@nestermn as @benwtrent already said: |
@benwtrent @hendrikmuhs Great to hear. Looking forward to try it |
…odes and test which nodes are allowed to do remote connections closes elastic#52200 closes elastic#50033 closes elastic#48734
Transform supports CCS indexes starting with 7.6.
The problem
A transform using remote source requires
cluster.remote.connect:true
(default) to execute search queries, ifcluster.remote.connect:false
is set on a node the transform will fail.Transform validation will disallow preview/create if
cluster.remote.connect:false
is set to false on the executing node, assuming the situation is the same for all nodes.However,
cluster.remote.connect
is a node setting. Worst-case a user has a mixed environment with some nodes enabled and some not. In this case p-task placement is unable to find out and can not choose a data node that has CCS support and validation might fail or not.The problem applies not only to transform but also to ML, see #46025.
The workaround: Known Limitation
For #43201 this issue is out of scope, in order to use transform CCS you must have remote-support enabled on all data and the master node.
In case remote is disabled on the executing node transform will fail at start with a descriptive error.
A possible Solution: node attributes
Whether
cluster.remote.connect
istrue
orfalse
can not be retrieved from cluster state. A nodeinfo call would be possible, but calls all nodes in the cluster and is therefore not considered as an option.Similar to the ML plugin, we could use node attributes which are available in cluster state. A special attribute could be set if
cluster.remote.connect:false
at the start of the node, e.g. as part of x-pack core (all features that require remote aware placement are part of x-pack, ignoring that CCS is a core feature (OSS)).Other solutions
@elastic/es-search any ideas? comments?
The text was updated successfully, but these errors were encountered: