alternative #1
alternative #1
Conversation
| val offerAttributes = toAttributeMap(offer.getAttributesList) | ||
| matchesAttributeRequirements(slaveOfferConstraints, offerAttributes) | ||
| if (blacklist.contains(offer.getHostname)) { | ||
| false |
There was a problem hiding this comment.
This will save many computation on the invalid offers.
There was a problem hiding this comment.
I don't mind to add this optimisation, but imo, you can't remove original check. Since they happening in two different places and the code that matches offer is "repeated" at canLaunchTask
There was a problem hiding this comment.
It sounds like you are saying unmatchedOffers will be evaluated at canLaunchTask. Does it what you mean really? At line 392, we simple do declineUnmatchedOffers(d, unmatchedOffers).
There was a problem hiding this comment.
I mean that even though the offer is matched, then there is code in canLaunchTask that verifies that task can be launched with respect to cpu/memory requirements(so also blacklisting)
There was a problem hiding this comment.
I said the computation on the invalid offers (not matched).
This will save many computation on the invalid offers.
There was a problem hiding this comment.
I agree with @dongjoon-hyun , seems this would be more efficient. I don't understand the objection
|
Since it's already over 20 hours and it seems you disagree, I'll wait for 4 hours from now and make independent PR on the Apache Spark community officially for the wider review. At that time, I'll close this PR. |
IgorBerman
merged commit dbe7d18
into
IgorBerman:SPARK-19755-Blacklist-is-always-active-for-MesosCoarseGrainedSchedulerBackend
|
@squito, @dongjoon-hyun I'm not against this PR, what I "against" is removing check from canLaunchTask method, but I agree that checking blacklisting in match offers will be better. |
No description provided.