[Roadmap] Multiple outputs.

[trivialfis]

Since the XGBoost 1.6, we have been working on having multi-output support for the tree model. In 2.0, we will have the initial implementation for the vector-leaf-based multi-output model. This issue is a tracker for future development and for related discussion. The original feature request is here: #2087 . The related features are for vector leaf instead of general multi-output.

Feel free to share your suggestions or make related feature requests in the comments.

Implementation Optimization

• Use f-order for the gradient. Currently, the gradient has one column for each target but is written in C-order. The transformation takes about one-fifth of the training time. (Use matrix for gradient. #9508)
• Use f-order for the custom objective. (Inefficient Casting Grad and Hess to c_float for Custom Obj in Python API #9089)
• Improve array type dispatching by moving the dispatch logic from per-element to per-array. This enables us to have a more efficient custom objective interface. (Optimize array interface input. #9090)

Algorithmic Optimization

We are still looking for potential algorithmic optimization for vector-leaf and here's the pool of candidates. We need to survey all available options. Feel free to share if you have ideas or paper recommendations.

• Sketch boost.
• https://arxiv.org/abs/2201.06239
• Extra tree.

GPU Implementation

• Evaluation
• Histogram
• Prediction

Documentation

• Derive the approximated hessian in the context of boosting trees.

Multi-task

• Multi-task xgboost. This is not yet decided. I think it's wise to at least do some exploration before forging the rest of the implementation since we will have a very different interface if we need to consider multi-task. Related: [RFC] Exposing objectives and metrics as part of the API. #7693 .

Features

• Tree SHAP
• Plotting (Support graphviz plot for multi-target tree. #10093)
• Model text dump (JSON, txt, graphviz) (Support graphviz plot for multi-target tree. #10093)
• Tree data frame.
• Categorical feature.
• Constraints
• Approx tree method
• Exact tree method
• Loss weight
• Feature importance (be careful with tree index) ([multi] Implement weight feature importance. #10700)

Learning to rank

We can have a ranking model to consider multiple criteria. This might require multi-task to be supported.

Quantile regression

• l1
• quantile

Distributed

• Dask
• PySpark
• Spark
• Flink?
• Federated (Support column split in multi-target hist #9171)

Binding

• R ([R] Support multi-class custom objective. #9526)
• Scala
• Python
• Java
• C

HPO

• Check compatibility with major HPO frameworks.

Other extensions

• Sparse label. (multi-label classification optimization)
• Missing label.
• Early stopping for each target?

Applications

• https://arxiv.org/abs/2210.06831
• FIL

Benchmarks

• Collection of datasets for future comparison.

[CarloLepelaars]

Hi, great work on the initial multitarget implementation!

Given the roadmap when can we expect GPU support for multi output regression? When this support is added will xgboost-ray also support it?

[trivialfis]

Hi @CarloLepelaars ,

• For model-per-target, it's already implemented.
• For vector leaf, it will take some more work, but eventually yes. I don't have an eta for when it will be available.
• I think there is on-going work on the ray-xgboost, but you will need to open an issue on that repository for concrete answers.

[wiktorolszowy]

Hi, very nice work! I am wondering how SHAP should be used for multi-output models, e.g. how to explain links between the Ys, and how to interpret the effects of Xs - e.g., which Xs display common effects across the Ys, and which Xs display differential effects. Do you know a good example of using SHAP for a multi-output model?

[trivialfis]

For model per target, it's the same as single target. As for vector leaf, I haven't looked into it yet, but no significant difference on top of my mind.

[lcrmorin]

I am currently toying with multitargets approach ... I have a hard time defining a custom metric (haven't tried custom loss). Preds seems to be of size (len(y) x len(targets)) while y_true is of shape (len(y), len(targets)), I have managed to handle this internally to my metric to return one value. But now I have an error about an output being a tuple instead of a number.
Any way to handle this properly or is it too early ?

[Reederey87]

Hi.

Did anybody train the multiple outputs XGBoost model on Mac arm64 machine?

On recent stable version I have got error:
XGBoostError('[...] Check failed: !trees.front()->IsMultiTarget(): Update tree leaf support for multi-target tree is not yet implemented.

On latest nightly version xgboost-2.1.0.dev0+a7226c02223246be78a59c3a4e8c32d1c68c1ff9 - I have managed load CPU, but it was no feedback on terminal window.

[aniruddhghatpande]

Is the vector-leaf-based multi-output model still work in progress ? Also what research paper based on which splitting mechanism for decision trees is working for this ? @trivialfis

[trivialfis]

yes, it's still working in progress.

[mxdub]

Hi @trivialfis,

I'm currently working on some models using XGBoostLSS which as far as I understand is based on the multi-output feature of XGBoost. I wonder how monotonic constraints are considered in the multi-ouput case ? It seems constraints are shared among trees built for each target, could you confirm ?

Thanks for your work on this feature !

[abseejp]

Hello @trivialfis, I'm working on a multi label binary classification problem (I have three targets) and all my targets are highly imbalanced. But I don't seem to understand how I can leverage scale_pos_weight to help with that.

Also, will I be able to access functionalities such as shapley values computation?

[asrpv]

Hi @trivialfis , I have been using the muli-ouput-tree feature on a couple of real world datasets across tasks of regression and classification and also with custom loss functions, the results are great. An observation:

The JSON model file of the multi-output-tree model doesn't include loss_changesand sum_hessian. Any particular reason for not providing that? Given that we do a diagonal approximation of hessian, won't the sum_hessian simply be a vector of each target's hessian? and likewise for loss_changes? Is it possible to provide them? I believe providing them would be beneficial in both making sense of the tree splits and also in Tree SHAP (by modifying the JSON of the multi-output-tree model into that of a single target by keeping the target of interest).

Thanks for the great work on this! Looking forward for the successful implementation of the roadmap.

[trivialfis]

Any particular reason for not providing that

It's still working in progress. We still need to add these fields.

Given that we do a diagonal approximation of hessian

This happens only if the output targets are correlated, like softmax. For multi-target regression, we assume targets are independent in the gradient calculation. The potential dependencies are captured in the output tree instead of the gradient calculation.

[miladbarooni]

Hi @trivialfis
I encountered a problem when attempting to refresh a multi-output tree using the xgboost.Booster class from the Python API. The error is:

Check failed: gpair->Shape(1) == 1 (10 vs. 1): support for multi-target tree is not yet implemented. (Which is likely related to this line)

Why can't we perform the same procedure as single-output trees? In my specific scenario, I aim to adjust the leaf values using the updated gradient statistics from a different dataset without changing thresholds. Due to the small sample size, each iteration requires me to update the previous trees for each new batch of samples based on their gradients.

[trivialfis]

I'm currently trying to work on some other priorities, will get back to this.