[R-package] Add sparse feature contribution predictions #5108

david-cortes · 2022-03-30T20:51:50Z

This PR adds a missing C-level function for predictions on sparse inputs to the R interface.

Docs are not updated in order to avoid future merge conflicts with following PRs.

jameslamb

Thanks very much for this!

To fix errors like this in CI:

── 1. Error (test_Predictor.R:85:5): start_iteration works correctly ───────────
Error in `predictor$predict(data = data, start_iteration = start_iteration, 
    num_iteration = num_iteration, rawscore = rawscore, predleaf = predleaf, 
    predcontrib = predcontrib, header = header, reshape = reshape)`: object 'LGBM_BoosterPredictSparseOutput_R' not found
Backtrace:
 1. stats::predict(bst, test$data, predcontrib = TRUE)
      at test_Predictor.R:85:4
 2. lightgbm:::predict.lgb.Booster(bst, test$data, predcontrib = TRUE)
 3. object$predict(...)
 4. predictor$predict(...)

── 2. Error (test_Predictor.R:130:5): Feature contributions from sparse inputs p
Error in `predictor$predict(data = data, start_iteration = start_iteration, 
    num_iteration = num_iteration, rawscore = rawscore, predleaf = predleaf, 
    predcontrib = predcontrib, header = header, reshape = reshape)`: object 'LGBM_BoosterPredictSparseOutput_R' not found
Backtrace:
 1. stats::predict(bst, Xcsc, predcontrib = TRUE)
      at test_Predictor.R:130:4
 2. lightgbm:::predict.lgb.Booster(bst, Xcsc, predcontrib = TRUE)
 3. object$predict(...)
 4. predictor$predict(...)

I think you need to add LGBM_BoosterPredictSparseOutput_R() in this registration table:

LightGBM/R-package/src/lightgbm_R.cpp

Line 937 in 60244e4

static const R_CallMethodDef CallEntries[] = {

david-cortes · 2022-03-31T07:11:30Z

Added.

jameslamb

Looks great to me! Just left two small sugestions.

Can you please merge in the latest changes from master? After that, I'll run this project's comment-triggered valgrind and Solaris tests.

I'd also like a second opinion from a maintainer who's more experience with C/C++. @shiyu1994 and/or @guolinke can you help review this PR?

jameslamb · 2022-04-02T16:34:56Z

R-package/tests/testthat/test_Predictor.R

+
+    Xspv <- as(X[1L, , drop = FALSE], "sparseVector")
+    pred_spv <- predict(bst, Xspv, predcontrib = TRUE)
+    expect_s4_class(pred_spv, "dsparseVector")


Beyond just testing the type of the returned objects, can you also please add assertions that the predicted values are the same for all of these cases, and that they're the same as those predicted for a regular R matrix?

Those .Call() calls involve passing a lot of positional arguments with similar values, so such assertions would give us greater confidence that this is working correctly.

I thought missing data was handled the same way as xgboost, which means predictions for sparse outputs should be different from those of dense inputs.

I believe the sparse and dense data structures here are just different representations in memory of the exact same matrices, and that specialized methods for them in LightGBM are just intended to allow that sparse data to stay sparse throughout training + scoring.

And I believe that's not directly related to the handling of missing data (which is described in more detail in the discussion at #2921 (comment) and at https://lightgbm.readthedocs.io/en/latest/Advanced-Topics.html?highlight=missing#missing-value-handle).

Consider the following example:

library(lightgbm) library(Matrix) set.seed(708L) data("EuStockMarkets") stockDF <- as.data.frame(EuStockMarkets) feature_names <- c("SMI", "CAC", "FTSE") target_name <- "DAX" # randomly set a portion of each feature to NA or 0 for (col_name in feature_names) { stockDF[ sample( x = seq_len(nrow(stockDF)) , size = as.integer(0.01 * nrow(stockDF)) , replace = FALSE ) , col_name ] <- NA_real_ stockDF[ sample( x = seq_len(nrow(stockDF)) , size = as.integer(0.01 * nrow(stockDF)) , replace = FALSE ) , col_name ] <- 0.0 } X_mat <- data.matrix(stockDF[, feature_names]) y <- stockDF[[target_name]] X_dgCMatrix <- as(X_mat, "dgCMatrix") bst_mat <- lightgbm::lightgbm( data = X_mat , label = y , objective = "regression" , nrounds = 10L ) bst_dgCMatrix <- lightgbm::lightgbm( data = X_dgCMatrix , label = y , objective = "regression" , nrounds = 10L ) # predicted values don't depend on input type from training time or the type of newdata preds_mat_mat <- predict(bst_mat, X_mat) preds_dgCMatrix_mat <- predict(bst_mat, X_dgCMatrix) preds_mat_dgCMatrix <- predict(bst_dgCMatrix, X_mat) preds_dgCMatrix_dgCMatrix <- predict(bst_dgCMatrix, X_dgCMatrix) stopifnot( all( all(preds_mat_mat == preds_dgCMatrix_mat) , all(preds_dgCMatrix_mat == preds_mat_dgCMatrix) , all(preds_mat_dgCMatrix == preds_dgCMatrix_dgCMatrix) ) )

If you find a case where this is not true and LightGBM is creating different predictions for sparse and, I'd consider that a bug worth addressing.

@shiyu1994 @guolinke @StrikerRUS please correct me if I've misspoken.

Good to know, would be useful to have that in the docs, since xgboost works differently (treats non-present sparse entries as missing instead of as zeros) and one might assume both libraries would work the same way.

Oh interesting, I did not know that. https://xgboost.readthedocs.io/en/stable/faq.html#why-do-i-see-different-results-with-sparse-and-dense-data

“Sparse” elements are treated as if they were “missing” by the tree booster, and as zeros by the linear booster. For tree models, it is important to use consistent data formats during training and scoring.

would be useful to have that in the docs

LightGBM's documentation does already describe this behavior directly. Please see https://lightgbm.readthedocs.io/en/latest/Advanced-Topics.html#missing-value-handle

LightGBM uses NA (NaN) to represent missing values by default. Change it to use zero by setting zero_as_missing=true

When zero_as_missing=false (default), the unrecorded values in sparse matrices (and LightSVM) are treated as zeros.

R-package/src/lightgbm_R.cpp

jameslamb · 2022-04-04T00:21:16Z

/gha run r-solaris

Workflow Solaris CRAN check has been triggered! 🚀
https://github.com/microsoft/LightGBM/actions/runs/2086975644

solaris-x86-patched: https://builder.r-hub.io/status/lightgbm_3.3.2.99.tar.gz-f68864de656e497890f659ea1d7a1c83
solaris-x86-patched-ods: https://builder.r-hub.io/status/lightgbm_3.3.2.99.tar.gz-42d22bc664b9485fa8d7c2a982d84405
Reports also have been sent to LightGBM public e-mail: https://yopmail.com?lightgbm_rhub_checks
Status: success ✔️.

jameslamb · 2022-04-04T00:21:23Z

/gha run r-valgrind

Workflow R valgrind tests has been triggered! 🚀
https://github.com/microsoft/LightGBM/actions/runs/2086975964

Status: success ✔️.

david-cortes · 2022-04-04T15:29:02Z

Since PR #4977 is going to be merged, I've now modified this PR to also keep row names when applicable, and added a comparison against dense inputs in the tests.

david-cortes · 2022-04-04T16:49:47Z

Failing CI checks are not related to this PR:
https://github.com/microsoft/LightGBM/runs/5819840473?check_suite_focus=true

jameslamb · 2022-04-04T18:16:48Z

failing CI checks are not related to this PR

I agree, we see this occasionally with our CI jobs that use Homebrew to set up LaTeX.

LaTeX errors when creating PDF version.
This typically indicates Rd problems.
LaTeX errors found:
! LaTeX Error: File `inconsolata.sty' not found.

Type X to quit or <RETURN> to proceed,
or enter new name. (Default extension: sty)

! Emergency stop.
<read *> 
         
l.287 ^^M

Will manually re-run them.

jameslamb

This is great, thanks so much! I left a few more suggested changes.

I'm very happy that the Solaris and valgrind tests are passing 🎉

Since this PR introduces so much new C++ code, I'd like my review to not be the only one that counts towards a merge. @shiyu1994 @guolinke could one of you please help review this as well?

jameslamb · 2022-04-09T01:47:20Z

R-package/R/lgb.Predictor.R

+        if (NROW(row.names(data))) {
+          out@Dimnames[[1L]] <- row.names(data)
+        }


Can you please test this behavior, by adding these new combinations of predcontrib = TRUE + new {Matrix} classes to the tests from #4977?

LightGBM/R-package/tests/testthat/test_Predictor.R

Line 155 in 3798f87

# sparse matrix with row names

LightGBM/R-package/tests/testthat/test_Predictor.R

Line 166 in 3798f87

# sparse matrix without row names

Every PR adding new behavior to the package should include tests on that behavior, to catch unexpected issues with the implementation and to prevent future development from accidentally breaking that behavior.

Those are already tested in the tests from the PR for row names.

They are not. The links in my comment point to tests on "CsparseMatrix" objects, but if you click them you won't see tests on the types referenced in this PR: "dsparseMatrix", "dsparseVector", "dgRMatrix", "dgCMatrix".

Is there something I've misunderstood about the relationship between these classes?

They are not. The links in my comment point to tests on "CsparseMatrix" objects, but if you click them you won't see tests on the types referenced in this PR: "dsparseMatrix", "dsparseVector", "dgRMatrix", "dgCMatrix".

Is there something I've misunderstood about the relationship between these classes?

There's a class hierarchy...

Please tell us specifically what you mean by "there's a class hierarchy", and why it means that you don't want to add the tests I'm asking you to add.

dgCMatrix is a subclass of CsparseMatrix, which is a subclass of sparseMatrix, and so on. Classes like dsparseMatrix are abstract.

Thanks for that. I'm still struggling to understand how that means that the tests I'm asking for shouldn't be added.

Consider the case added in this PR beginning with } else if (inherits(data, "dgRMatrix")) {. It doesn't contain a return() statement, so that at the end of the if - else if block, the "possibly add row names" code (the line this comment thread is on) will run.

If someone were to add return(out) on line 193, that "possibly add row names" code wouldn't be reached. I think it's desirable for a test to fail in that case, to inform us that adding that return statement is a breaking change that causes row names to not be set on the predictions.

The tests I linked test a regular dense R matrix and a CsparseMatrix. As the example below shows, that means they don't currently cover the cases where the input to predict() is a "dgRMatrix" or a "dsparseVector".

library(Matrix) # the first batch of test cases use a regular R dense matrix X <- matrix(rnorm(100), ncol = 4) inherits(X, "dgRMatrix") # FALSE inherits(X, "dsparseMatrix") # FALSE inherits(X, "dsparseVector") # FALSE inherits(X, "dgCMatrix") # FALSE # the second batch of test cases converts that to a CsparseMatrix Xcsc <- as(X, "CsparseMatrix") inherits(Xcsc, "dgRMatrix") # FALSE inherits(Xcsc, "dsparseMatrix") # TRUE inherits(Xcsc, "dsparseVector") # FALSE inherits(Xcsc, "dgCMatrix") # TRUE

Ok, if you really want to have tests for everything, I've added a test for CSR matrices. A vector representing a single row cannot have row names so I left that out of tests.

if you really want to have tests for everything

Thank you, yes I do. The more of this project's behaviors are reflected in tests, the less likely it is that future changes silently break that behavior. This project is too large for all of these concerns to just be kept in maintainers' heads and enforced through PR comments.

R-package/R/lgb.Predictor.R

jameslamb · 2022-04-09T02:06:55Z

R-package/tests/testthat/test_Predictor.R

+
+    Xspv <- as(X[1L, , drop = FALSE], "sparseVector")
+    pred_spv <- predict(bst, Xspv, predcontrib = TRUE)
+    expect_s4_class(pred_spv, "dsparseVector")


Oh interesting, I did not know that. https://xgboost.readthedocs.io/en/stable/faq.html#why-do-i-see-different-results-with-sparse-and-dense-data

“Sparse” elements are treated as if they were “missing” by the tree booster, and as zeros by the linear booster. For tree models, it is important to use consistent data formats during training and scoring.

would be useful to have that in the docs

LightGBM's documentation does already describe this behavior directly. Please see https://lightgbm.readthedocs.io/en/latest/Advanced-Topics.html#missing-value-handle

LightGBM uses NA (NaN) to represent missing values by default. Change it to use zero by setting zero_as_missing=true

When zero_as_missing=false (default), the unrecorded values in sparse matrices (and LightSVM) are treated as zeros.

Co-authored-by: James Lamb <[email protected]>

david-cortes · 2022-04-14T18:53:25Z

Failing CI checks like this are definitely not from this PR: https://github.com/microsoft/LightGBM/runs/6029069414?check_suite_focus=true

jameslamb · 2022-04-15T00:48:27Z

Failing CI checks like this are definitely not from this PR

Yep, a recent security patch to git broke some of our containerized CI jobs (#5151). Once #5152 is merged, they should be fixed.

david-cortes · 2022-05-14T14:15:51Z

The linter raised a curious comment:

warning: Function "cbind" is undesirable. As an alternative, cbind is an unsafe way to build up a data frame. merge() or direct column assignment is preferred..

Says that "cbind is an unsafe way to build up a data frame", but the code is not building a data.frame, nor is it meant to do so.

It then suggests merge, which does not do the same operation as cbind.

Then it suggests direct column assignment, which will not work if the LHS is a matrix instead of data.frame.

The linter thus leaves one out of efficient options for concatenating matrices or other non-data-frame objects.

jameslamb

Thanks very much for adding tests. I left one more suggestion for those tests...otherwise, I approve of this from the R side.

Before merging though, I would still like some help from @guolinke or @shiyu1994 to review the changes in lightgbm_R.cpp (#5108 (review)).

The linter raised a curious comment ... The linter thus leaves one out of efficient options for concatenating matrices

Since you haven't mentioned the specific place where you wanted to use cbind() and what you did instead, it's not possible for maintainers to judge whether whatever pattern you used instead is less efficient than using cbind().

The warning you've mentioned was added a while ago in this project, to nudge contributors who were not familiar with R into using safer patterns when creating data frames. If you have a specific proposal for why that restriction should be removed, with a specific example of a place in {lightgbm}'s code that would benefit from the use of cbind(), I'd be happy to consider removing that restriction.

jameslamb · 2022-05-22T04:58:05Z

R-package/tests/testthat/test_Predictor.R

+
+    X_wrong <- X[, c(1L:10L, 1L:10L)]
+    X_wrong <- as(X_wrong, "CsparseMatrix")
+    expect_error(predict(bst, X_wrong, predcontrib = TRUE))


for these expect_error() calls, can you please use argument regexp to be sure that these tests are matching the specific error message they're intended to catch?

Similar to these test cases:

LightGBM/R-package/tests/testthat/test_basic.R

Lines 721 to 729 in c000b8c

expect_error({

bst <- lgb.train(

data = dtrain

, params = list(

objective_type = "not_a_real_objective"

, verbosity = VERBOSITY

)

)

}, regexp = "Unknown objective type name: not_a_real_objective")

LightGBM/tests/python_package_test/test_basic.py

Lines 602 to 604 in c000b8c

with pytest.raises(lgb.basic.LightGBMError,

match="Cannot find parser class 'dummy', please register first or check config format"):

data.construct()

We've found that approach to error-catching tests useful to prevent the case where tests silently pass in the presence of other, unexpected errors happening on the codepaths the test touches.

R-package/R/lgb.Predictor.R

david-cortes · 2022-05-24T17:07:10Z

Thanks very much for adding tests. I left one more suggestion for those tests...otherwise, I approve of this from the R side.

Before merging though, I would still like some help from @guolinke or @shiyu1994 to review the changes in lightgbm_R.cpp (#5108 (review)).

The linter raised a curious comment ... The linter thus leaves one out of efficient options for concatenating matrices

Since you haven't mentioned the specific place where you wanted to use cbind() and what you did instead, it's not possible for maintainers to judge whether whatever pattern you used instead is less efficient than using cbind().

The warning you've mentioned was added a while ago in this project, to nudge contributors who were not familiar with R into using safer patterns when creating data frames. If you have a specific proposal for why that restriction should be removed, with a specific example of a place in {lightgbm}'s code that would benefit from the use of cbind(), I'd be happy to consider removing that restriction.

Any usage of cbind with CSC matrices would be more efficient and take fewer lines of code than creating an empty matrix and assigning columns from the cbinded objects. Don't see any usage for that right now, but something like a test checking that it throws an error with inputs of incorrect shape would benefit from calling cbind.

jameslamb · 2022-06-14T01:13:22Z

@shiyu1994 @guolinke can you please provide a review?

guolinke · 2022-06-14T14:45:08Z

C++ part looks good to me

jameslamb

Thanks for looking @guolinke !

Ok @david-cortes , I think we can move forward with this! Since it's been a few weeks since the last commit on this PR, can you please update it to the latest state of master? If CI passes after that, I'll merge this up.

Thanks for all the hard work!

david-cortes · 2022-06-16T17:28:38Z

Updated.

david-cortes · 2022-06-16T17:34:50Z

The linter is interpreting regular expressions as file paths:

[[1]]
/home/runner/work/LightGBM/LightGBM/R-package/tests/testthat/test_Predictor.R:167:71: warning: [non_portable_path] Use file.path() to construct portable file paths.
    expect_error(predict(bst, X_wrong, predcontrib = TRUE), regexp = "input data has \\d+ columns")
                                                                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~

jameslamb · 2022-06-16T18:16:36Z

The linter is interpreting regular expressions as file paths:

I noticed something similar upgrading to {lintr} 3.0, in #5294 . In that PR, I used a # nolint comment. But for this PR, could you please just replace \\d+ with the literal number of columns expected? Since you'll have to touch that line anyway, might as well also make the test a bit stricter.

david-cortes · 2022-06-17T11:38:50Z

Updated.

github-actions · 2023-08-19T03:51:47Z

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.

add predcontrib for sparse inputs

508fb1b

david-cortes requested review from Laurae2, jameslamb and jmoralez as code owners March 30, 2022 20:51

jameslamb requested changes Mar 31, 2022

View reviewed changes

jameslamb added the feature label Mar 31, 2022

register newly-added function

557a005

jameslamb requested changes Apr 2, 2022

View reviewed changes

jameslamb requested review from shiyu1994 and guolinke April 2, 2022 16:43

david-cortes added 4 commits April 3, 2022 18:43

solve merge conflicts

a740dc8

comments

631c935

correct wrong types in test

82900f6

forcibly take transpose function from Matrix

6fd0868

jameslamb added the awaiting review label Apr 4, 2022

keep row names, test comparison to dense inputs

398ddb9

workaround for passing test while PR for row names is not merged

7d53353

solve merge conflict

19b706f

jameslamb self-requested a review April 9, 2022 02:07

jameslamb requested changes Apr 9, 2022

View reviewed changes

david-cortes and others added 4 commits April 10, 2022 11:21

Update R-package/R/lgb.Predictor.R

8b016b8

Co-authored-by: James Lamb <[email protected]>

Update R-package/R/lgb.Predictor.R

df49aaa

Co-authored-by: James Lamb <[email protected]>

Update R-package/R/lgb.Predictor.R

5df97de

Co-authored-by: James Lamb <[email protected]>

proper handling of integer overflow

6eeddc4

jameslamb mentioned this pull request Apr 28, 2022

[R-package] allow use of categorical_features in Dataset when raw data does not have column names (fixes #4374) #5184

Merged

david-cortes added 3 commits May 14, 2022 15:32

solve merge conflicts

4ac5a3b

add more tests for predict(<sparse>, predcontrib=TRUE)

e38a6d7

make linter happy

5b65fd1

david-cortes added 2 commits May 14, 2022 16:16

linter

c00a8ff

linter

d905910

jameslamb self-requested a review May 22, 2022 04:59

jameslamb requested changes May 22, 2022

View reviewed changes

david-cortes added 2 commits May 26, 2022 22:10

check error messages for bad input shapes

edaac4e

fix regex

ab86ad3

guolinke approved these changes Jun 14, 2022

View reviewed changes

jameslamb approved these changes Jun 16, 2022

View reviewed changes

jameslamb removed the awaiting review label Jun 16, 2022

Merge github.com:microsoft/lightgbm into Rcsr1

005707d

hard-coded number of columns in regex for tests

3f8467a

jameslamb merged commit 6f92d47 into microsoft:master Jun 17, 2022

david-cortes mentioned this pull request Jun 20, 2022

[R-package] Add remainder of prediction funtions #5312

Merged

jameslamb mentioned this pull request Oct 7, 2022

[DO NOT MERGE] Release v3.3.3 #5525

Closed

40 tasks

github-actions bot locked as resolved and limited conversation to collaborators Aug 19, 2023

	expect_error({
	bst <- lgb.train(
	data = dtrain
	, params = list(
	objective_type = "not_a_real_objective"
	, verbosity = VERBOSITY
	)
	)
	}, regexp = "Unknown objective type name: not_a_real_objective")

	with pytest.raises(lgb.basic.LightGBMError,
	match="Cannot find parser class 'dummy', please register first or check config format"):
	data.construct()

[R-package] Add sparse feature contribution predictions #5108

[R-package] Add sparse feature contribution predictions #5108

Conversation

david-cortes commented Mar 30, 2022

jameslamb left a comment

Choose a reason for hiding this comment

david-cortes commented Mar 31, 2022

jameslamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

david-cortes Apr 4, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jameslamb commented Apr 4, 2022 • edited by guolinke Loading

jameslamb commented Apr 4, 2022 • edited by guolinke Loading

david-cortes commented Apr 4, 2022

david-cortes commented Apr 4, 2022

jameslamb commented Apr 4, 2022

jameslamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jameslamb Apr 13, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

david-cortes commented Apr 14, 2022

jameslamb commented Apr 15, 2022

david-cortes commented May 14, 2022

jameslamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

david-cortes commented May 24, 2022

jameslamb commented Jun 14, 2022

guolinke commented Jun 14, 2022

jameslamb left a comment

Choose a reason for hiding this comment

david-cortes commented Jun 16, 2022

david-cortes commented Jun 16, 2022

jameslamb commented Jun 16, 2022

david-cortes commented Jun 17, 2022

github-actions bot commented Aug 19, 2023

david-cortes Apr 4, 2022 •

edited

Loading

jameslamb commented Apr 4, 2022 •

edited by guolinke

Loading

jameslamb commented Apr 4, 2022 •

edited by guolinke

Loading

jameslamb Apr 13, 2022 •

edited

Loading