Fix config-settings handling in pip install

[hcho3]

Also test --config-settings in the CI so that we know it is not broken.

[hcho3]

Collecting pyspark (from -r doc/requirements.txt (line 13))
  Downloading pyspark-3.4.0.tar.gz (310.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 310.8/310.8 MB 174.2 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'error'
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
      [end of output]  

For some reason, the doc build is broken :(

[hcho3]

I manually tested this change by:

1. Create a new Conda environment.
2. Build XGBoost from the source using CMake:

conda activate my_env
cmake .. -GNinja -DCMAKE_INSTALL_PREFIX="$CONDA_PREFIX" -DCMAKE_INSTALL_LIBDIR="lib"

3. Install libxgboost into the Conda env:

ninja install

4. Install the Python package with use_system_libxgboost flag:

pip install -v .  --config-settings use_system_libxgboost=True

5. Inspect $CONDA_PREFIX to ensure that only one copy of libxgboost.so is installed in the Conda environment.

$ find . -name libxgboost.so
./lib/libxgboost.so

[hcho3]

I filed readthedocs/readthedocs.org#10285 to notify RTD about the broken doc build.

[trivialfis]

LGTM, assuming all CI tests pass.

[hcho3]

@trivialfis Can we ignore the RTD error for now? The RTD won't deploy the latest bug fix until next Tuesday.

[trivialfis]

Yup, let's skip the unrelated errors.

@rongou Have you seen this error before https://buildkite.com/xgboost/xgboost-ci/builds/2386 ?

[rongou]

@trivialfis no, but will take a look.

[trivialfis]

Feel free to merge it.