Adding and running tests

A high-quality suite of tests is crucial in ensuring correctness and robustness of the codebase. Here, we provide instructions how to run unit tests, and also how to add a new one.

Contents

Adding a new unit test

Python package: pytest

Add your test under the directories

Refer to the PyTest tutorial to learn how to write tests for Python code.

You may try running your test by following instructions in this section.

C++: Google Test

Add your test under the directory tests/cpp/. Refer to this excellent tutorial on using Google Test.

You may try running your test by following instructions in this section. Note. Google Test version 1.8.1 or later is required.

JVM packages: JUnit / scalatest

The JVM packages for XGBoost (XGBoost4J / XGBoost4J-Spark) use the Maven Standard Directory Layout. Specifically, the tests for the JVM packages are located in the following locations:

To write a test for Java code, see JUnit 5 tutorial. To write a test for Scala, see Scalatest tutorial.

You may try running your test by following instructions in this section.

R package: testthat

Add your test under the directory R-package/tests/testthat. Refer to this excellent tutorial on testthat.

You may try running your test by following instructions in this section.

Running Unit Tests Locally

R package

Run

python ./ops/script/test_r_package.py --task=check

at the root of the project directory. The command builds and checks the XGBoost r-package. Alternatively, if you want to just run the tests, you can use the following commands after installing XGBoost:

cd R-package/tests/
Rscript testthat.R

JVM packages

Maven is used

mvn test

Python package: pytest

To run Python unit tests, first install pytest package:

pip3 install pytest

Then compile XGBoost according to instructions in Building the Shared Library. Finally, invoke pytest at the project root directory:

# Tell Python where to find XGBoost module
export PYTHONPATH=./python-package
pytest -v -s --fulltrace tests/python

In addition, to test CUDA code, run:

# Tell Python where to find XGBoost module
export PYTHONPATH=./python-package
pytest -v -s --fulltrace tests/python-gpu

(For this step, you should have compiled XGBoost with CUDA enabled.)

For testing with distributed frameworks like Dask and PySpark:

# Tell Python where to find XGBoost module
export PYTHONPATH=./python-package
pytest -v -s --fulltrace tests/test_distributed

C++: Google Test

To build and run C++ unit tests enable tests while running CMake:

cmake -B build -S . -GNinja -DGOOGLE_TEST=ON -DUSE_DMLC_GTEST=ON -DUSE_CUDA=ON -DUSE_NCCL=ON
cmake --build build
cd ./build
./testxgboost

Flags like USE_CUDA, USE_DMLC_GTEST are optional. For more info about how to build XGBoost from source, see Building From Source. One can also run all unit tests using ctest tool which provides higher flexibility. For example:

ctest --verbose

If you need to debug errors on Windows using the debugger from VS, you can append the gtest flags in test_main.cc:

::testing::GTEST_FLAG(filter) = "Suite.Test";
::testing::GTEST_FLAG(repeat) = 10;

Sanitizers: Detect memory errors and data races

By default, sanitizers are bundled in GCC and Clang/LLVM. One can enable sanitizers with GCC >= 4.8 or LLVM >= 3.1, But some distributions might package sanitizers separately. Here is a list of supported sanitizers with corresponding library names:

  • Address sanitizer: libasan

  • Undefined sanitizer: libubsan

  • Leak sanitizer: liblsan

  • Thread sanitizer: libtsan

Memory sanitizer is exclusive to LLVM, hence not supported in XGBoost. With latest compilers like gcc-9, when sanitizer flags are specified, the compiler driver should be able to link the runtime libraries automatically.

How to build XGBoost with sanitizers

One can build XGBoost with sanitizer support by specifying -DUSE_SANITIZER=ON. By default, address sanitizer and leak sanitizer are used when you turn the USE_SANITIZER flag on. You can always change the default by providing a semicolon separated list of sanitizers to ENABLED_SANITIZERS. Note that thread sanitizer is not compatible with the other two sanitizers.

cmake -DUSE_SANITIZER=ON -DENABLED_SANITIZERS="address;undefined" /path/to/xgboost

By default, CMake will search regular system paths for sanitizers, you can also supply a specified SANITIZER_PATH.

cmake -DUSE_SANITIZER=ON -DENABLED_SANITIZERS="address;undefined" \
-DSANITIZER_PATH=/path/to/sanitizers /path/to/xgboost

How to use sanitizers with CUDA support

Running XGBoost on CUDA with address sanitizer (asan) will raise memory error. To use asan with CUDA correctly, you need to configure asan via ASAN_OPTIONS environment variable:

ASAN_OPTIONS=protect_shadow_gap=0 ${BUILD_DIR}/testxgboost

Other sanitizer runtime options

By default undefined sanitizer doesn’t print out the backtrace. You can enable it by exporting environment variable:

UBSAN_OPTIONS=print_stacktrace=1 ${BUILD_DIR}/testxgboost

For details, please consult official documentation for sanitizers.