CUDA_LOG_FILE.........................................

  CUDA_LOG_FILE

Another good way to identify CUDA errors is with the CUDA_LOG_FILE environment variable. When this environment variable is set, the CUDA driver will write error messages encountered out to a file whose path is specified in the environment variable. For example, take the following incorrect CUDA code, which attempts to launch a thread block which is larger than the maximum supported by any architecture.

__global__ void k()

{ }

int main()

{

        k<<<8192, 4096>>>(); // Invalid block size

        CUDA_CHECK(cudaGetLastError());

        return 0;

}

Building and running this, the check after the kernel launch detects and reports the error using the macros illustrated in Section 2.1.7.

$ nvcc errorLogIllustration.cu -o errlog

$ ./errlog

CUDA Runtime Error: /home/cuda/intro-cpp/errorLogIllustration.cu:24:1 = invalid argument

However, when the application is run with CUDA_LOG_FILE set to a text file, that file contains a bit more information about the error.

$ env CUDA_LOG_FILE=cudaLog.txt ./errlog

CUDA Runtime Error: /home/cuda/intro-cpp/errorLogIllustration.cu:24:1 = invalid argument

$ cat cudaLog.txt

[12:46:23.854][137216133754880][CUDA][E] One or more of block dimensions of (4096,1,1) exceeds corresponding maximum value of (1024,1024,64)

[12:46:23.854][137216133754880][CUDA][E] Returning 1 (CUDA_ERROR_INVALID_VALUE) from cuLaunchKernel

Setting CUDA_LOG_FILE to stdout or stderr will print to standard out and standard error, respectively. Using the CUDA_LOG_FILE environment variable, it is possible to capture and identify CUDA errors, even if the application does not implement proper error checking on CUDA return values. This approach can be extremely powerful for debugging, but the environment variable alone does not allow an application to handle and recover from CUDA errors at runtime. The error log management feature of CUDA also allows a callback function to be registered with the driver which will be called whenever an error is detected. This can be used to capture and handle errors at runtime, and also to integrate CUDA error logging seamlessly into an application’s existing logging system.

Section 4.8 shows more examples of the error log management feature of CUDA. Error log management and CUDA_LOG_FILE are available with NVIDIA Driver version r570 and later.