CUDA fails to initialize on H100 SXM5 instance

Uncategorized
1

Whenever I try calling pytorch, I run into this error:

RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda 
functions before calling NumCudaDevices() that might have already set an error? 
Error 802: system not yet initialized

by running this command:

python -c "import torch; print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0))"

Checked nvidia-smi and the GPU is available and is persistent.