MPI CUDA-aware support

Opening this issue to find a solution to MPI CUDA-aware support. In this NVIDIA blog it is mentioned the need to call cudaSetDevice before MPI_Init, depending on the implementation. However Kokkos::initialize is supposed to be called after MPI_Init, reference.

Solutions:

  • Should we manually do the GPU binding ?
  • Drop support for MPI CUDA-aware ?
  • Patch only for Omni-Path, hence Jean-Zay

Third option may be the most pragmatic solution for now.

Edited by thomas.padioleau