The NVIDIA Grid K1 card is not supported in the x3650m4 system. The main reason for this is that the passive cooling solution on this card doesn't work well in this system due to the pre-heated air supplied to the PCIe slots. Actively cooled cards work better in this system, such as the Quadro cards from NVIDIA. The Grid K1 card is supported in the IBM iDataPlex dx360m4 system as well as the IBM PureFlex x220/x240 with the PCIe Expansion Node.
IBM Redbooks | IBM System x3650 M4
Have you tried disabling the "64 bit PCI resource" uEFI setting? This may prevent the Grid K1 card from working since this card requires a lot of system resources for the 4 GPUs on this card and there may not be enough room in 32bit space. However based on your description it sounds like your running into a similar issue seen on other platforms where memory resources are being assigned to the GPUs above where VMware ESXi 5.1 can access them and disabling 64bit PCI Resources setting would prevent this from happening, but may also prevent the card from being assigned the resources it needs.