Can you ensure that all Connection Servers have at least 10GB of RAM and 4vCPUs configured at the time of install of the Connection Server software? If you are not sure your server was configured this way at install, you can check the Java Heap value (JvmOptions) at HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VDM\plugins\wsnm\TomcatService\Params which should read a 4gb value (e.g., it should read -Xmx4096m as per http://kb.vmware.com/kb/2009877)
If it doesn't show 4GB for the java heap, then you'll only have 1GB allocated which could cause this type of behavior.