Have you tried connecting after a reboot? Is it something the "other" machine is doing that may be causing this?
How often does the "normal connection" occur? or is it a hit and miss?
Have you tried uninstalling the agent, uninstall VMware tools; then, reinstalling VMware Tools and re-installing the Agent?
What version of View is this and what version of vSphere (vCenter and ESXi)? What OS is the client and the remote desktop?