As i understand from vmware documentation IPSec needs powered on windows firewall,. So i've tried with ipsec + powered on firewall and also witout ipsec + powered off firewall.
Today i've noticed interesting thing. I turn on widows firewall on both VM, restart them and start installing secure server immediatly after the reboot. In this way i didn't receive error message. I entered pair password and security server installation start. But it failed at the end with error message about IPSec. So the question is what can change in 3-5 minutes after OS start that causes IPSec failure. (because if i wait 5 minutes and then start installation of security server i receive error on the begining)