We size the virtual desktops with 2 vCPU and 3GB. We found this to deliver a much better performing VM compared to our former standard 1 vCPU and 2GB ram. Although average load is important I think key is the VMs are capable enough to handle the peak load. If a virtual desktop has a low laod 95% of the time and a high load 5% of the time the average may be justifying a 1 vCPU and 2GB ram. Our users however complained about the 5% of the time their desktop was not performing well.
I am not saying you should size your Vms 2vCPU and 3GB but I am just sharing what we have experienced in our environment. And of course it all depends on the aplications used. the OS tuning, etc.
Regards.