You can tag the connections servers and tag the pool going out the security server for example "external" and tag the connection server paired to the security server "external"
Using Tags to control Location Based Access to pool Desktops | VMware Support Insider - VMware Blogs