Only error I'd get when attempting to delete the folder while browsing the datastore is "Cannot delete file [<datastore name>] replica-<long string of characters>
They shouldn't be in use (in theory) as I rebuilt the image and recomposed all desktops.