A follow up to #482 which was solved by disabling Istio sidecar injection on our pods in #496. It would be nice to get actual support working.

Tasks

• Remove the "sidecar.istio.io/inject": "false", label
• Add a service in front of each worker Pod
• Ensure workers share their service name instead of Pod IP as their contact address
• Test and ensure scheduler<>worker and worker<>worker communication works with Istio in enforcing mode

#488 makes a start but isn't working yet.

Some context from #482:

The advice we were given by @stephan-erb-by and @philipp-sontag-by is to put a service in front of every worker and have all inter-pod communication go via services. This should then play nicely with Istio.

Thanks to the changes added in #452 it should be straightforward to extend the CRDs to also include a service for the workers. We would just need to ensure that the workers advertise the service address to the scheduler instead of its own pod IP, this can probably be done by injecting environment variables.