Locking Down Istio Egress with Automatic Traffic Discovery

By Dor Kolog

The Istio servicemesh provides visibility and control over Kubernetes micro-service flows — you can use it to monitor performance, route traffic and segment the network at layer-7.

The Problem

Most applications have some dependencies on external services for things like storage, message queues and authentication.

When deploying Istio onto a Kubernetes cluster, you need to explicitly define the external endpoints that the application relies on. This is a manual process involving trial and error, something that negatively impacts the on-boarding user experience.

For example, if your application uses Google Cloud Pub/Sub, you need to instruct Istio to allow access to this service’s endpoint by adding a Service Entry definition:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: pubsub.googleapis.com
  namespace: default
spec:
  hosts:
  - pubsub.googleapis.com
  ports:
  - name: https
    number: 443
    protocol: HTTPS

Up until version 1.0, Istio’s default behavior was to block access to external endpoints — this created a connectivity issue and applications were breaking until the user could discover all the endpoints and configure them manually.

Istio 1.1 changed the default to allow access to all external endpoints:

This trades security for user experience; it’s now easier to deploy Istio but until you define those external endpoints the door to the Internet is left wide open, allowing malicious agents to reach out to their command-and-control servers or to exfiltrate stolen data.

The Solution — Automation

We created this open source project to automate the process. It inserts an additional container into the Kubernetes DNS service which acts as a DNS proxy, monitors DNS lookups and automatically configures Istio to allow access to the endpoints by adding a Service Entry for each external hostname.

The proxy can be easily deployed to learn the egress communication patterns automatically. When the learning phase is done, you can configure Istio to block access to all non-explicitly allowed endpoints.

When to Discover Egress?

Here are a few options ordered from simple to advanced:

1. Discover egress traffic in the production environment
2. Discover egress traffic in a staging environment
3. If you have comprehensive integration tests, or system tests, you can perform discovery during the testing phase
4. If you run the tests in a CI pipeline, perhaps in a dedicated k8s namespace, you can discover egress traffic there

Summary

dns-discovery is an open-source project that automatically detects egress traffic in a Kubernetes cluster and creates a matching Service Entry object for each external endpoint. This improves visibility to the cluster traffic and allows Istio users to enforce a tight network security policy.

See dns-discovery in action:

dns-discovery demo

I hope that you will find this beneficial and I look forward to your feedback through issues and pull requests.

Dor