Flink Rightsizing

The Problem

When Apache Flink runs on Kubernetes (via the official Flink Kubernetes Operator, in either native or standalone mode), the JobManager and TaskManager processes don't get distinct Kubernetes resource kinds. Both roles run as pods owned by the same Deployment identity — in native mode, Flink's own resource manager creates the TaskManager pods and points them directly at the single JobManager Deployment, so from Kubernetes' point of view there is just one workload.

The problem is that JobManagers and TaskManagers have very different resource profiles: the JobManager is a lightweight coordinator that is mostly idle (handling scheduling and checkpointing), while TaskManagers do the actual, often heavy, stream-processing compute. Without a way to tell the two apart, right-sizing had to blend both roles' usage into a single recommendation — one that fit neither: JobManagers ended up over-provisioned, TaskManagers under-provisioned (or the reverse).

How We Solved It

Komodor's in-cluster admission controller now reads the standard labels the Flink Kubernetes Operator already applies to every pod it creates:

  • type: flink-native-kubernetes or type: flink-standalone-kubernetes
  • component: jobmanager or component: taskmanager

From these labels we derive a Flink role, which is recorded as the pod's execution unit. Right-sizing recommendations and the underlying metrics are then keyed per (workload, container, role) instead of just (workload, container) — so JobManager and TaskManager pods sharing the same Deployment now get separate, accurate recommendations.

This is handled automatically across every way Flink is commonly deployed:

  • Native mode — a single Deployment owning both JobManager and TaskManager pods
  • Standalone mode — a separate Deployment per role
  • High-availability JobManager — multiple JobManager pods correctly share the same role
  • Manually configured Flink deployments — as long as the same type/component labels are present

The Flink Kubernetes Operator's own controller deployment (which is not a Flink job) is correctly excluded and does not get a Flink role. Workloads with no discernible role continue to receive a single unified recommendation exactly as before, so there's no behavior change for non-Flink workloads.

Get Started

There is nothing to enable — this works automatically, with no feature flag and no configuration, as soon as:

  1. The Komodor agent and admission controller are installed in the cluster (standard Komodor installation).
  2. Flink is deployed via the official Flink Kubernetes Operator (native or standalone mode), or a manual deployment carrying the same type/component labels.
  3. A right-sizing policy is scoped to the Flink Deployment(s), the same way you'd scope a policy to any other Deployment — there's no separate "Flink" resource kind to select, since Flink still runs as a standard Kubernetes Deployment underneath.

What's Available in the UI

When you open the right-sizing recommendation view for a Flink Deployment, the containers table now includes an Execution Unit column whenever a workload has more than one role. Each row shows jobmanager or taskmanager alongside its own independent CPU and memory request/limit recommendations, so you can review and apply them separately instead of getting one blended number for the whole Deployment.

This reuses the same "Execution Unit" UI already in place for other multi-role workloads (for example, Ray head/worker and Airflow tasks), so the experience will look familiar if you've used right-sizing with those workload types before.

Was this article helpful?

0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.