To allow users to close the troubleshooting loop through Komodor, Komodor offers the ability to perform a number of Actions against Kubernetes resources. These Actions are auditable, meaning that a record is made of what Action was performed and by whom.

Prerequisites

• Agent version 0.1.104
• Required permissions (permissions can be modified as needed)

  - apiGroups:
    - apps
    resources:
    - deployments/scale
    - statefulsets/scale
    - deployments
    - replicasets
    - statefulsets
    - daemonsets
    verbs:
    - patch
  - apiGroups:
    - ""
    resources:
    - pods
    verbs:
    - delete
  - apiGroups:
    - batch
    resources:
    - jobs
    verbs:
    - delete
    - create
  - apiGroups:
    - ""
    resources:
    - pods
    - persistentvolumeclaims
    - configmaps
    - services
    - persistentvolumes
    - storageclasses
    verbs:
    - delete
    - patch
    - update
    - create
  - apiGroups:
    - apps
    resources:
    - replicasets
    - deployments
    - statefulsets
    - daemonsets
    verbs:
    - delete
    - patch
    - update
    - create
  - apiGroups:
    - batch
    resources:
    - cronjobs
    - jobs
    verbs:
    - delete
    - patch
    - update
    - create
  - apiGroups:
    - networking.k8s.io
    resources:
    - ingresses
    - networkpolicies
    verbs:
    - delete
    - patch
    - update
    - create
  - apiGroups:
    - ""
    resources:
    - nodes
    verbs:
    - patch
  - apiGroups:
    - ""
    resources:
    - pods/eviction
    verbs:
    - create

• Please note: To perform actions against Kubernetes resources, the user has to be either an account-admin or be provided with permission to perform actions, to read more about Komodor RBAC

How to opt-in

For the sake of convenience, we've separated the Actions Helm chart values into two sections
- watcher.actions.basic - Enables basic actions (Delete pods, Scale and restart deployments, statefulsets, replicasets. Restart and trigger jobs)
- watcher.actions.advanced - Enables advanced actions (Update, Create, and Delete resources, Cordon/Uncordon nodes)

New agent installation

To install a new agent with Actions enabled, ensure that the Helm chart installation process includes the following flags

Cluster upgrade

helm repo add komodorio https://helm-charts.komodor.io ; helm repo update; helm upgrade --install k8s-watcher komodorio/k8s-watcher --set watcher.actions.basic=true --set watcher.actions.advanced=true --reuse-values

How to revoke

To disable the usage of Actions using Helm, apply the following command:

helm repo add komodorio https://helm-charts.komodor.io ; helm repo update; helm upgrade --install k8s-watcher komodorio/k8s-watcher --set watcher.actions.basic=false --set watcher.actions.advanced=false --reuse-values

How does it work?

• User triggers a Manual Action through the Komodor platform
• A command is registered to the Komodor SaaS
• The Agent running in the cluster fetches the command from the Komodor SaaS (communication is always done from the Agent outside of the cluster)
• The command is triggered against the Kubernetes API
• Kubernetes will now execute the command
• During the entire process you can track the changes/events through a dedicated Event that will be created on the Komodor timeline.

Please note: Due to Kubernetes' nature, this feature is built asynchronously, review the timeline after triggering any action for updates

Komodor Actions Demo - Watch Video

Audit

For auditing purposes, Manual Action events are created on the Komodor timeline

What type of Actions are supported and where can they be triggered from?

We currently support the following actions: 
- Scale service - Allows modifying the number of replicas for a Service. Can be triggered from Deployment/StatefulSet inspection pages (under Workloads) and also from a Service timeline page
- Delete Pod - Deletes/kills a specific Pod. Can be triggered from both the Pod inspection page (under Workloads) and the Pods & Logs screen
- Restart service - Triggers a rolling restart of all the Pods of a Service. Can be triggered from Deployment/StatefulSet inspection page (under Workloads) and also from a Service timeline page
- Rollback service - Rolls back a service to the previous generation
- Re-trigger Job/CronJob - Re-creates the Job to trigger a new run of it. Can be triggered from a Job/CronJob timeline, Job/CronJob inspection pages (under Workloads) and from a Job event drawer
- Cordon/Uncordon node - Allows marking a node as unscheduable, preventing new Pods from being scheduled on it. You can revert this by using the Uncordon action
- Delete/Edit resources

 

Pod Shell

Pod shell allows you to initiate a shell session against containers in your clusters so you can run various commands to examine the state of the container from within.

In essence, it is similar to the experience provided by the kubectl exec -it POD_NAME -- /bin/bash command

Prerequisites

• Agent version >= 0.1.173
• Prerequisites for performing actions via the agent (see here)
• Required Permissions

  - apiGroups:
      - ""
    resources:
      - pods/exec
    verbs:
      - create

User permissions

Pod Shell is available for admin users or users with the pod:exec Komodor RBAC permission, read more about Komodor RBAC here

Opting-in

For convenience, the following flag has been added to configure the agent to support pod shell access:

watcher.actions.podExec=true

Agent Upgrade

To upgrade an existing agent to support pod shell access execute the following command:

helm repo add komodorio https://helm-charts.komodor.io ; helm repo update; helm upgrade --install k8s-watcher komodorio/k8s-watcher --set watcher.actions.basic=true --set watcher.actions.advanced=true --set watcher.actions.podExec=true --reuse-values

Revoking Shell Access

If you would like to revoke only pod shell access run the following command:

helm repo add komodorio https://helm-charts.komodor.io ; helm repo update; helm upgrade --install k8s-watcher komodorio/k8s-watcher --set watcher.actions.podExec=false --reuse-values

If you would also like to revoke actions permission from the agent run the following command:

helm repo add komodorio https://helm-charts.komodor.io ; helm repo update; helm upgrade --install k8s-watcher komodorio/k8s-watcher --set watcher.actions.basic=false --set watcher.actions.advanced=false --set watcher.actions.podExec=false --reuse-values

Accessing pod shell

Pod shell is accessible from two main places in Komodor

Option 1: Via Pods Page

Navigate to the pods page on the left navbar

Select a pod in the Running state

Either Click on actions -> PodExec

OR Click the pod row -> Click the Pod Shell button on the top

The shell is opened

Option 2: Via Service Page

Select the services page on the left navbar

Select a service with at least one healthy pod

Select the Pods tab

Select a pod in the Running state

Either click on actions -> Pod Exec

Or click on the pod row -> Click on the Pod Shell button on the top

The shell is opened

How does it work?

• Both client and agent keep a persistent connection to Komodor's backend
• Client initiates a pod shell session
• If there are multiple containers the user will have to choose the container to open the shell session to
• Authentication and authorization take place to make sure the client has permission to run a shell session against the requested pod
• An audit record and an event on the corresponding service timeline are created so you can track pod shell usage
• The agent receives the message and opens a shell session against the requested pod and container using the Kubernetes API
• All subsequent messages are passed from the Client to the Agent, injected into the session, and then routed back to the Client for display
• The agent will close the shell session if the client closes the session voluntarily or if the session has been idle for more than 5 minutes.

Please note: Agent restarts will cause the session to restart and any in-session state will be lost (environment variables for example)

The remote container has to have either sh or bash installed on it to open a shell session against it

Multiple Shells & Shell Minimization

You can minimize a shell session. The session will be kept in a tab on the bottom side of the web application. While it is minimized, you can keep exploring your system, move between different pages, and interact regularly with the web application. When you'd like to resume the shell session, click the tab at the bottom of the screen and the terminal will be displayed again.

You can also open and keep multiple shell sessions minimized. In this case, you will be able to choose which session you'd like to resume by using the select menu from the shells tab on the bottom side of the web application.