Troubleshooting Click here for latest

How to address commonly encountered KEDA issues

Why is Kubernetes unable to get metrics from KEDA?

If while setting up KEDA, you get an error: ( status FailedDiscoveryCheck with a message: no response from https://ip:443: Get https://ip:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).

One of the reason for this can be that you are behind a proxy network.

Before you start

  • Make sure no network policies are blocking traffic

Check the status:

Find the api service name for the service keda/keda-metrics-apiserver:

kubectl get apiservice --all-namespaces

Check for the status of the api service found in previous step:

kubectl get apiservice <apiservicename' -o yaml


kubectl get apiservice -o yaml

If the status is False, then there seems to be an issue and proxy network might be the primary reason for it.

Solution for self-managed Kubernetes cluster:

Find the cluster IP for the keda-metrics-apiserver and keda-operator-metrics:

kubectl get services --all-namespaces

In the /etc/kubernetes/manifests/kube-apiserver.yaml - add the cluster IPs found in the previous step in no_proxy variable.

Reload systemd manager configuration:

sudo systemctl daemon-reload

Restart kubelet:

sudo systemctl restart kubelet

Check the API service status and the pods now. Should work!

Solution for managed Kubernetes services:

In managed Kubernetes services you might solve the issue by updating firewall rules in your cluster.

E.g. in GKE private cluster add port 6443 (kube-apiserver) to allowed ports in master node firewall rules.

Why does Google Kubernetes Engine (GKE) 1.16 fail to fetch external metrics?

If you are running Google Kubernetes Engine (GKE) version 1.16, and are receiving the following error:

unable to fetch metrics from external metrics API: <METRIC> is forbidden: User "system:vpa-recommender" cannot list resource "<METRIC>" in API group "" in the namespace "<NAMESPACE>": RBAC: "external-metrics-reader" not found

You are almost certainly running into a known issue.

The workaround is to recreate the external-metrics-reader role using the following YAML:

kind: ClusterRole
  name: external-metrics-reader
- apiGroups:
  - ""
  - "*"
  - list
  - get
  - watch

The GKE team is currently working on a fix that they expect to have out in version >= 1.16.13.

Why does KEDA operator error with NoCredentialProviders

If you are running KEDA on AWS using IRSA or KIAM for pod identity and seeing the following error messages:

  Type     Reason                      Age                From           Message
  ----     ------                      ----               ----           -------
  Normal   KEDAScalersStarted          31s                keda-operator  Started scalers watch
  Normal   KEDAScaleTargetDeactivated  31s                keda-operator  Deactivated apps/v1.Deployment default/my-event-based-deployment from 1 to 0
  Normal   ScaledObjectReady           13s (x2 over 31s)  keda-operator  ScaledObject is ready for scaling
  Warning  KEDAScalerFailed            1s (x2 over 31s)   keda-operator  NoCredentialProviders: no valid providers in chain. Deprecated.
           For verbose messaging see aws.Config.CredentialsChainVerboseErrors

And the operator logs:

2021-11-02T23:50:29.688Z    ERROR    controller    Reconciler error    {"reconcilerGroup": "", "reconcilerKind": "ScaledObject", "controller": "scaledobject", "name": "my-event-based-deployment-scaledobject", "namespace": "default"
, "error": "error getting scaler for trigger #0: error parsing SQS queue metadata: awsAccessKeyID not found"}

This means hat the KEDA operator is not receiving valid credentials, even before attempting to assume the IAM role associated with the scaleTargetRef.

Some things to check:

  • Ensure the keda-operator deployment has the annotation under deployment.spec.template.metadata not deployment.metadata - if using KIAM
  • Ensure the keda-operator serviceAccount is annotated - if using IRSA
  • Check kiam-server logs, successful provisioning of credentials looks like: kube-system kiam-server-6bb67587bd-2f47p kiam-server {"level":"info","msg":"found role","pod.iam.role":"arn:aws:iam::1234567890:role/my-service-role","pod.ip":"","time":"2021-11-05T03:13:34Z"}.
    • Use grep to filter the kiam-server logs, searching for the keda-operator pod ip.

Why is Helm not able to upgrade to v2.2.1 or above?

Our initial approach to manage CRDs through Helm was not ideal given it didn’t update existing CRDs.

This is a known limitation of Helm:

There is no support at this time for upgrading or deleting CRDs using Helm. This was an explicit decision after much community discussion due to the danger for unintentional data loss. Furthermore, there is currently no community consensus around how to handle CRDs and their lifecycle. As this evolves, Helm will add support for those use cases.

As of v2.2.1 of our Helm chart, we have changed our approach so that we automatically managing the CRDs through our Helm chart.

Due to this transition, it can cause upgrade failures if you started using KEDA before v2.2.1 and will cause errors during upgrades such as the following:

Error: UPGRADE FAILED: rendered manifests contain a resource that already exists. Unable to continue with update: CustomResourceDefinition “” in namespace "” exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key “”: must be set to “Helm”; annotation validation error: missing key “”: must be set to “keda”; annotation validation error: missing key “”: must be set to “keda”

In order to fix this, you will need to manually add the following attributes to our CRDs:

  • Helm label
  • keda annotation
  • keda annotation

Future upgrades should work seamlessly.