Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]

[Orygeunik]

What Happened?

Hello!
I'm tried to enable ingress addon, but takes error such as from title
partial reference for issue #13841
Originally posted by @spowelljr in #13841 (comment)

root@orion:~# minikube addons enable ingress --alsologtostderr
I0328 21:23:12.286390 1954403 out.go:297] Setting OutFile to fd 1 ...
I0328 21:23:12.286699 1954403 out.go:349] isatty.IsTerminal(1) = true
I0328 21:23:12.286760 1954403 out.go:310] Setting ErrFile to fd 2...
I0328 21:23:12.286819 1954403 out.go:349] isatty.IsTerminal(2) = true
I0328 21:23:12.287050 1954403 root.go:315] Updating PATH: /root/.minikube/bin
W0328 21:23:12.287255 1954403 root.go:293] Error reading config file at /root/.minikube/config/config.json: open /root/.minikube/config/config.json: no such file or directory
I0328 21:23:12.288046 1954403 config.go:176] Loaded profile config "minikube": Driver=none, ContainerRuntime=docker, KubernetesVersion=v1.23.4-rc.0
I0328 21:23:12.288111 1954403 addons.go:65] Setting ingress=true in profile "minikube"
I0328 21:23:12.288159 1954403 addons.go:153] Setting addon ingress=true in "minikube"
I0328 21:23:12.288490 1954403 host.go:66] Checking if "minikube" exists ...
I0328 21:23:12.289084 1954403 exec_runner.go:51] Run: systemctl --version
I0328 21:23:12.294233 1954403 kubeconfig.go:92] found "minikube" server: "https://192.168.188.160:8443"
I0328 21:23:12.294262 1954403 api_server.go:165] Checking apiserver status ...
I0328 21:23:12.294304 1954403 exec_runner.go:51] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0328 21:23:12.305745 1954403 exec_runner.go:51] Run: sudo egrep ^[0-9]+:freezer: /proc/1951927/cgroup
W0328 21:23:12.314202 1954403 api_server.go:176] unable to find freezer cgroup: sudo egrep ^[0-9]+:freezer: /proc/1951927/cgroup: exit status 1
stdout:

stderr:
I0328 21:23:12.314232 1954403 api_server.go:240] Checking apiserver healthz at https://192.168.188.160:8443/healthz ...
I0328 21:23:12.319308 1954403 api_server.go:266] https://192.168.188.160:8443/healthz returned 200:
ok
I0328 21:23:12.334410 1954403 out.go:176]     ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.1.1
I0328 21:23:12.344282 1954403 out.go:176]     ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
I0328 21:23:12.354291 1954403 out.go:176]     ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
I0328 21:23:12.354596 1954403 addons.go:348] installing /etc/kubernetes/addons/ingress-deploy.yaml
I0328 21:23:12.354654 1954403 exec_runner.go:144] found /etc/kubernetes/addons/ingress-deploy.yaml, removing ...
I0328 21:23:12.354670 1954403 exec_runner.go:207] rm: /etc/kubernetes/addons/ingress-deploy.yaml
I0328 21:23:12.354731 1954403 exec_runner.go:151] cp: memory --> /etc/kubernetes/addons/ingress-deploy.yaml (15567 bytes)
I0328 21:23:12.354865 1954403 exec_runner.go:51] Run: sudo cp -a /tmp/minikube3233579437 /etc/kubernetes/addons/ingress-deploy.yaml
I0328 21:23:12.363407 1954403 exec_runner.go:51] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.23.4-rc.0/kubectl apply -f /etc/kubernetes/addons/ingress-deploy.yaml
I0328 21:23:12.936710 1954403 addons.go:386] Verifying addon ingress=true in "minikube"
I0328 21:23:12.999722 1954403 out.go:176] 
🔎  Verifying ingress addon...
I0328 21:23:13.000806 1954403 kapi.go:75] Waiting for pod with label "app.kubernetes.io/name=ingress-nginx" in ns "ingress-nginx" ...
I0328 21:23:13.044867 1954403 kapi.go:86] Found 3 Pods for label selector app.kubernetes.io/name=ingress-nginx
I0328 21:23:13.044897 1954403 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0328 21:23:13.548198 1954403 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0328 21:23:14.048591 1954403 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0328 21:23:14.547859 1954403 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
... cut ...
I0328 21:29:13.050823 1954403 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0328 21:29:13.050850 1954403 kapi.go:108] duration metric: took 6m0.050047714s to wait for app.kubernetes.io/name=ingress-nginx ...
I0328 21:29:13.063110 1954403 out.go:176]

W0328 21:29:13.063269 1954403 out.go:241] ❌  Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
❌  Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
W0328 21:29:13.063341 1954403 out.go:241]

W0328 21:29:13.072474 1954403 out.go:241] 

Attach the log file

logs.txt

Operating System

Other (Debian 11.3)

Driver

none

[spowelljr]

Hi @Orygeunik, thanks for reporting your issue with minikube!

Just looking at the logs I'm seeing bind: address already in use and Operation cannot be fulfilled on jobs.batch "ingress-nginx-admission-create": the object has been modified.

I also see that the cluster was originally created with 1.24.0 and now you're using 1.25.2.

The original cluster might have discrepancies with the updated version, I'd recommend starting with deleting the existing clusters minikube delete --all and then trying starting minikube again and enabling ingress again and let me know if that resolves your issue or your get different errors, thanks!

[Orygeunik]

Thanks for quick answer!

At first i want to remark that this error was received me on version 1.24 and i was hopes that update fixit it :)

So i was executed command minikube delete --all (little question, how to update minikube without errors like this?) and again run command minikube addons enable ingress --alsologtostderr
But errror ocured:

W0329 00:26:38.460857 1983999 out.go:241] ❌  Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]

[Orygeunik]

root@orion:~# minikube addons enable ingress --alsologtostderr
I0329 00:20:36.650345 1983999 out.go:297] Setting OutFile to fd 1 ...
I0329 00:20:36.652417 1983999 out.go:349] isatty.IsTerminal(1) = true
I0329 00:20:36.652433 1983999 out.go:310] Setting ErrFile to fd 2...
I0329 00:20:36.652492 1983999 out.go:349] isatty.IsTerminal(2) = true
I0329 00:20:36.652659 1983999 root.go:315] Updating PATH: /root/.minikube/bin
I0329 00:20:36.653128 1983999 config.go:176] Loaded profile config "minikube": Driver=none, ContainerRuntime=docker, KubernetesVersion=v1.23.4-rc.0
I0329 00:20:36.653186 1983999 addons.go:65] Setting ingress=true in profile "minikube"
I0329 00:20:36.653203 1983999 addons.go:153] Setting addon ingress=true in "minikube"
I0329 00:20:36.653407 1983999 host.go:66] Checking if "minikube" exists ...
I0329 00:20:36.653684 1983999 exec_runner.go:51] Run: systemctl --version
I0329 00:20:36.682942 1983999 kubeconfig.go:92] found "minikube" server: "https://192.168.188.160:8443"
I0329 00:20:36.683005 1983999 api_server.go:165] Checking apiserver status ...
I0329 00:20:36.683048 1983999 exec_runner.go:51] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0329 00:20:36.730755 1983999 exec_runner.go:51] Run: sudo egrep ^[0-9]+:freezer: /proc/1983293/cgroup
W0329 00:20:36.754976 1983999 api_server.go:176] unable to find freezer cgroup: sudo egrep ^[0-9]+:freezer: /proc/1983293/cgroup: exit status 1
stdout:

stderr:
I0329 00:20:36.755079 1983999 api_server.go:240] Checking apiserver healthz at https://192.168.188.160:8443/healthz ...
I0329 00:20:36.759621 1983999 api_server.go:266] https://192.168.188.160:8443/healthz returned 200:
ok
I0329 00:20:36.770284 1983999 out.go:176]     ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.1.1
I0329 00:20:36.779802 1983999 out.go:176]     ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
I0329 00:20:36.788875 1983999 out.go:176]     ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
I0329 00:20:36.789150 1983999 addons.go:348] installing /etc/kubernetes/addons/ingress-deploy.yaml
I0329 00:20:36.789188 1983999 exec_runner.go:144] found /etc/kubernetes/addons/ingress-deploy.yaml, removing ...
I0329 00:20:36.789207 1983999 exec_runner.go:207] rm: /etc/kubernetes/addons/ingress-deploy.yaml
I0329 00:20:36.789274 1983999 exec_runner.go:151] cp: memory --> /etc/kubernetes/addons/ingress-deploy.yaml (15567 bytes)
I0329 00:20:36.789416 1983999 exec_runner.go:51] Run: sudo cp -a /tmp/minikube101386137 /etc/kubernetes/addons/ingress-deploy.yaml
I0329 00:20:36.795590 1983999 exec_runner.go:51] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.23.4-rc.0/kubectl apply -f /etc/kubernetes/addons/ingress-deploy.yaml
I0329 00:20:38.395316 1983999 exec_runner.go:84] Completed: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.23.4-rc.0/kubectl apply -f /etc/kubernetes/addons/ingress-deploy.yaml: (1.599607702s)
I0329 00:20:38.395385 1983999 addons.go:386] Verifying addon ingress=true in "minikube"
I0329 00:20:38.409437 1983999 out.go:176] 🔎  Verifying ingress addon...
🔎  Verifying ingress addon...
I0329 00:20:38.410507 1983999 kapi.go:75] Waiting for pod with label "app.kubernetes.io/name=ingress-nginx" in ns "ingress-nginx" ...
I0329 00:20:38.433520 1983999 kapi.go:86] Found 3 Pods for label selector app.kubernetes.io/name=ingress-nginx
I0329 00:20:38.433657 1983999 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0329 00:20:38.937061 1983999 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0329 00:20:39.437494 1983999 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
...cut...
I0329 00:26:38.440491 1983999 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0329 00:26:38.440518 1983999 kapi.go:108] duration metric: took 6m0.030023123s to wait for app.kubernetes.io/name=ingress-nginx ...
I0329 00:26:38.460716 1983999 out.go:176]

W0329 00:26:38.460857 1983999 out.go:241] ❌  Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
❌  Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
W0329 00:26:38.460873 1983999 out.go:241]

W0329 00:26:38.462117 1983999 out.go:241] 

new logs.txt

[Orygeunik]

I'm sorry @spowelljr, anything other need?

[spowelljr]

Hi @Orygeunik, sorry for the delayed response.

After the ingress fails to enable, you can run the command kubectl get pods -A and you should see something like the following:

NAMESPACE       NAME                                       READY   STATUS      RESTARTS   AGE
ingress-nginx   ingress-nginx-admission-create-8dnv9       0/1     Completed   0          3m14s
ingress-nginx   ingress-nginx-admission-patch-2x7q2        0/1     Completed   1          3m14s
ingress-nginx   ingress-nginx-controller-cc8496874-xxjhc   1/1     Running     0          3m15s
kube-system     coredns-64897985d-zlbf8                    1/1     Running     0          5m3s
kube-system     etcd-minikube                              1/1     Running     0          5m17s
kube-system     kube-apiserver-minikube                    1/1     Running     0          5m15s
kube-system     kube-controller-manager-minikube           1/1     Running     0          5m15s
kube-system     kube-proxy-lzrng                           1/1     Running     0          5m3s
kube-system     kube-scheduler-minikube                    1/1     Running     0          5m15s
kube-system     storage-provisioner                        1/1     Running     0          5m14s

Then if you do kubectl describe pod -n ingress-nginx <pod-name> you should be able to see all of the logs within the pod, I'm assuming one of those ingress pods is going to be failing to start and will have useful errors in there.

[Orygeunik]

Hello @spowelljr

I was rerun again deleting and enabling, such as in comment: #13872 (comment)

And i have below situation:

root@orion:~# kubectl get pods -A
NAMESPACE       NAME                                       READY   STATUS    RESTARTS   AGE
ingress-nginx   ingress-nginx-admission-create-8nlcb       0/1     Pending   0          7m
ingress-nginx   ingress-nginx-admission-patch-jp8wz        0/1     Pending   0          7m
ingress-nginx   ingress-nginx-controller-cc8496874-9lhhr   0/1     Pending   0          7m
kube-system     coredns-64897985d-5hzh5                    0/1     Pending   0          7m18s
kube-system     etcd-orion                                 1/1     Running   6          7m30s
kube-system     kube-apiserver-orion                       1/1     Running   7          7m30s
kube-system     kube-controller-manager-orion              1/1     Running   7          7m30s
kube-system     kube-proxy-pl8nc                           1/1     Running   0          7m19s
kube-system     kube-scheduler-orion                       1/1     Running   6          7m30s
kube-system     storage-provisioner                        0/1     Pending   0          7m24s

kubectl describe pod -n ingress-nginx ingress-nginx-admission-create-8nlcb

Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

  Warning  FailedScheduling  31s (x390 over 6h40m)  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.

kubectl describe pod -n ingress-nginx ingress-nginx-admission-patch-jp8wz

Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

  Warning  FailedScheduling  86s (x390 over 6h41m)  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.

kubectl describe pod -n ingress-nginx ingress-nginx-controller-cc8496874-9lhhr

Tolerations:                 node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

  Warning  FailedScheduling  35s (x396 over 6h45m)  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.

[16871560]

@Orygeunik do you solve this problem , I got the same issue. oops....

[16871560]

[Orygeunik]

do you solve this problem , I got the same issue. oops....

@16871560 no, the problem still persisted

[16871560]

@Orygeunik I reinstall 1.23.2. it seems work well.