Fix Docker Crio Test Image Tests

[medyagh]

The following Tests are failing on Docker CRIO Linux, we need to figoure out why image tests are failing,
the image tests are supposed to run simple commands such as
minikube image ls

is that actually failing on our CIRO ?

98	TestFunctional/parallel/ServiceCmdConnect	602.99
115	TestFunctional/parallel/ServiceCmd/DeployApp	600.65
118	TestFunctional/parallel/ImageCommands/ImageListShort	2.27
124	TestFunctional/parallel/ImageCommands/ImageLoadDaemon	0.93
127	TestFunctional/parallel/ImageCommands/ImageReloadDaemon	0.94
131	TestFunctional/parallel/ImageCommands/ImageTagAndLoadDaemon	1.52
132	TestFunctional/parallel/ImageCommands/ImageSaveToFile	0.47
134	TestFunctional/parallel/ImageCommands/ImageLoadFromFile	0.19
135	TestFunctional/parallel/ImageCommands/ImageSaveDaemon	0.37
153	TestFunctional/parallel/ServiceCmd/HTTPS	0.53
154	TestFunctional/parallel/ServiceCmd/Format	0.53
155	TestFunctional/parallel/ServiceCmd/URL	0.53

98 TestFunctional/parallel/ServiceCmdConnect 602.99
115 TestFunctional/parallel/ServiceCmd/DeployApp 600.65
118 TestFunctional/parallel/ImageCommands/ImageListShort 2.27
124 TestFunctional/parallel/ImageCommands/ImageLoadDaemon 0.93
127 TestFunctional/parallel/ImageCommands/ImageReloadDaemon 0.94
131 TestFunctional/parallel/ImageCommands/ImageTagAndLoadDaemon 1.52
132 TestFunctional/parallel/ImageCommands/ImageSaveToFile 0.47
134 TestFunctional/parallel/ImageCommands/ImageLoadFromFile 0.19
135 TestFunctional/parallel/ImageCommands/ImageSaveDaemon 0.37
153 TestFunctional/parallel/ServiceCmd/HTTPS 0.53
154 TestFunctional/parallel/ServiceCmd/Format 0.53
155 TestFunctional/parallel/ServiceCmd/URL 0.53

https://storage.googleapis.com/minikube-builds/logs/21724/41893/Docker_Linux_crio.html#fail_TestFunctional%2fparallel%2fServiceCmdConnect

[obnoxxx]

/asign

[obnoxxx]

/assign

[obnoxxx]

note to self locally try to repro with this:

$ make integration TEST_ARGS="-minikube-start-args='--container-runtime=crio --driver=docker'  -test.run TestFunctional"
$ make html_report

[obnoxxx]

note to self locally try to repro with this:

$ make integration TEST_ARGS="-minikube-start-args='--container-runtime=crio --driver=docker' -test.run TestFunctional"
$ make html_report

running the test command yields this output:

...
...
...
        --- FAIL: TestFunctional/parallel/ServiceCmd (607.19s)
            --- FAIL: TestFunctional/parallel/ServiceCmd/DeployApp (600.63s)
            --- PASS: TestFunctional/parallel/ServiceCmd/List (1.67s)
            --- PASS: TestFunctional/parallel/ServiceCmd/JSONOutput (1.67s)
            --- FAIL: TestFunctional/parallel/ServiceCmd/HTTPS (0.49s)
            --- FAIL: TestFunctional/parallel/ServiceCmd/Format (0.49s)
            --- FAIL: TestFunctional/parallel/ServiceCmd/URL (0.48s)
        --- FAIL: TestFunctional/parallel/ServiceCmdConnect (602.58s)
    --- PASS: TestFunctional/delete_echo-server_images (0.04s)
    --- PASS: TestFunctional/delete_my-image_image (0.01s)
    --- PASS: TestFunctional/delete_minikube_cached_images (0.01s)
=== RUN   TestFunctionalNewestKubernetes
    functional_test.go:82: 
--- SKIP: TestFunctionalNewestKubernetes (0.00s)
FAIL
Tests completed in 13m25.989375868s (result code 1)
FAIL    k8s.io/minikube/test/integration        806.033s
FAIL

but exit code 0.

[medyagh]

$ make integration TEST_ARGS="-minikube-start-args='--container-runtime=crio --driver=docker' -test.run TestFunctional"

install gopogh

$ make html_report

[Stefan956]

/assign

[Stefan956]

@medyagh
Quick update:
I need a little help. I ran the tests locally both with and without the CRI-O runtime:

• Without CRI-O: ✅ all tests passed
• With CRI-O: ❌ tests failed

I manually ran all the commands from functional_test.go - ImageLoadDaemonTest(for example), but I didn't catch any issues regarding the manual testing.
The commands:

minikube start --driver=docker --container-runtime=crio -p "functional"

docker pull "kicbase/echo-server:1.0" || docker pull "kicbase/echo-server:functional"

minikube -p "functional" image load "kicbase/echo-server:functional" --daemon --alsologtostderr -v=6

minikube -p "functional" ssh -- sudo crictl images | sed -n '1,200p'

This is the output of:

minikube -p "functional" image ls

registry.k8s.io/pause:3.10.1
registry.k8s.io/kube-scheduler:v1.34.0
registry.k8s.io/kube-proxy:v1.34.0
registry.k8s.io/kube-controller-manager:v1.34.0
registry.k8s.io/kube-apiserver:v1.34.0
registry.k8s.io/etcd:3.6.4-0
registry.k8s.io/coredns/coredns:v1.12.1
localhost/kicbase/echo-server:functional
gcr.io/k8s-minikube/storage-provisioner:v5
docker.io/kindest/kindnetd:v20250512-df8de77b

• I can see that the kicbase image is present, but there is added 'localhost/' prior to it. When ran without the CRI-O runtime the 'localhost/' is not there.
• I tried adding "localhost/" to the image name inside the tests, but the functional tests still failed with the CRI-O runtime.

Could you please provide a hint on how I should continue debugging this, because I haven't noticed any irregularities in the code?

[medyagh]

@Stefan956 nice discovery ! so it seems like test fails because only in CRIO image names are listed differently !

so I guess we have two ways to fix this, either in the Test add an if statement if it is crio, Expect localhost in the image name

or we could change the implementation of Minikube image ls to remove the localhost/ prefix....

I think I prefer the first option....
@afbjorklund what do you think ?

[Stefan956]

@medyagh I’ve already tried modifying the Test to expect localhost/ when running with CRIO runtime.

In functional_test.go, I updated the checkImageExists function (line 454) to expect localhost/ + image in order to see if the functional tests would pass with CRIO runtime. This made all images expected to have the localhost/ prefix, but unfortunately the tests still fail. Otherwise, I would have added an if statement specifically for CRIO.

Is this leaving us with only the second solution - to change the implementation of minikube image ls to remove the localhost/ prefix?

[medyagh]

@Stefan956 I think the solution is,
we need to check is the image usable wihtout the localhost ?

how about
miniube ssh

then try to
sudo crictl pull (image_name_with_loclalhost)

and

sudo crictl pull (image_name_without_localhost)

and if it does work without localhost then lets trim it from minikube image ls, otherwise we need to make changes to our tests