How to handle irrelevant v2 custom content when upgrading to 3.6 of etcd

[Timothy-Dougherty]

Recently I've been trying to upgrade some etcd instances to 3.6 and I'm running into an error:

"msg":"illegal v2store content","error":"detected disallowed custom content in v2store"

Now we may have some old v2 custom content, but we migrated a long time ago to v3 and it might just been hanging around doing nothing.
We've also had the v2 endpoint disabled for a long time.

My question is how do we get etcd to ignore or remove this custom content when upgrading to 3.6?

I've tried using the flag:
- --v2-deprecation=write-only-drop-data
but etcd 3.5.21 says its invalid when it starts up.

Perhaps our original migration process was invalid, however all the data is so old that it's truely irrelevant now.

Any guidance would be greatly appreciated.

[lukasertl]

I have exactly the same issue. I even restarted the cluster with V2 enabled to identify any old V2 data, then deleted the old data.

etcdutl check v2store still complains that there is V2 data but I'm at a loss how to identify it.

[jmhbnz]

Hey @Timothy-Dougherty & @lukasertl - Thanks for your question.

If you have a testing environment available, try starting the 3.5.x cluster with v2 enabled you can then try running ETCDCTL_API=2 ./etcdctl ls to review what custom content is stored, followed by ETCDCTL_API=2 ./etcdctl rm <content> to remove keys then ETCDCTL_API=2 ./etcdctl rmdir <content> to remove any directories.

What is the output from ETCDCTL_API=2 ./etcdctl ls?

@ahrtr Do you have any further ideas on what steps to run here?

[ahrtr]

but we migrated a long time ago to v3 and it might just been hanging around doing nothing.

Did you remove the flag --enable-v2 or set it to false a long time ago when you are still on etcd 3.5? If yes, then the reason should be what you already pointed out that there is not much traffic/writes, so the the old snapshot (which contains custom data) isn't rotated yet. So a simple workaround is to set a small value (i.e. 5) to --snapshot-count, and write several key/values to create a new v2 snapshot.

[kvaikla]

etcd cluster is successfully migrated to 3.6.4

ty ahrtr

[DTLP]

$ etcd --version
etcd Version: 3.5.22
Git SHA: 7862c85
Go Version: go1.23.11
Go OS/Arch: linux/amd64

$ sudo ls -la /var/lib/etcd/member/snap/
total 36820
drwx------. 2 etcd etcd     4096 Jul 31 11:03 .
drwx------. 4 etcd etcd     4096 Jul 31 11:02 ..
-rw-r--r--. 1 etcd etcd   894493 Jul 30 22:15 000000000004a80e-00000000592a90b0.snap
-rw-r--r--. 1 etcd etcd   894494 Jul 31 01:23 000000000004a80e-00000000592c1751.snap
-rw-r--r--. 1 etcd etcd   894494 Jul 31 04:34 000000000004a80e-00000000592d9df2.snap
-rw-r--r--. 1 etcd etcd   894494 Jul 31 07:42 000000000004a80e-00000000592f2493.snap
-rw-r--r--. 1 etcd etcd   894486 Jul 31 11:02 000000000004a81b-000000005930acd2.snap
-rw-------. 1 etcd etcd 44797952 Jul 31 11:33 db


$ sudo etcdutl check v2store --data-dir /var/lib/etcd/
Error: detected custom content in v2store

$ ETCDCTL_API=2 etcdctl ls
$ ETCDCTL_API=2 etcdctl user list
$ ETCDCTL_API=2 etcdctl role list
root

$ curl http://127.0.0.1:2379/v2/keys/?recursive=true
{"action":"get","node":{"dir":true}}

In my case upgrading to v3.5.22 and v3.6.4 doesn't help, I'm still getting "error":"detected disallowed custom content in v2store for stage --v2-deprecation=write-only"

[ahrtr]

@DTLP can you share your latest *snap file?

[lukasertl]

@DLTP beat me to it - I just wanted to reply that 3.6.4 didn't fix the issue for me.

The situation looks to me exactly like @DLTP describes. The only V2 data remaining is the root role, and I can't delete it.

[DTLP]

@ahrtr, sorry, I'm afraid I can't share it. Perhaps I can open it up myself and send you the contents of the v2store?

[lukasertl]

Hi @ahrtr - I've shared my snap-file with you. Can you work with this?

[ahrtr]

@mavpath

Just to double check:

• Have you removed the flag --enable-v2 or set it to false?
• If yes, can you share the complete etcd log?

Also refer to https://etcd.io/docs/v3.6/upgrades/upgrade_3_6/#upgrade-checklists

[mavpath]

I don't have the flag --enable-v2. Regarding the upgrade-checkllist - i don't have data that i need in v2store, there's only that root role that i don't care about and i was not aware of his existence until now and i cannot delete it.
Here are some logs from when trying to start with v3.6.4

{"level":"info","ts":"2025-11-21T14:30:17.927005+0200","caller":"flags/flag.go:114","msg":"recognized and used environment variable","variable-name":"ETCD_ADVERTISE_CLIENT_URLS","variable-value":"http://g2.domain.lan:2379"}
{"level":"info","ts":"2025-11-21T14:30:17.927733+0200","caller":"flags/flag.go:114","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_MODE","variable-value":"revision"}
{"level":"info","ts":"2025-11-21T14:30:17.927833+0200","caller":"flags/flag.go:114","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_RETENTION","variable-value":"1000"}
{"level":"info","ts":"2025-11-21T14:30:17.927965+0200","caller":"flags/flag.go:114","msg":"recognized and used environment variable","variable-name":"ETCD_DATA_DIR","variable-value":"/var/lib/etcd/default.etcd"}
{"level":"info","ts":"2025-11-21T14:30:17.928071+0200","caller":"flags/flag.go:114","msg":"recognized and used environment variable","variable-name":"ETCD_ELECTION_TIMEOUT","variable-value":"1500"}
{"level":"info","ts":"2025-11-21T14:30:17.928241+0200","caller":"flags/flag.go:114","msg":"recognized and used environment variable","variable-name":"ETCD_HEARTBEAT_INTERVAL","variable-value":"300"}
{"level":"info","ts":"2025-11-21T14:30:17.928365+0200","caller":"flags/flag.go:114","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_ADVERTISE_PEER_URLS","variable-value":"http://10.255.1.2:2380"}
{"level":"info","ts":"2025-11-21T14:30:17.928452+0200","caller":"flags/flag.go:114","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER","variable-value":"G1=http://10.255.1.1:2380,G2=http://10.255.1.2:2380,G3=http://10.255.1.3:2380,G4=http://10.255.1.4:2380"}
{"level":"info","ts":"2025-11-21T14:30:17.928545+0200","caller":"flags/flag.go:114","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_STATE","variable-value":"existing"}
{"level":"info","ts":"2025-11-21T14:30:17.928665+0200","caller":"flags/flag.go:114","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_TOKEN","variable-value":"production-cluster"}
{"level":"info","ts":"2025-11-21T14:30:17.928756+0200","caller":"flags/flag.go:114","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_CLIENT_URLS","variable-value":"http://10.2.0.182:2379, http://localhost:2379"}
{"level":"info","ts":"2025-11-21T14:30:17.928844+0200","caller":"flags/flag.go:114","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_PEER_URLS","variable-value":"http://10.255.1.2:2380"}
{"level":"info","ts":"2025-11-21T14:30:17.928951+0200","caller":"flags/flag.go:114","msg":"recognized and used environment variable","variable-name":"ETCD_LOGGER","variable-value":"zap"}
{"level":"info","ts":"2025-11-21T14:30:17.929044+0200","caller":"flags/flag.go:114","msg":"recognized and used environment variable","variable-name":"ETCD_MAX_WALS","variable-value":"5"}
{"level":"info","ts":"2025-11-21T14:30:17.929135+0200","caller":"flags/flag.go:114","msg":"recognized and used environment variable","variable-name":"ETCD_NAME","variable-value":"G2"}
{"level":"info","ts":"2025-11-21T14:30:17.929268+0200","caller":"flags/flag.go:114","msg":"recognized and used environment variable","variable-name":"ETCD_QUOTA_BACKEND_BYTES","variable-value":"4294965097"}
{"level":"warn","ts":"2025-11-21T14:30:17.929522+0200","caller":"embed/config.go:1209","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2025-11-21T14:30:17.929643+0200","caller":"etcdmain/etcd.go:64","msg":"Running: ","args":["/usr/bin/etcd"]}
{"level":"info","ts":"2025-11-21T14:30:17.929816+0200","caller":"etcdmain/etcd.go:107","msg":"server has already been initialized","data-dir":"/var/lib/etcd/default.etcd","dir-type":"member"}
{"level":"warn","ts":"2025-11-21T14:30:17.929915+0200","caller":"embed/config.go:1209","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2025-11-21T14:30:17.930010+0200","caller":"embed/etcd.go:138","msg":"configuring peer listeners","listen-peer-urls":["http://10.255.1.2:2380"]}
{"level":"info","ts":"2025-11-21T14:30:17.930337+0200","caller":"embed/etcd.go:146","msg":"configuring client listeners","listen-client-urls":["http://10.2.0.182:2379","http://localhost:2379"]}
{"level":"info","ts":"2025-11-21T14:30:17.931053+0200","caller":"embed/etcd.go:323","msg":"starting an etcd server","etcd-version":"3.6.4","git-sha":"5400cdc","go-version":"go1.23.11","go-os":"linux","go-arch":"amd64","max-cpu-set":2,"max-cpu-available":2,"member-initialized":true,"name":"G2","data-dir":"/var/lib/etcd/default.etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/etcd/default.etcd/member","force-new-cluster":false,"heartbeat-interval":"300ms","election-timeout":"1.5s","initial-election-tick-advance":true,"snapshot-count":10000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://10.255.1.2:2380"],"listen-peer-urls":["http://10.255.1.2:2380"],"advertise-client-urls":["http://g2.domain.lan:2379"],"listen-client-urls":["http://10.2.0.182:2379","http://localhost:2379"],"listen-metrics-urls":[],"experimental-local-address":"","cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"existing","initial-cluster-token":"","quota-backend-bytes":4294965097,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"feature-gates":"","initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-interval":"1m0s","auto-compaction-mode":"revision","auto-compaction-retention":"1µs","auto-compaction-interval":"1µs","discovery-url":"","discovery-proxy":"","discovery-token":"","discovery-endpoints":"","discovery-dial-timeout":"2s","discovery-request-timeout":"5s","discovery-keepalive-time":"2s","discovery-keepalive-timeout":"6s","discovery-insecure-transport":true,"discovery-insecure-skip-tls-verify":false,"discovery-cert":"","discovery-key":"","discovery-cacert":"","discovery-user":"","downgrade-check-interval":"5s","max-learners":1,"v2-deprecation":"write-only"}
{"level":"info","ts":"2025-11-21T14:30:17.931975+0200","logger":"bbolt","caller":"backend/backend.go:203","msg":"Opening db file (/var/lib/etcd/default.etcd/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: hashmap, ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 4724461606, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc00011e9b8}"}
{"level":"info","ts":"2025-11-21T14:30:17.934841+0200","logger":"bbolt","caller":"[email protected]/db.go:321","msg":"Opening bbolt db (/var/lib/etcd/default.etcd/member/snap/db) successfully"}
{"level":"info","ts":"2025-11-21T14:30:17.935003+0200","caller":"storage/backend.go:80","msg":"opened backend db","path":"/var/lib/etcd/default.etcd/member/snap/db","took":"3.189514ms"}
{"level":"info","ts":"2025-11-21T14:30:17.935171+0200","caller":"etcdserver/bootstrap.go:220","msg":"restore consistentIndex","index":5855784560}
{"level":"error","ts":"2025-11-21T14:30:19.725364+0200","caller":"etcdserver/bootstrap.go:409","msg":"illegal v2store content","error":"detected disallowed custom content in v2store for stage --v2-deprecation=write-only","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.recoverSnapshot\n\tgo.etcd.io/etcd/server/v3/etcdserver/bootstrap.go:409\ngo.etcd.io/etcd/server/v3/etcdserver.bootstrapBackend\n\tgo.etcd.io/etcd/server/v3/etcdserver/bootstrap.go:225\ngo.etcd.io/etcd/server/v3/etcdserver.bootstrap\n\tgo.etcd.io/etcd/server/v3/etcdserver/bootstrap.go:80\ngo.etcd.io/etcd/server/v3/etcdserver.NewServer\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:307\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\tgo.etcd.io/etcd/server/v3/embed/etcd.go:262\ngo.etcd.io/etcd/server/v3/etcdmain.startEtcd\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:207\ngo.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:114\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:272"}
{"level":"error","ts":"2025-11-21T14:30:19.728122+0200","caller":"etcdserver/server.go:309","msg":"bootstrap failed","error":"detected disallowed custom content in v2store for stage --v2-deprecation=write-only","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.NewServer\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:309\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\tgo.etcd.io/etcd/server/v3/embed/etcd.go:262\ngo.etcd.io/etcd/server/v3/etcdmain.startEtcd\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:207\ngo.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:114\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:272"}
{"level":"info","ts":"2025-11-21T14:30:19.728213+0200","caller":"embed/etcd.go:426","msg":"closing etcd server","name":"G2","data-dir":"/var/lib/etcd/default.etcd","advertise-peer-urls":["http://10.255.1.2:2380"],"advertise-client-urls":["http://g2.domain.lan:2379"]}
{"level":"info","ts":"2025-11-21T14:30:19.728447+0200","caller":"embed/etcd.go:428","msg":"closed etcd server","name":"G2","data-dir":"/var/lib/etcd/default.etcd","advertise-peer-urls":["http://10.255.1.2:2380"],"advertise-client-urls":["http://g2.domain.lan:2379"]}
{"level":"fatal","ts":"2025-11-21T14:30:19.728548+0200","caller":"etcdmain/etcd.go:183","msg":"discovery failed","error":"detected disallowed custom content in v2store for stage --v2-deprecation=write-only","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:183\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:272"}

[ahrtr]

I don't have the flag --enable-v2.

I meant that ensure the flag --enable-v2 and env variable ETCD_ENABLE_V2 aren't configured when you are still on v3.5.25. Please confirm this.

there's only that root role that i don't care about and i was not aware of his existence until now and i cannot delete it.

try ETCDCTL_API=3 etcdctl role remove root when you are still on v3.5.25. Afterwards, do the workaround in the section V2 Store in https://etcd.io/docs/v3.6/upgrades/upgrade_3_6/#upgrade-checklists. (Set --snapshot-count=1 for all etcd instances, and restart).

At last, run etcdutl check v2store (note: use etcdutl from v3.5.25), if there is no custom v2 data, then try to upgrade to v3.6.4 again.

Pls feedback whether it works.

[mavpath]

So it finally worked with this steps:

• stop the cluster
• add --snapshot-count=1 and the flag --enable-v2 so i can delete the user
• start the cluster
• delete role and user (user is deleted and the role it says auth: Cannot modify role root: is root role)

ETCDCTL_API=2 etcdctl user remove root
ETCDCTL_API=2 etcdctl role remove root

• add and delete some keys

for i in {1..100}; do etcdctl put test${i} hello;done
for i in {1..100}; do etcdctl del test${i};done

• stop the cluster
• remove the flag --enable-v2 and restore the previous value of --snapshot-count
• check the store

etcdutl check v2store --data-dir /var/lib/etcd/default.etcd/
No custom content found in v2store.

• start the cluster
• upgrade one by one to 3.6.4

[ahrtr]

• the flag --enable-v2 so i can delete the user

Correct. My previous comment try ETCDCTL_API=3 etcdctl role remove root isn't correct, it only removes roles/users in v3store. v2store isn't serving any request at all after removing --enable-v2.