|
1 | | -# Upgrading Gaia |
| 1 | +# Upgrade Gaia from v19.1.0 to v19.2.0 |
2 | 2 |
|
3 | | -This guide provides instructions for upgrading Gaia from v18.1.x. to v19.1.x. |
| 3 | +This release is consensus breaking. |
| 4 | +More than a third of the Hub’s validator set has already upgraded to this release. |
| 5 | +Thanks to the responsiveness and quick actions of these validators, **there is no further risk to security due to this vulnerability**. |
4 | 6 |
|
5 | | -This document describes the steps for validators, full node operators and relayer operators, to upgrade successfully for the Gaia v18 release. |
| 7 | +The rest of the Hub's validators will now have a coordinated upgrade scheduled for Thursday, September 5, 2024 at **block height 22051400**. |
| 8 | +Given that a third of the Hub’s validator set has already upgraded, **it is safe to upgrade to Gaia v19.2.0 before the coordinated upgrade**. |
6 | 9 |
|
7 | | -For more details on the release, please see the [release notes](https://github.com/cosmos/gaia/releases/tag/v19.1.0) |
| 10 | +### Release Details |
| 11 | +* https://github.com/cosmos/gaia/releases/tag/v19.2.0 |
| 12 | +* Chain upgrade height : `22051400`. Exact upgrade time can be checked [here](https://www.mintscan.io/cosmos/block/22051400). |
| 13 | +* Go version has been frozen at `1.22`. If you are going to build `gaiad` binary from source, make sure you are using the right GO version! |
8 | 14 |
|
9 | | -**Relayer Operators** for the Cosmos Hub and consumer chains, will also need to update to use [Hermes v1.10.0](https://github.com/informalsystems/hermes/releases/tag/v1.10.0) or higher. You may need to restart your relayer software after a major chain upgrade. |
| 15 | +# Performing the co-ordinated upgrade |
10 | 16 |
|
11 | | -## Release Binary |
| 17 | +This co-ordinated upgrades requires validators to stop their validators at `halt-height`, switch their binary to `v19.2.0` and restart their nodes with the new version. |
12 | 18 |
|
13 | | -Please use the correct release binary: `v19.1.0`. |
| 19 | +The exact sequence of steps depends on your configuration. Please take care to modify your configuration appropriately if your setup is not included in the instructions. |
14 | 20 |
|
15 | | -## Instructions |
| 21 | +# Manual steps |
16 | 22 |
|
17 | | -- [Upgrading Gaia](#upgrading-gaia) |
18 | | - - [Release Binary](#release-binary) |
19 | | - - [Instructions](#instructions) |
20 | | - - [On-chain governance proposal attains consensus](#on-chain-governance-proposal-attains-consensus) |
21 | | - - [Upgrade date](#upgrade-date) |
22 | | - - [Preparing for the upgrade](#preparing-for-the-upgrade) |
23 | | - - [Backups](#backups) |
24 | | - - [Testing](#testing) |
25 | | - - [Current runtime](#current-runtime) |
26 | | - - [Target runtime](#target-runtime) |
27 | | - - [Upgrade steps](#upgrade-steps) |
28 | | - - [Method I: Manual Upgrade](#method-i-manual-upgrade) |
29 | | - - [Method II: Upgrade using Cosmovisor](#method-ii-upgrade-using-cosmovisor) |
30 | | - - [Manually preparing the binary](#manually-preparing-the-binary) |
31 | | - - [Preparation](#preparation) |
32 | | - - [Expected upgrade result](#expected-upgrade-result) |
33 | | - - [Auto-Downloading the Gaia binary](#auto-downloading-the-gaia-binary) |
34 | | - - [Upgrade duration](#upgrade-duration) |
35 | | - - [Rollback plan](#rollback-plan) |
36 | | - - [Communications](#communications) |
37 | | - - [Risks](#risks) |
38 | | - - [Reference](#reference) |
| 23 | +## Step 1: Configure `halt-height` using v19.1.0 and restart the node. |
39 | 24 |
|
40 | | -## On-chain governance proposal attains consensus |
| 25 | +This upgrade requires `gaiad` halting execution at a pre-selected `halt-height`. |
41 | 26 |
|
42 | | -Once a software upgrade governance proposal is submitted to the Cosmos Hub, both a reference to this proposal and an `UPGRADE_HEIGHT` are added to the [release notes](https://github.com/cosmos/gaia/releases/tag/v19.1.0). |
43 | | -If and when this proposal reaches consensus, the upgrade height will be used to halt the "old" chain binaries. You can check the proposal on one of the block explorers or using the `gaiad` CLI tool. |
| 27 | +There are two mutually exclusive options for this stage: |
44 | 28 |
|
45 | | -## Upgrade date |
| 29 | +### Option 1: Set the halt height by modifying `app.toml` |
46 | 30 |
|
47 | | -The date/time of the upgrade is subject to change as blocks are not generated at a constant interval. You can stay up-to-date by checking the estimated estimated time until the block is produced one of the block explorers (e.g. https://www.mintscan.io/cosmos/blocks/`UPGRADE_HEIGHT`). |
| 31 | +* Stop the gaiad process. |
48 | 32 |
|
49 | | -## Preparing for the upgrade |
| 33 | +* Edit the application configuration file at `~/.gaia/config/app.toml` so that `halt-height` reflects the upgrade plan: |
50 | 34 |
|
51 | | -### Backups |
52 | | - |
53 | | -Prior to the upgrade, validators are encouraged to take a full data snapshot. Snapshotting depends heavily on infrastructure, but generally this can be done by backing up the `.gaia` directory. |
54 | | -If you use Cosmovisor to upgrade, by default, Cosmovisor will backup your data upon upgrade. See below [upgrade using cosmovisor](#method-ii-upgrade-using-cosmovisor) section. |
| 35 | +```toml |
| 36 | +# Note: Commitment of state will be attempted on the corresponding block. |
| 37 | +halt-height = 22051400 |
| 38 | +``` |
| 39 | +* Restart gaiad process |
55 | 40 |
|
56 | | -It is critically important for validator operators to back-up the `.gaia/data/priv_validator_state.json` file after stopping the gaiad process. This file is updated every block as your validator participates in consensus rounds. It is a critical file needed to prevent double-signing, in case the upgrade fails and the previous chain needs to be restarted. |
| 41 | +* Wait for the upgrade height and confirm that the node has halted |
57 | 42 |
|
58 | | -### Testing |
| 43 | +### Option 2: Restart the `gaiad` binary with command line flags |
59 | 44 |
|
60 | | -For those validator and full node operators that are interested in ensuring preparedness for the impending upgrade, you can run a [v19 Local Testnet](https://github.com/cosmos/testnets/tree/master/local) or join in our [Cosmos Hub Public Testnet](https://github.com/cosmos/testnets/tree/master/public). |
| 45 | +* Stop the gaiad process. |
61 | 46 |
|
62 | | -### Current runtime |
| 47 | +* Do not modify `app.toml`. Restart the `gaiad` process with the flag `--halt-height`: |
| 48 | +```shell |
| 49 | +gaiad start --halt-height 22051400 |
| 50 | +``` |
63 | 51 |
|
64 | | -The Cosmos Hub mainnet network, `cosmoshub-4`, is currently running [Gaia v18.1.0](https://github.com/cosmos/gaia/releases/v18.1.0). We anticipate that operators who are running on v18.1.0, will be able to upgrade successfully. Validators are expected to ensure that their systems are up to date and capable of performing the upgrade. This includes running the correct binary and if building from source, building with the appropriate `go` version. |
| 52 | +* Wait for the upgrade height and confirm that the node has halted |
65 | 53 |
|
66 | | -### Target runtime |
| 54 | +Upon reaching the `halt-height` you need to replace the `v19.1.0` gaiad binary with the new `gaiad v19.2.0` binary and remove the `halt-height` constraint. |
| 55 | +Depending on your setup, you may need to set `halt-height = 0` in your `app.toml` before resuming operations. |
| 56 | +```shell |
| 57 | + git clone https://github.com/cosmos/gaia.git |
| 58 | +``` |
67 | 59 |
|
68 | | -The Cosmos Hub mainnet network, `cosmoshub-4`, will run **[Gaia v19.1.0](https://github.com/cosmos/gaia/releases/tag/v19.1.0)**. Operators _**MUST**_ use this version post-upgrade to remain connected to the network. The new version requires `go v1.22` to build successfully. |
| 60 | +## Step 2: Build and start the v19.2.0 binary |
69 | 61 |
|
70 | | -## Upgrade steps |
| 62 | +### Remember to revert `gaiad` configurations |
| 63 | +* Reset `halt-height = 0` option in the `app.toml` or |
| 64 | +* Remove it from start parameters of the gaiad binary before restarting the node |
71 | 65 |
|
72 | | -There are 2 ways to upgrade a node: |
| 66 | +We recommend you perform a backup of your data directory before switching to `v19.2.0`. |
73 | 67 |
|
74 | | -- Manual upgrade |
75 | | -- Upgrade using [Cosmovisor](https://pkg.go.dev/cosmossdk.io/tools/cosmovisor) |
76 | | - - Either by manually preparing the new binary |
77 | | - - Or by using the auto-download functionality (this is not yet recommended) |
| 68 | +```shell |
| 69 | +cd $HOME/gaia |
| 70 | +git pull |
| 71 | +git fetch --tags |
| 72 | +git checkout v19.2.0 |
| 73 | +make install |
| 74 | + |
| 75 | +# verify install |
| 76 | +gaiad version |
| 77 | +# v19.2.0 |
| 78 | +``` |
78 | 79 |
|
79 | | -If you prefer to use Cosmovisor to upgrade, some preparation work is needed before upgrade. |
| 80 | +```shell |
| 81 | +gaiad start # starts the v19.2.0 node |
| 82 | +``` |
80 | 83 |
|
81 | | -### Method I: Manual Upgrade |
| 84 | +# Cosmovisor steps |
82 | 85 |
|
83 | | -Make sure **Gaia v18.1.0** is installed by either downloading a [compatible binary](https://github.com/cosmos/gaia/releases/tag/v18.1.0), or building from source. Check the required version to build this binary in the `Makefile`. |
| 86 | +## Prerequisite: Alter systemd service configuration |
84 | 87 |
|
85 | | -Run Gaia v18.1.0 till upgrade height, the node will panic: |
| 88 | +Disable automatic restart of the node service. To do so please alter your `gaiad.service` file configuration and set appropriate lines to following values. |
86 | 89 |
|
87 | | -```shell |
88 | | -ERR UPGRADE "v19" NEEDED at height: <UPGRADE_HEIGHT>: upgrade to v19 and applying upgrade "v19" at height:<UPGRADE_HEIGHT> |
89 | 90 | ``` |
| 91 | +Restart=no |
90 | 92 |
|
91 | | -Stop the node, and switch the binary to **Gaia v19.1.0** and re-start by `gaiad start`. |
| 93 | +Environment="DAEMON_ALLOW_DOWNLOAD_BINARIES=false" |
| 94 | +Environment="DAEMON_RESTART_AFTER_UPGRADE=false" |
| 95 | +``` |
92 | 96 |
|
93 | | -It may take several minutes to a few hours until validators with a total sum voting power > 2/3 to complete their node upgrades. After that, the chain can continue to produce blocks. |
| 97 | +After that you will need to run `sudo systemctl daemon-reload` to apply changes in the service configuration. |
94 | 98 |
|
95 | | -### Method II: Upgrade using Cosmovisor |
| 99 | +There is no need to restart the node yet; these changes will get applied during the node restart in the next step. |
96 | 100 |
|
97 | | -#### Manually preparing the binary |
| 101 | +## Setup Cosmovisor |
| 102 | +### Create the updated gaiad binary of v19.2.0 |
98 | 103 |
|
99 | | -##### Preparation |
| 104 | +### Remember to revert `gaiad` configurations |
| 105 | +* Reset `halt-height = 0` option in the `app.toml` or |
| 106 | +* Remove it from start parameters of the gaiad binary before starting the node |
100 | 107 |
|
101 | | -- Install the latest version of Cosmovisor (`1.5.0`): |
| 108 | +#### Go to gaiad directory if present else clone the repository |
102 | 109 |
|
103 | 110 | ```shell |
104 | | -go install cosmossdk.io/tools/cosmovisor/cmd/cosmovisor@latest |
105 | | -cosmovisor version |
106 | | -# cosmovisor version: v1.5.0 |
| 111 | + git clone https://github.com/cosmos/gaia.git |
107 | 112 | ``` |
108 | 113 |
|
109 | | -- Create a `cosmovisor` folder inside `$GAIA_HOME` and move Gaia `v18.1.0` into `$GAIA_HOME/cosmovisor/genesis/bin`: |
| 114 | +#### Follow these steps if gaiad repo already present |
110 | 115 |
|
111 | 116 | ```shell |
112 | | -mkdir -p $GAIA_HOME/cosmovisor/genesis/bin |
113 | | -cp $(which gaiad) $GAIA_HOME/cosmovisor/genesis/bin |
| 117 | + cd $HOME/gaia |
| 118 | + git pull |
| 119 | + git fetch --tags |
| 120 | + git checkout v19.2.0 |
| 121 | + make install |
114 | 122 | ``` |
115 | 123 |
|
116 | | -- Build Gaia `v19.1.0`, and move gaiad `v19.1.0` to `$GAIA_HOME/cosmovisor/upgrades/v19/bin` |
117 | | - |
| 124 | +#### Check the new gaiad version, verify the latest commit hash |
118 | 125 | ```shell |
119 | | -mkdir -p $GAIA_HOME/cosmovisor/upgrades/v19/bin |
120 | | -cp $(which gaiad) $GAIA_HOME/cosmovisor/upgrades/v19/bin |
| 126 | + $ gaiad version --long |
| 127 | + name: gaiad |
| 128 | + server_name: gaiad |
| 129 | + version: 19.2.0 |
| 130 | + commit: <commit-hash> |
| 131 | + ... |
121 | 132 | ``` |
122 | 133 |
|
123 | | -At this moment, you should have the following structure: |
| 134 | +#### Or check checksum of the binary if you decided to download it |
124 | 135 |
|
125 | | -```shell |
126 | | -. |
127 | | -├── current -> genesis or upgrades/<name> |
128 | | -├── genesis |
129 | | -│ └── bin |
130 | | -│ └── gaiad # old: v18.1.0 |
131 | | -└── upgrades |
132 | | - └── v19 |
133 | | - └── bin |
134 | | - └── gaiad # new: v19.1.0 |
135 | | -``` |
| 136 | +Checksums can be found on the official release page: |
| 137 | +* https://github.com/cosmos/gaia/releases/tag/v19.2.0 |
136 | 138 |
|
137 | | -- Export the environmental variables: |
| 139 | +The checksums file is located in the `Assets` section: |
| 140 | +* e.g. [SHA256SUMS-v19.2.0.txt](https://github.com/cosmos/gaia/releases/download/v19.2.0/SHA256SUMS-v19.2.0.txt) |
138 | 141 |
|
139 | 142 | ```shell |
140 | | -export DAEMON_NAME=gaiad |
141 | | -# please change to your own gaia home dir |
142 | | -# please note `DAEMON_HOME` has to be absolute path |
143 | | -export DAEMON_HOME=$GAIA_HOME |
144 | | -export DAEMON_RESTART_AFTER_UPGRADE=true |
| 143 | +$ shasum -a 256 gaiad-v19.2.0-linux-amd64 |
| 144 | +<checksum> gaiad-v19.2.0-linux-amd64 |
145 | 145 | ``` |
146 | 146 |
|
147 | | -- Start the node: |
148 | | - |
| 147 | +### Copy the new gaiad (v19.2.0) binary to cosmovisor current directory |
149 | 148 | ```shell |
150 | | -cosmovisor run start --x-crisis-skip-assert-invariants --home $DAEMON_HOME |
| 149 | + cp $GOPATH/bin/gaiad ~/.gaiad/cosmovisor/current/bin |
151 | 150 | ``` |
152 | 151 |
|
153 | | -Skipping the invariant checks is strongly encouraged since it decreases the upgrade time significantly and since there are some other improvements coming to the crisis module in the next release of the Cosmos SDK. |
154 | | - |
155 | | -##### Expected upgrade result |
| 152 | +### Restore service file settings |
156 | 153 |
|
157 | | -When the upgrade block height is reached, Gaia will panic and stop: |
158 | | - |
159 | | -This may take a few minutes. |
160 | | -After upgrade, the chain will continue to produce blocks when validators with a total sum voting power > 2/3 complete their node upgrades. |
161 | | - |
162 | | -#### Auto-Downloading the Gaia binary |
163 | | - |
164 | | -## Upgrade duration |
165 | | - |
166 | | -The upgrade may take a few minutes to complete because cosmoshub-4 participants operate globally with differing operating hours and it may take some time for operators to upgrade their binaries and connect to the network. |
167 | | - |
168 | | -## Rollback plan |
169 | | - |
170 | | -During the network upgrade, core Cosmos teams will be keeping an ever vigilant eye and communicating with operators on the status of their upgrades. During this time, the core teams will listen to operator needs to determine if the upgrade is experiencing unintended challenges. In the event of unexpected challenges, the core teams, after conferring with operators and attaining social consensus, may choose to declare that the upgrade will be skipped. |
171 | | - |
172 | | -Steps to skip this upgrade proposal are simply to resume the cosmoshub-4 network with the (downgraded) v18.1.0 binary using the following command: |
173 | | - |
174 | | -```shell |
175 | | -gaiad start --unsafe-skip-upgrade <UPGRADE_HEIGHT> |
| 154 | +If you are using a service file, restore the previous `Restart` settings in your service file: |
176 | 155 | ``` |
| 156 | +Restart=On-failure |
| 157 | +``` |
| 158 | +Reload the service control `sudo systemctl daemon-reload`. |
177 | 159 |
|
178 | | -Note: There is no particular need to restore a state snapshot prior to the upgrade height, unless specifically directed by core Cosmos teams. |
179 | | - |
180 | | -Important: A social consensus decision to skip the upgrade will be based solely on technical merits, thereby respecting and maintaining the decentralized governance process of the upgrade proposal's successful YES vote. |
181 | | - |
182 | | -## Communications |
183 | | - |
184 | | -Operators are encouraged to join the `#cosmos-hub-validators-verified` channel of the Cosmos Hub Community Discord. This channel is the primary communication tool for operators to ask questions, report upgrade status, report technical issues, and to build social consensus should the need arise. This channel is restricted to known operators and requires verification beforehand. Requests to join the `#cosmos-hub-validators-verified` channel can be sent to the `#general-support` channel. |
185 | | - |
186 | | -## Risks |
187 | | - |
188 | | -As a validator performing the upgrade procedure on your consensus nodes carries a heightened risk of double-signing and being slashed. The most important piece of this procedure is verifying your software version and genesis file hash before starting your validator and signing. |
189 | | - |
190 | | -The riskiest thing a validator can do is discover that they made a mistake and repeat the upgrade procedure again during the network startup. If you discover a mistake in the process, the best thing to do is wait for the network to start before correcting it. |
| 160 | +# Revert `gaiad` configurations |
191 | 161 |
|
192 | | -## Reference |
| 162 | +Depending on which path you chose for Step 1, either: |
193 | 163 |
|
194 | | -[Join Cosmos Hub Mainnet](https://github.com/cosmos/mainnet) |
| 164 | +* Reset `halt-height = 0` option in the `app.toml` or |
| 165 | +* Remove it from start parameters of the gaiad binary and start node again |
0 commit comments