Thanks to the Agoric team for the wonderful help and documentation.
The overall experience was really good we all testnet participants enjoyed it a lot. Late night and early morning sync ups for emergency updates and time-based tasks all.
A few of my mistakes taught me to be more careful while resetting the validator. Especially during the emergency update task 5. I built the new software but did not stop the validator due to this my validator got jailed for the double sign but it helped to vote for 67% genesis consensus block generation start. Later I have we reinstall completely from scratch to get a new operators, node keys, and addresses. Moniker can be the same.
Task 1: Setup your node & sign Genesis block
I followed the below steps to join the new testnet by updating genesis file and resetting the local data directory
service ag-chain-cosmos stop
export GIT_BRANCH=agorictest-10
cd $HOME
rm -r agoric-sdk
git clone https://github.com/Agoric/agoric-sdk -b $GIT_BRANCH
cd agoric-sdk
yarn install
yarn build
(cd packages/cosmic-swingset && make)
curl https://testnet.agoric.net/network-config > chain.json
chainName=`jq -r .chainName < chain.json`
echo $chainName
curl https://testnet.agoric.net/genesis.json > $HOME/.ag-chain-cosmos/config/genesis.json
ag-chain-cosmos unsafe-reset-all
peers=$(jq ‘.peers | join(“,”)’ < chain.json)
seeds=$(jq ‘.seeds | join(“,”)’ < chain.json)
echo $peers
echo $seeds
sed -i.bak ‘s/^log_level/# log_level/’ $HOME/.ag-chain-cosmos/config/config.toml
sed -i.bak -e “s/^seeds *=.*/seeds = $seeds/; s/^persistent_peers *=.*/persistent_peers = $peers/” $HOME/.ag-chain-cosmos/config/config.toml
service ag-chain-cosmos start
Task 2: Install Prometheus Node Exporter (OS)
During Phase 2, Agoric will actively evaluate CPU and memory use during the course of the incentivized testnet.
Install node exporter https://prometheus.io/docs/guides/node-exporter/
Create service file use below format
sudo tee <<EOF >/dev/null /etc/systemd/system/node-exporter.service
[Unit]
Description=Prometheus Node Exporter
After=network-online.target[Service]
User=$USER
ExecStart=$HOME/node_exporter-1.1.2.linux-amd64/node_exporter
Restart=on-failure
RestartSec=3
LimitNOFILE=4096[Install]
WantedBy=multi-user.target
EOF# Check the contents of the file, especially User, Environment and ExecStart lines
cat /etc/systemd/system/node-exporter.service
Expose node exporter port
iptables -A INPUT -p tcp — dport 9100 -j ACCEPT
Expose Cosmos SDK metrics, Agoric VM (SwingSet) metrics and Tendermint metrics ports respectively
iptables -A INPUT -p tcp — dport 1317 -j ACCEPT
iptables -A INPUT -p tcp — dport 9464 -j ACCEPT
iptables -A INPUT -p tcp — dport 26660 -j ACCEPT
Task 3: Keep 80% uptime during load bursts
Throughout Phase 2, Agoric will send load bursts across network to test activity. This task is all about ensuring that validators maintain greater than 80% uptime throughout the phase. Agoric will make sure to announce any scheduled load bursts. Ensure that your node is active and running during scheduled load bursts.
Task 4: Cause your validator to be jailed and then recover
Your validator will be jailed after missing 50 blocks out of the previous 100 blocks (~10 minutes). Jailing should be planned during pauses in load bursts to not affect critical uptime. Jailed validators will need to rejoin the network.
Following are the steps I executed to cause Jail and Unjail
Step 1: Stop validator and monitor the missing blocks at
https://testnet.explorer.agoric.net/validator/agoricvaloper18sgcxclppmpe62rkv5yu6a3ylj802pn8qzmfzv
once the validator missed > 50 blocks my node was Jailed. This I verified in the list of inactive nodes.
https://testnet.explorer.agoric.net/validators/inactive
root@e:~# date
Mon Apr 19 20:11:13 IST 2021
root@e:~# sudo systemctl stop ag-chain-cosmos
root@e:~# date
Mon Apr 19 20:17:46 IST 2021
Step 2: Set the chain name
# Set the chainName value again
chainName=`curl https://testnet.agoric.net/network-config | jq -r .chainName`
# Confirm value: should be something like agoricdev-N
echo $chainName
Step 3: Start the validator
sudo systemctl start ag-chain-cosmos
monitor the logs using below command
journalctl -u ag-chain-cosmos -f
Expected log entries are
Apr 19 20:29:19 e ag-chain-cosmos[292323]: 2021–04–19T14:59:19.676Z launch-chain: Launching SwingSet kernel
Apr 19 20:29:19 e ag-chain-cosmos[292323]: Prometheus scrape endpoint: http://0.0.0.0:9464/metrics
After 10 mins you should start seeing the block commits
Step 4: Unjail validator using below command
root@e:~# ag-cosmos-helper tx slashing unjail — broadcast-mode=block — from=almuezza — chain-id=$chainName — gas=auto — gas-adjustment=1.4
Enter keyring passphrase:
gas estimate: 80120
{“body”:{“messages”:[{“@type”:”/cosmos.slashing.v1beta1.MsgUnjail”,”validator_addr”:”agoricvaloper18sgcxclppmpe62rkv5yu6a3ylj802pn8qzmfzv”}],”memo”:””,”timeout_height”:”0",”extension_options”:[],”non_critical_extension_options”:[]},”auth_info”:{“signer_infos”:[],”fee”:{“amount”:[],”gas_limit”:”80120",”payer”:””,”granter”:””}},”signatures”:[]}
confirm transaction before signing and broadcasting [y/N]: y
{“height”:”336100",”txhash”:”FF59CF27AA0A4F940E02D7A8B1AE08C1615E8AB3780C28F9AC694A1386FC562D”,”codespace”:””,”code”:0,”data”:”0A080A06756E6A61696C”,”raw_log”:”[{\”events\”:[{\”type\”:\”message\”,\”attributes\”:[{\”key\”:\”action\”,\”value\”:\”unjail\”},{\”key\”:\”module\”,\”value\”:\”slashing\”},{\”key\”:\”sender\”,\”value\”:\”agoricvaloper18sgcxclppmpe62rkv5yu6a3ylj802pn8qzmfzv\”}]}]}]”,”logs”:[{“msg_index”:0,”log”:””,”events”:[{“type”:”message”,”attributes”:[{“key”:”action”,”value”:”unjail”},{“key”:”module”,”value”:”slashing”},{“key”:”sender”,”value”:”agoricvaloper18sgcxclppmpe62rkv5yu6a3ylj802pn8qzmfzv”}]}]}],”info”:””,”gas_wanted”:”80120",”gas_used”:”55747",”tx”:null,”timestamp”:””}
Step 5: Verify the validator is showing up in the active list.
https://testnet.explorer.agoric.net/validators
Note: I lost around 5–6 mins in between the unjail start. I was not aware that the service takes up 10 mins to start the sync process as there were no log entries I restarted the validator.
Task 5: Emergency Upgrade: Restart node
This task is focused on simulating an emergency upgrade on our future network. Coordinated Event: On Monday, April 19th @ (2021–04–19T19:00:00Z) we will halt our chain nodes, release new software, and export a new genesis.json. It is up to the community to how fast they can start a new network with that configuration.
Following are the steps
- Stop validator node (Ctrl +c or service stop)
- Copy old config dir for safety (
/home/user/.ag-chain-cosmos/config) - Do
unsafe-reset-all(This will not change your validator id / keys) - Build new version of the software
- Replace old genesis with new one in your config dir
- Start server and monitor chain startup.
Actual commands
service ag-chain-cosmos stop
export GIT_BRANCH=agorictest-10
cd $HOME
rm -r agoric-sdk
git clone https://github.com/Agoric/agoric-sdk -b $GIT_BRANCH
cd agoric-sdk
yarn install
yarn build
(cd packages/cosmic-swingset && make)
curl https://testnet.agoric.net/network-config > chain.json
chainName=`jq -r .chainName < chain.json`
echo $chainName
curl https://testnet.agoric.net/genesis.json > $HOME/.ag-chain-cosmos/config/genesis.json
ag-chain-cosmos unsafe-reset-all
peers=$(jq ‘.peers | join(“,”)’ < chain.json)
seeds=$(jq ‘.seeds | join(“,”)’ < chain.json)
echo $peers
echo $seeds
sed -i.bak ‘s/^log_level/# log_level/’ $HOME/.ag-chain-cosmos/config/config.toml
sed -i.bak -e “s/^seeds *=.*/seeds = $seeds/; s/^persistent_peers *=.*/persistent_peers = $peers/” $HOME/.ag-chain-cosmos/config/config.toml
service ag-chain-cosmos start
Task 6: Write & publish an article
Write and publish a 400–750 word article demonstrating knowledge of Agoric’s technology and fit within the DeFi ecosystem. Here are a few ideas!
- Agoric’s Technology Stack (Zoe, ERTP, Offer Safety, Object Capabilities, etc.)
- JavaScript Components (AMMs, Vaults & Loans, Covered Call, OTC Desk, etc.)
- Future of interchain via IBC (Agoric connecting to Cosmos Hub chains)
- Incentivized Testnet (Process, experience, your challenge task submissions, etc.)
Task 7: Build a system performance analysis tool for validators
What tool do you expect to want during the stress test phase to maximize the load your system can handle? Build a tool that analyzes resource usage of a system so that you can tune your hardware and/or bring attention to the Agoric engineering team to improve performance. Be sure to indicate any compatibility requirements for others to use your tool.
If possible, include the JavaScript smart contract execution layer in the tool’s analysis!
For this task, I am working on the network, infrastructure, and transaction analysis on Splunk but it is still incomplete at the time of writing this article.
