Hi,
I just updated out nodes (2 testnet and 1 mainnet nodes) to v.4.2.3 today - however, I can only see 1 of the testnet nodes “syscon-dev-03” on the dashboard. I was expecting to also see “syscon-dev-02” and “syscon-dev-01”.
I figured it could be due to some catching up, but it has been some time now since I updated them, and I still don’t see them.
How do I get them to show again? (our mainnet node is also a baker).
Hello
Did you make the node public, and gave a name for them?
In case you did not, the nodes would be available with a generated ID.
Regards,
Concordium Tech Support
I haven’t changed any settings to the nodes, I just updated them from their previous versions to the latest.
Before I did that they were visible on the dashboard.
What platform/ OS have you the missing ones?
They are all on Ubuntu 20.04.
One of the systems (I have 2 systems running) is running both a mainnet node and a testnet node.
The other is only running a testnet node.
On the system running 2 nodes, the testnet node is visible on the dashboard while the mainnet node isn’t.
The other testnet node running on it’s own system isn’t visible either.
Previously all 3 nodes were visible.
Good day Petlan
Well, if you can’t see the node on the dashboard, there are two possibilities - either it works but the collector service doesn’t work, or the node really doesn’t work. To check if the node is working, run any concordium-client command, such as concordium-client block show. If you get a response without an error, then your node is working and just needs to restart the collector. To restart the collector, just enter a command, which depends on your system and type of installation.
Restart collector-service:
for deb: sudo systemctl restart concordium-mainnet-node-collector
for docker: docker-compose -f mainnet-node.yaml restart mainnet-node-collector
then the node should appear on the dashboard if it works without errors and if there are no errors in the configuration. If after running this command, the node does not appear on the dashboard, then you should check the logs. This is usually done with the command sudo journalctl -u concordium-mainnet-node -f
Both testnet nodes give a response to “block show” though still it’s only one of the nodes visible on the dashbard.
The mainnet node however doesn’t response on port 10000 nor 8888.
I have restarted the collector for each node type on both systems but it doesn’t seem to change anything.
Log for the mainnet node:
Aug 15 13:26:41 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8853]: [INFO concordium_node::p2p::maintenance] Listening on 10.0.20.211:8888
Aug 15 13:26:41 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8853]: [INFO concordium_node] Starting consensus layer
Aug 15 13:26:41 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8853]: [INFO concordium_node::plugins::consensus] Starting up the consensus thread
Aug 15 13:26:41 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8853]: [INFO concordium_node::consensus_ffi::consensus] Starting up the consensus layer
Aug 15 13:26:57 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8853]: [ERROR concordium_node::consensus_ffi::ffi] TreeState: Database invariant violation: Could not read last finalized block
Aug 15 13:26:57 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8853]: [ERROR concordium_node::consensus_ffi::ffi] External: Database invariant violation: Could not read last finalized block
Aug 15 13:26:57 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8853]: Error: Database invariant violation. See logs for details.
Aug 15 13:26:58 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Main process exited, code=exited, status=1/FAILURE
Aug 15 13:26:58 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Failed with result 'exit-code'.
Aug 15 13:26:58 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Consumed 12.046s CPU time.
Aug 15 13:27:18 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Scheduled restart job, restart counter is at 88.
Aug 15 13:27:18 peter-Virtual-Machine systemd[1]: Stopped Concordium Mainnet Node.
Aug 15 13:27:18 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Consumed 12.046s CPU time.
Aug 15 13:27:18 peter-Virtual-Machine systemd[1]: Started Concordium Mainnet Node.
Aug 15 13:27:18 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8911]: [INFO concordium_node::utils] Starting up concordium_node version 4.2.3!
Aug 15 13:27:18 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8911]: [INFO concordium_node::utils] Application data directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/data
Aug 15 13:27:18 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8911]: [INFO concordium_node::utils] Application config directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/config
Aug 15 13:27:18 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8911]: [INFO concordium_node::utils] Network: enabled
Aug 15 13:27:18 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8911]: [INFO concordium_node::utils] Log level: info
Aug 15 13:27:18 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8911]: [INFO concordium_node::p2p::maintenance] My Node ID is a1d3f6431f7b520d
Aug 15 13:27:18 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8911]: [INFO concordium_node::p2p::maintenance] Listening on 10.0.20.211:8888
Aug 15 13:27:18 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8911]: [INFO concordium_node] Starting consensus layer
Aug 15 13:27:18 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8911]: [INFO concordium_node::plugins::consensus] Starting up the consensus thread
Aug 15 13:27:18 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8911]: [INFO concordium_node::consensus_ffi::consensus] Starting up the consensus layer
Aug 15 13:27:33 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8911]: [ERROR concordium_node::consensus_ffi::ffi] TreeState: Database invariant violation: Could not read last finalized block
Aug 15 13:27:33 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8911]: [ERROR concordium_node::consensus_ffi::ffi] External: Database invariant violation: Could not read last finalized block
Aug 15 13:27:33 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8911]: Error: Database invariant violation. See logs for details.
Aug 15 13:27:34 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Main process exited, code=exited, status=1/FAILURE
Aug 15 13:27:34 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Failed with result 'exit-code'.
Aug 15 13:27:34 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Consumed 12.101s CPU time.
Aug 15 13:27:54 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Scheduled restart job, restart counter is at 89.
Aug 15 13:27:54 peter-Virtual-Machine systemd[1]: Stopped Concordium Mainnet Node.
Aug 15 13:27:54 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Consumed 12.101s CPU time.
Aug 15 13:27:54 peter-Virtual-Machine systemd[1]: Started Concordium Mainnet Node.
Aug 15 13:27:54 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8960]: [INFO concordium_node::utils] Starting up concordium_node version 4.2.3!
Aug 15 13:27:54 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8960]: [INFO concordium_node::utils] Application data directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/data
Aug 15 13:27:54 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8960]: [INFO concordium_node::utils] Application config directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/config
Aug 15 13:27:54 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8960]: [INFO concordium_node::utils] Network: enabled
Aug 15 13:27:54 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8960]: [INFO concordium_node::utils] Log level: info
Aug 15 13:27:54 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8960]: [INFO concordium_node::p2p::maintenance] My Node ID is a1d3f6431f7b520d
Aug 15 13:27:54 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8960]: [INFO concordium_node::p2p::maintenance] Listening on 10.0.20.211:8888
Aug 15 13:27:54 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8960]: [INFO concordium_node] Starting consensus layer
Aug 15 13:27:54 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8960]: [INFO concordium_node::plugins::consensus] Starting up the consensus thread
Aug 15 13:27:54 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8960]: [INFO concordium_node::consensus_ffi::consensus] Starting up the consensus layer
Aug 15 13:28:09 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8960]: [ERROR concordium_node::consensus_ffi::ffi] TreeState: Database invariant violation: Could not read last finalized block
Aug 15 13:28:09 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8960]: [ERROR concordium_node::consensus_ffi::ffi] External: Database invariant violation: Could not read last finalized block
Aug 15 13:28:09 peter-Virtual-Machine concordium-mainnet-node-4.2.3[8960]: Error: Database invariant violation. See logs for details.
Aug 15 13:28:10 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Main process exited, code=exited, status=1/FAILURE
Aug 15 13:28:10 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Failed with result 'exit-code'.
Aug 15 13:28:10 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Consumed 12.010s CPU time.
Aug 15 13:28:30 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Scheduled restart job, restart counter is at 90.
Aug 15 13:28:30 peter-Virtual-Machine systemd[1]: Stopped Concordium Mainnet Node.
Aug 15 13:28:30 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Consumed 12.010s CPU time.
Aug 15 13:28:30 peter-Virtual-Machine systemd[1]: Started Concordium Mainnet Node.
Aug 15 13:28:30 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9011]: [INFO concordium_node::utils] Starting up concordium_node version 4.2.3!
Aug 15 13:28:30 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9011]: [INFO concordium_node::utils] Application data directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/data
Aug 15 13:28:30 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9011]: [INFO concordium_node::utils] Application config directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/config
Aug 15 13:28:30 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9011]: [INFO concordium_node::utils] Network: enabled
Aug 15 13:28:30 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9011]: [INFO concordium_node::utils] Log level: info
Aug 15 13:28:30 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9011]: [INFO concordium_node::p2p::maintenance] My Node ID is a1d3f6431f7b520d
Aug 15 13:28:30 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9011]: [INFO concordium_node::p2p::maintenance] Listening on 10.0.20.211:8888
Aug 15 13:28:30 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9011]: [INFO concordium_node] Starting consensus layer
Aug 15 13:28:30 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9011]: [INFO concordium_node::plugins::consensus] Starting up the consensus thread
Aug 15 13:28:30 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9011]: [INFO concordium_node::consensus_ffi::consensus] Starting up the consensus layer
Aug 15 13:28:45 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9011]: [ERROR concordium_node::consensus_ffi::ffi] TreeState: Database invariant violation: Could not read last finalized block
Aug 15 13:28:45 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9011]: [ERROR concordium_node::consensus_ffi::ffi] External: Database invariant violation: Could not read last finalized block
Aug 15 13:28:45 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9011]: Error: Database invariant violation. See logs for details.
Aug 15 13:28:45 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Main process exited, code=exited, status=1/FAILURE
Aug 15 13:28:45 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Failed with result 'exit-code'.
Aug 15 13:28:45 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Consumed 11.989s CPU time.
Aug 15 13:29:05 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Scheduled restart job, restart counter is at 91.
Aug 15 13:29:05 peter-Virtual-Machine systemd[1]: Stopped Concordium Mainnet Node.
Aug 15 13:29:05 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Consumed 11.989s CPU time.
Aug 15 13:29:05 peter-Virtual-Machine systemd[1]: Started Concordium Mainnet Node.
Aug 15 13:29:05 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9047]: [INFO concordium_node::utils] Starting up concordium_node version 4.2.3!
Aug 15 13:29:05 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9047]: [INFO concordium_node::utils] Application data directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/data
Aug 15 13:29:05 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9047]: [INFO concordium_node::utils] Application config directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/config
Aug 15 13:29:05 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9047]: [INFO concordium_node::utils] Network: enabled
Aug 15 13:29:05 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9047]: [INFO concordium_node::utils] Log level: info
Aug 15 13:29:05 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9047]: [INFO concordium_node::p2p::maintenance] My Node ID is a1d3f6431f7b520d
Aug 15 13:29:05 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9047]: [INFO concordium_node::p2p::maintenance] Listening on 10.0.20.211:8888
Aug 15 13:29:06 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9047]: [INFO concordium_node] Starting consensus layer
Aug 15 13:29:06 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9047]: [INFO concordium_node::plugins::consensus] Starting up the consensus thread
Aug 15 13:29:06 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9047]: [INFO concordium_node::consensus_ffi::consensus] Starting up the consensus layer
Aug 15 13:29:21 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9047]: [ERROR concordium_node::consensus_ffi::ffi] TreeState: Database invariant violation: Could not read last finalized block
Aug 15 13:29:21 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9047]: [ERROR concordium_node::consensus_ffi::ffi] External: Database invariant violation: Could not read last finalized block
Aug 15 13:29:21 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9047]: Error: Database invariant violation. See logs for details.
Aug 15 13:29:21 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Main process exited, code=exited, status=1/FAILURE
Aug 15 13:29:21 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Failed with result 'exit-code'.
Aug 15 13:29:21 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Consumed 12.148s CPU time.
Aug 15 13:29:41 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Scheduled restart job, restart counter is at 92.
Aug 15 13:29:41 peter-Virtual-Machine systemd[1]: Stopped Concordium Mainnet Node.
Aug 15 13:29:41 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Consumed 12.148s CPU time.
Aug 15 13:29:41 peter-Virtual-Machine systemd[1]: Started Concordium Mainnet Node.
Aug 15 13:29:41 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9099]: [INFO concordium_node::utils] Starting up concordium_node version 4.2.3!
Aug 15 13:29:41 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9099]: [INFO concordium_node::utils] Application data directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/data
Aug 15 13:29:41 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9099]: [INFO concordium_node::utils] Application config directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/config
Aug 15 13:29:41 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9099]: [INFO concordium_node::utils] Network: enabled
Aug 15 13:29:41 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9099]: [INFO concordium_node::utils] Log level: info
Aug 15 13:29:41 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9099]: [INFO concordium_node::p2p::maintenance] My Node ID is a1d3f6431f7b520d
Aug 15 13:29:41 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9099]: [INFO concordium_node::p2p::maintenance] Listening on 10.0.20.211:8888
Aug 15 13:29:41 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9099]: [INFO concordium_node] Starting consensus layer
Aug 15 13:29:41 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9099]: [INFO concordium_node::plugins::consensus] Starting up the consensus thread
Aug 15 13:29:41 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9099]: [INFO concordium_node::consensus_ffi::consensus] Starting up the consensus layer
Aug 15 13:29:56 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9099]: [ERROR concordium_node::consensus_ffi::ffi] TreeState: Database invariant violation: Could not read last finalized block
Aug 15 13:29:56 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9099]: [ERROR concordium_node::consensus_ffi::ffi] External: Database invariant violation: Could not read last finalized block
Aug 15 13:29:56 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9099]: Error: Database invariant violation. See logs for details.
Aug 15 13:29:56 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Main process exited, code=exited, status=1/FAILURE
Aug 15 13:29:56 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Failed with result 'exit-code'.
Aug 15 13:29:56 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Consumed 11.994s CPU time.
Aug 15 13:30:17 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Scheduled restart job, restart counter is at 93.
Aug 15 13:30:17 peter-Virtual-Machine systemd[1]: Stopped Concordium Mainnet Node.
Aug 15 13:30:17 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Consumed 11.994s CPU time.
Aug 15 13:30:17 peter-Virtual-Machine systemd[1]: Started Concordium Mainnet Node.
Aug 15 13:30:17 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9156]: [INFO concordium_node::utils] Starting up concordium_node version 4.2.3!
Aug 15 13:30:17 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9156]: [INFO concordium_node::utils] Application data directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/data
Aug 15 13:30:17 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9156]: [INFO concordium_node::utils] Application config directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/config
Aug 15 13:30:17 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9156]: [INFO concordium_node::utils] Network: enabled
Aug 15 13:30:17 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9156]: [INFO concordium_node::utils] Log level: info
Aug 15 13:30:17 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9156]: [INFO concordium_node::p2p::maintenance] My Node ID is a1d3f6431f7b520d
Aug 15 13:30:17 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9156]: [INFO concordium_node::p2p::maintenance] Listening on 10.0.20.211:8888
Aug 15 13:30:17 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9156]: [INFO concordium_node] Starting consensus layer
Aug 15 13:30:17 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9156]: [INFO concordium_node::plugins::consensus] Starting up the consensus thread
Aug 15 13:30:17 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9156]: [INFO concordium_node::consensus_ffi::consensus] Starting up the consensus layer
Aug 15 13:30:32 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9156]: [ERROR concordium_node::consensus_ffi::ffi] TreeState: Database invariant violation: Could not read last finalized block
Aug 15 13:30:32 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9156]: [ERROR concordium_node::consensus_ffi::ffi] External: Database invariant violation: Could not read last finalized block
Aug 15 13:30:32 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9156]: Error: Database invariant violation. See logs for details.
Aug 15 13:30:32 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Main process exited, code=exited, status=1/FAILURE
Aug 15 13:30:32 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Failed with result 'exit-code'.
Aug 15 13:30:32 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Consumed 12.029s CPU time.
Aug 15 13:30:53 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Scheduled restart job, restart counter is at 94.
Aug 15 13:30:53 peter-Virtual-Machine systemd[1]: Stopped Concordium Mainnet Node.
Aug 15 13:30:53 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Consumed 12.029s CPU time.
Aug 15 13:30:53 peter-Virtual-Machine systemd[1]: Started Concordium Mainnet Node.
Aug 15 13:30:53 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9204]: [INFO concordium_node::utils] Starting up concordium_node version 4.2.3!
Aug 15 13:30:53 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9204]: [INFO concordium_node::utils] Application data directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/data
Aug 15 13:30:53 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9204]: [INFO concordium_node::utils] Application config directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/config
Aug 15 13:30:53 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9204]: [INFO concordium_node::utils] Network: enabled
Aug 15 13:30:53 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9204]: [INFO concordium_node::utils] Log level: info
Aug 15 13:30:53 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9204]: [INFO concordium_node::p2p::maintenance] My Node ID is a1d3f6431f7b520d
Aug 15 13:30:53 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9204]: [INFO concordium_node::p2p::maintenance] Listening on 10.0.20.211:8888
Aug 15 13:30:53 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9204]: [INFO concordium_node] Starting consensus layer
Aug 15 13:30:53 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9204]: [INFO concordium_node::plugins::consensus] Starting up the consensus thread
Aug 15 13:30:53 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9204]: [INFO concordium_node::consensus_ffi::consensus] Starting up the consensus layer
Aug 15 13:31:08 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9204]: [ERROR concordium_node::consensus_ffi::ffi] TreeState: Database invariant violation: Could not read last finalized block
Aug 15 13:31:08 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9204]: [ERROR concordium_node::consensus_ffi::ffi] External: Database invariant violation: Could not read last finalized block
Aug 15 13:31:08 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9204]: Error: Database invariant violation. See logs for details.
Aug 15 13:31:08 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Main process exited, code=exited, status=1/FAILURE
Aug 15 13:31:08 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Failed with result 'exit-code'.
Aug 15 13:31:08 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Consumed 12.135s CPU time.
Aug 15 13:31:28 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Scheduled restart job, restart counter is at 95.
Aug 15 13:31:28 peter-Virtual-Machine systemd[1]: Stopped Concordium Mainnet Node.
Aug 15 13:31:28 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Consumed 12.135s CPU time.
Aug 15 13:31:28 peter-Virtual-Machine systemd[1]: Started Concordium Mainnet Node.
Aug 15 13:31:29 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9240]: [INFO concordium_node::utils] Starting up concordium_node version 4.2.3!
Aug 15 13:31:29 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9240]: [INFO concordium_node::utils] Application data directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/data
Aug 15 13:31:29 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9240]: [INFO concordium_node::utils] Application config directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/config
Aug 15 13:31:29 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9240]: [INFO concordium_node::utils] Network: enabled
Aug 15 13:31:29 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9240]: [INFO concordium_node::utils] Log level: info
Aug 15 13:31:29 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9240]: [INFO concordium_node::p2p::maintenance] My Node ID is a1d3f6431f7b520d
Aug 15 13:31:29 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9240]: [INFO concordium_node::p2p::maintenance] Listening on 10.0.20.211:8888
Aug 15 13:31:29 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9240]: [INFO concordium_node] Starting consensus layer
Aug 15 13:31:29 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9240]: [INFO concordium_node::plugins::consensus] Starting up the consensus thread
Aug 15 13:31:29 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9240]: [INFO concordium_node::consensus_ffi::consensus] Starting up the consensus layer
Aug 15 13:31:48 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9240]: [ERROR concordium_node::consensus_ffi::ffi] TreeState: Database invariant violation: Could not read last finalized block
Aug 15 13:31:48 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9240]: [ERROR concordium_node::consensus_ffi::ffi] External: Database invariant violation: Could not read last finalized block
Aug 15 13:31:48 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9240]: Error: Database invariant violation. See logs for details.
Aug 15 13:31:48 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Main process exited, code=exited, status=1/FAILURE
Aug 15 13:31:48 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Failed with result 'exit-code'.
Aug 15 13:31:48 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Consumed 12.102s CPU time.
Aug 15 13:32:09 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Scheduled restart job, restart counter is at 96.
Aug 15 13:32:09 peter-Virtual-Machine systemd[1]: Stopped Concordium Mainnet Node.
Aug 15 13:32:09 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Consumed 12.102s CPU time.
Aug 15 13:32:09 peter-Virtual-Machine systemd[1]: Started Concordium Mainnet Node.
Aug 15 13:32:09 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9303]: [INFO concordium_node::utils] Starting up concordium_node version 4.2.3!
Aug 15 13:32:09 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9303]: [INFO concordium_node::utils] Application data directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/data
Aug 15 13:32:09 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9303]: [INFO concordium_node::utils] Application config directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/config
Aug 15 13:32:09 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9303]: [INFO concordium_node::utils] Network: enabled
Aug 15 13:32:09 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9303]: [INFO concordium_node::utils] Log level: info
Aug 15 13:32:09 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9303]: [INFO concordium_node::p2p::maintenance] My Node ID is a1d3f6431f7b520d
Aug 15 13:32:09 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9303]: [INFO concordium_node::p2p::maintenance] Listening on 10.0.20.211:8888
Aug 15 13:32:09 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9303]: [INFO concordium_node] Starting consensus layer
Aug 15 13:32:09 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9303]: [INFO concordium_node::plugins::consensus] Starting up the consensus thread
Aug 15 13:32:09 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9303]: [INFO concordium_node::consensus_ffi::consensus] Starting up the consensus layer
Aug 15 13:32:24 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9303]: [ERROR concordium_node::consensus_ffi::ffi] TreeState: Database invariant violation: Could not read last finalized block
Aug 15 13:32:24 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9303]: [ERROR concordium_node::consensus_ffi::ffi] External: Database invariant violation: Could not read last finalized block
Aug 15 13:32:24 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9303]: Error: Database invariant violation. See logs for details.
Aug 15 13:32:24 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Main process exited, code=exited, status=1/FAILURE
Aug 15 13:32:24 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Failed with result 'exit-code'.
Aug 15 13:32:24 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Consumed 12.018s CPU time.
Aug 15 13:32:44 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Scheduled restart job, restart counter is at 97.
Aug 15 13:32:44 peter-Virtual-Machine systemd[1]: Stopped Concordium Mainnet Node.
Aug 15 13:32:44 peter-Virtual-Machine systemd[1]: concordium-mainnet-node.service: Consumed 12.018s CPU time.
Aug 15 13:32:44 peter-Virtual-Machine systemd[1]: Started Concordium Mainnet Node.
Aug 15 13:32:45 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9353]: [INFO concordium_node::utils] Starting up concordium_node version 4.2.3!
Aug 15 13:32:45 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9353]: [INFO concordium_node::utils] Application data directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/data
Aug 15 13:32:45 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9353]: [INFO concordium_node::utils] Application config directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/config
Aug 15 13:32:45 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9353]: [INFO concordium_node::utils] Network: enabled
Aug 15 13:32:45 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9353]: [INFO concordium_node::utils] Log level: info
Aug 15 13:32:45 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9353]: [INFO concordium_node::p2p::maintenance] My Node ID is a1d3f6431f7b520d
Aug 15 13:32:45 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9353]: [INFO concordium_node::p2p::maintenance] Listening on 10.0.20.211:8888
Aug 15 13:32:45 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9353]: [INFO concordium_node] Starting consensus layer
Aug 15 13:32:45 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9353]: [INFO concordium_node::plugins::consensus] Starting up the consensus thread
Aug 15 13:32:45 peter-Virtual-Machine concordium-mainnet-node-4.2.3[9353]: [INFO concordium_node::consensus_ffi::consensus] Starting up the consensus layer
The logs let us know that there are an error in your local blockchain database, so you need to manually delete a couple of files. Follow this instruction:
In a nutshell: stop the node, navigate to /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/data/database-v4 and remove the last blockstate-3 and treestate-3, after this try to start the node.
I just wanted to try that, but the system with both a mainnet and a testnet node, doesn’t even have a database-v4 folder inside the data folder.
I tried to check that the nodes were indeed updated and in both cases it says it’s running the latest version 4.2.3.
What type of installation did you use ? Docker or deb package ?
You can find the location of a file or folder in your file system by using the command
sudo find / -name "blockstate-?.dat"
or
sudo find / -name "treestate-?"
Run these commands and you will determine where the files to be deleted are located. A normal user doesn’t have rights to navigate to this directory, so I usually use superuser root to repair database.
I’m using deb package for all nodes.
The files seems to be located at: var/lib/private/concordium-{id}/data/database-v4
I will try to delete the latest and see what happens.
ok, then if you didn’t change the configuration, your database is by default in /var/lib/concordium-9dd9…
Use superuser to navigate to this directory and delete what I wrote earlier.
If you still can’t find the database directory, I would ask you to provide a screenshot of the executing find command I provided in the previous answer.
Here is a screenshot from the system only running one testnet node.
I deleted the latest blockstate/treestate set in /var/lib/concordium-b6078154d6717e909ce0da4a45a25151b592824f31624b755900a74429e3073d/data/database-v4
The node still doesn’t appear on the dashboard though and the reason might be the following (I have no idea). I get this if I try to get status of the service:
This image only shows the right part of the message, because it’s too wide to include the whole message. The service is described as concordium-testnet-node-4.0.11 in the message, which is odd because it’s updated to 4.2.3 unless that number shown in the status message, has nothing to do with the version of the node.
Let’s start in order.
When you execute the systemctl start/restart concordium-mainnet-node.service command, the system accesses the config file, which contains the current variables and everything necessary to start the node process. You can look at this file with the command sudo systemctl cat concordium-mainnet-node.service. If you claim that instead of 4.2.3 you are running version 4.1.1, then in the execstart field you have something like ExecStart=/usr/bin/concordium-mainnet-node-4.1.1
Upgrading changes this file and 4.1.1 should be replaced automatically by 4.2.3 , although I have seen cases where this has to be changed manually.
If you downloaded the update and installed it, then check for the executable file concordium-mainnet-node-4.2.3 at /usr/bin/ , and also check that the config says ExecStart=/usr/bin/concordium-mainnet-node- 4.2.3
After checking, run the node and look at the logs.
It was correctly, though odd, that the node was 4.0.11 although I could have sworn I upgraded it.
I performed the update process and the node is now running as expected - showing on the testnet dashboard.
I did also increase the size of the partition for this system, since it appeared that it was about to run low, which could be part of cause of the db corruption I guess.
That was one of the systems - I will do a little more digging on the other system (that runs both testnet and mainnet nodes), to see if I can figure out what the problem is here.
UPDATE: I now also got the mainnet node to appear on the dashboard. For some reason, the baker-credentials.json file was empty at the new location (I’m guessing that some update along the way, made these new locations?).
So I copied the content of the old one into the new one and restarted the node.
UPDATE 2: I now got the last testnet node to appear as well. I believe another restart of the collector service for this node, did the last part.
Thanks for the help 
I’m glad your problem is solved.
About the empty baker-credentials.json file - if it is in var/lib/private/conc-{id} - it seems to me that it is a kind of link, this file is created automatically and it is far from being necessary to make changes in this file.
To become a baker, you have to just put your baker keys anywhere in your user’s home folder and then edit the concordium-mainnet-node service to add the path to this file according to this instruction: Run a baker node on Ubuntu — Concordium documentation
