How to Monitor Your Avail Validator Node

Telemetry

Maintaining a healthy, stable, and secure ecosystem involves closely monitoring network operations and performance. In the case of Avail nodes, they come equipped with built-in low-level telemetry components that automatically gather and transmit detailed information about each node's operation within the network. This backend telemetry data is streamed in real-time to a designated endpoint, allowing you to observe information regarding your nodes behavior.

Avail project has made available a public telemetry (opens in a new tab) for anyone to view the status of their nodes. This should not replace your own monitoring and alerts. However it is a key tool that anyone can use to view the status of their nodes at anytime.

Avail telemetry has been added to the chain specification file and should automatically send telemetry data to Avail public telemetry (opens in a new tab). You can however also add the telemetry --telemetry-url 'ws://telemetry.avail.so:8001/submit/ 0' to your avail node start command if your node is not visible on the public telemetry.

You can also use --name <Validator Node Name> to set the name that will display on telemetry. Without this it will display a random node name on the telemetry.

For example:

./data-avail --validator \
    --port 30333 \
    --base-path `pwd`/data \
    --chain `pwd`/chainspec.raw.json \
    --name AvailNode \
    --telemetry-url 'ws://telemetry.avail.tools:8001/submit/ 0'

Monitoring with Prometheus & Grafana

Prometheus serves as a monitoring platform that gathers metrics from monitored targets through scraping their metrics HTTP endpoints. In contrast, Grafana acts as a visualization dashboard specifically designed to present the collected data in an easily understandable and visually appealing manner.

The below guide is for setting up a local Prometheus & Grafana instance on the same server as your Avail node. If you're already experienced with Prometheus & Grafana and want to connect your Avail node to an existing Prometheus installation on a different server, include the --prometheus-external flag in your Avail node's startup command.

Install Prometheus

Install prometheus and prometheus node exporter.

sudo apt-get install -y prometheus prometheus-node-exporter

Create the prometheus.yml config file

cat > $HOME/prometheus.yml << EOF
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  # - "first.rules"
  # - "second.rules"

scrape_configs:
  - job_name: "prometheus"
    scrape_interval: 5s
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: "avail_node"
    scrape_interval: 5s
    static_configs:
      - targets: ["localhost:9615"]
  - job_name: node
    static_configs:
      - targets: ['localhost:9100']
EOF

Move prometheus.yml to the correct location

sudo mv $HOME/prometheus.yml /etc/prometheus/prometheus.yml

Update the file permissions

sudo chmod 644 /etc/prometheus/prometheus.yml

Ensure Prometheus starts automatically

sudo systemctl enable prometheus.service prometheus-node-exporter.service

Restart Prometheus to activate latest settings

sudo systemctl restart prometheus.service prometheus-node-exporter.service

Check the status, ensure Prometheus has started without errors

sudo systemctl status prometheus.service prometheus-node-exporter.service

Install Grafana

wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" > grafana.list
sudo mv grafana.list /etc/apt/sources.list.d/grafana.list
sudo apt-get update && sudo apt-get install -y grafana

Ensure Grafana starts automatically

sudo systemctl enable grafana-server.service

Start Grafana

sudo systemctl start grafana-server.service

Check the status, ensure Grafana has started without errors

sudo systemctl status grafana-server.service

Setup Grafana Dashboard

Ensure port 3000 is open, example of adding to ubuntu firewall

sudo ufw allow 3000/tcp

In your browser navigate to http://<your validators ip address>:3000. The default login username and password is admin/admin

You will be asked to reset your password, please write it down or remember the password as you will need it for the next login.

You will need to create a datasource. Navigate to Home->Connections->Data sources

Click on Add data source

Click on Prometheus

Set URL to "localhost:9090", then test and save the connection

Navigate back to your home page, on the top right in the menu select Import dashboard

Import the Avail Node Metrics file

You will have a new dashboard that opens and that you can use to monitor your node

Upgrade your validator Chill your validator