Masternode WATCHDOG_EXPIRED Troubleshooting

In Innovacoin, and other coins with masternode features alike, WATCHDOG_EXPIRED status is the most confusing part. So what does that mean? 
It is vital to talk a little about masternodes and their role before explaining this issue.

Introduction

Masternodes are special nodes that run Innova wallet program and participate in various decision making services. The services mainly facilitate decentralized governance of the cryptosystem.

When masternode is started the first time, its status is PRE_ENABLED. Soon after it should change to ENABLED. ENABLED status is the happy status that we want to see as masternode owners. Because to get rewards, masternode’s status must be ENABLED. But sometimes we see the status as WATCHDOG_EXPIRED. What does that mean?

When masternode is started it is added to the list of masternodes in the network. Now this masternode is like a member of board of directory that makes decisions. But unlike board of directory in a company, masternodes are not in the same room talking face-to-face. They are making decisions over the network with their votes. To stay as a member, the masternode pings every a few minutes and makes it known that it is still active. Besides pinging, the masternode should also be receiving messages from other masternodes (like pings that mentioned above).

When some time lapses without getting updates from this masternode, its status changes to WATCHDOG_EXPIRED.  So, what to do when you see that status is WATCHDOG_EXPIRED?

Troubleshooting
  • 1. Check if the masternode wallet is running with this command in vps:

    innova-cli getinfo

    If it prints json data then it is running, otherwise it gives the following error: “error: couldn't connect to server”.
    If it is not running start it with this command:

    innovad -daemon

  • 2. Check if your masternode port is open and it is receiving messages. The easiest way of checking is running this command in your local wallet Debug Console:

    masternode connect ip:port

  • 3. Check if the masternode is started in vps:

    innova-cli masternode status

    It should print json message containing the following line:
    "status": "Masternode successfully started"

    You may get the following message as well which is fine:
    "status": "Sync in progress. Must wait until sync is complete to start Masternode"

    If masternode has not started successfully status will be something like this:
    "status": "Not capable masternode: ..."
    Then you have to fix it accordingly.

  • 4. Check if sentinel is working:

    cd /root/.innovacore/sentinel
    ./venv/bin/python bin/sentinel.py

    If they run silently then sentinel is working
  • 5. Check if sentinel is set to cron (so that it is fired frequently):

    crontab -l

    You should see something like this in your cron list:
    * * * * * cd /root/.innovacore/sentinel && ./venv/bin/python bin/sentinel.py 2>&1 >>...

    If it is not listed add it to the cron:

    crontab -l > tempcron
    echo "* * * * * cd /root/.innovacore/sentinel && ./venv/bin/python bin/sentinel.py 2>&1 >> sentinel-cron.log" >> tempcron
    crontab tempcron
    rm tempcron

If everything above is fine when you checked or you resolved all of them but still you see WATCHDOG_EXPIRED, what to do now? Give it some time. Status update in distributed system is not instant. It takes time for the wallet to sync updated masternode list. If you leave it for some time it should change to ENABLED.
Sometimes it shows WATCHDOG_EXPIRED and ENABLED again. Does that mean you are in trouble? No, it is ok.
What should you not do when you see WATCHDOG_EXPIRED? Do not freak out and do not restart your masternode without troubleshooting. When you restart your masternode, it will be set to the end of the queue in getting rewards. Restart you masternode only if it is necessary.
You can also monitor your masternode status here: https://masternodes.online/monitoring/

For technical help go to our community

Button

We are building mobile wallets for IOS and Android.
We want to launch official Innova applications for IOS and Android, so you can easily control your Innova coins on the go. The exact time will be updated.