Mind the gap! How to fix a huge sync gap

ardevd
4 min readSep 7, 2021

--

Many of you have probably suffered from a large sync gap recently, leading to a lack of activity and rewards. The Bobcat Discord has been flooded with user complaints and the support channels are swamped and hence, a lot of hotspots are stuck with a steep mountain to climb. The current sync status can be viewed from the diagnoser web interface, or by using your favorite data transfer tool. In this example, we will be using curl. Firstly, lets look at what data the status endpoint .

$ curl 192.168.1.70/status.json
{
“status”: “synced”,
“gap”: “1”,
“miner_height”: “1001004”,
“blockchain_height”: “1001005”,
“epoch”: “25829”
}

The status indicates that we are fully synced with the block chain. The gap tells us we are currently one block behind the head of the blockchain. The miner height indicates the latest synced block of the hotspot, and the blockchain height indicates the current head of the blockchain. You can verify this through the helium API directly.

$ curl https://api.helium.io/v1/blocks/height
{"data”:{“height”:1001005}}

The Helium firmware updates include an updated blockchain snapshot and are pushed out at regular intervals, but blockchain snaphots are seemingly created and uploaded several times day and a newly introduced fastsync utility can be used to download and load the latest snapshot without having to update the firmware. If you’re stuck with a sync gap larger than 400 blocks, the fastsync utility can save you hours, days or weeks of syncing. The Helium network is currently experiencing massive growth and sync speeds seem to suffer because of it. As a result, a large number of Helium hotspots are currently way off sync and end up being stuck for days or weeks waiting to catch up the blockchain. The reason hotspots suddenly find themselves several thousand blocks behind the chain seems to stem from a bug in the Helium firmware where the local blockchain database gets deleted, forcing a resync. The issue has been reported to Helium and they are hopefully working on a fix.

UPDATE 10.09.2021: Bobcat has released an OTA that adds the fastsync command to the Bobcat diagnoser web interface. You can now trigger fastsync by clicking the fastsync button there. Personally, I still recommend reading the rest of this guide and doing it from the command line. It’s more reliable and you get a better sense of what’s happening. But at least now you have options.

Fast sync available through the Bobcat diagnoser

Before you proceed, the following requirements need to be fufilled

  • A computer with Windows 10, Linux or MacOS installed
  • Network connectivity with your hotspot
  • The hotspot needs to be online and syncing. Verify this through the diagnoser
  • The hotspot needs a reliable network connection. Weak WiFi connections may result in the snapshot downloading timing out and failing.

Once you’re ready, open up a terminal/command line prompt and issue the following command.

curl -i -H "Authorization:Basic Ym9iY2F0Om1pbmVy" --request POST http://192.168.1.20/admin/fastsync

Make sure to update the IP and set it to whatever local IP your hotspot has.

The Authorization header is a standard basic auth formatted header with the credentials bobcat:miner base64 encoded. The command will send a request to the fastsync endpoint which will download and load the latest available helium snapshot which should get your miner back in sync.

When running that command it will hang for about 1–2 minutes while the hotspot downloads the snapshot from AWS. Once the download is complete you will get a confirmation message before the hotspots proceeds with rebooting twice as it loads the snapshot data. This typically takes about 10 minutes. Have some patience, drink a cup of coffee, and dont touch your miner or the diagnoser.

This is how it should look like once the command is done running:

curl -i -H "Authorization:Basic Ym9iY2F0Om1pbmVy" --request POST http://192.168.1.150/admin/fastsync

HTTP/1.1 200 OK
Content-Type: text/plain;
charset=UTF-8
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN

Syncing your miner, please leave your power on.

Once the LED is green, check back on the sync status and you’ll hopefully be back in sync with the blockchain, ready to provide coverage to the people’s network (and earning a bit of HNT in the process).

I presume the fastsync feature will become more widely known in the coming days and weeks now that’s it has been officialy added to the diagnoser. I hope this guide saved you some time and taught you a thing or two in the process.

Oh, and did you know you can easily fast sync, resync, reset, reboot and even diagnose your hotspot using my python library? Check out bobcatpy.

If you have any questions, you can find me on Discord, or leave a comment here.

If I just saved your hotspot, feel free to leave a HNT tip at 13zSxUxreCdYyG3nhPo5xVK51SGTeV63gCy96r3vc3EeghGrxVu

Cheers!

--

--

Responses (6)