[Bug]: Device becomes unreachable Heltec & LILYGO

[TheDomanNow]

Category

BLE, WiFi

Hardware

Heltec V3, Other

Firmware Version

2.3.12

Description

Hi,

currently I have 2 "base" nodes in good position & with constant power, ROUTER_CLIENT mode. They are connected via WiFi to internet and have MQTT enabled. After like 2-3 days they become unresponsive via IP address (HTTP or Android app). This is not an network (WiFi) issue, also not a hardware issue (1x Heltec V3.1 & 1x LILYGO T3S3).
In this unreachable state it is possible to send a message via other node that is currently connected, so the radio is working. After power cycling the device all is back to normal, but all messages on router are lost...

I'm guessing this is related with current faulty implementation of sleep (?). Is there any workaround of this??? Is it possible to completely turn off all power saving "features"?

I see 2 solutions (probably both should be added):

1. Adding a toggle, like "Turn off all power saving things", that really turns off all power saving "features".
2. Adding auto power cycle feature, that after X time or at desired hour (internet needed) restarts the device.

Update;
after many tests it turned out that for BLE it was an Android app issue - fixed some time ago...
...as for WiFi... there is a bug that persists in many FW releases (2.4, 2.5...), it just drops WiFi after few hours and doesn't attempt to reconnect - not fixed and ignored. More information below...

BR, Doman.

[madeofstown]

Is the MQTT channel very active? About how many nodes are you seeing on your node list? Busy MQTT channels are know to overload these devices.

[HarukiToreda]

Power save mode settings are not on by default, if you did not set them, the device will not go to lite sleep as you're suspecting. The only thing that will turn off is the screen and it should not be that either. I believe mqtt is flooding the node with too many nodes. To work around this, I suggest using admin to be able to control your node from other nodes via the app.

[TheDomanNow]

Is the MQTT channel very active? About how many nodes are you seeing on your node list? Busy MQTT channels are know to overload these devices.

No it isn't. NodeDB is almost full or full, but there are like 30 nodes active at a time, sending maybe 1x msg/hour. It's not that.

Power save mode settings are not on by default, if you did not set them, the device will not go to lite sleep as you're suspecting. The only thing that will turn off is the screen and it should not be that either. I believe mqtt is flooding the node with too many nodes. To work around this, I suggest using admin to be able to control your node from other nodes via the app.

So, you're saying that settings like:
"Wait for Bluetooth duration"
"Light sleep duration"
"Minimum wake time"
are NOT active, unless "Enable power saving mode" is turned ON? Why aren't they greyed out then?

Like said above, MQTT doesn't have much traffic and nodes, it's not flooded. It's not that.

I still think it's related to some power saving "feature". Or... any other ideas?

[madeofstown]

Is the MQTT channel very active? About how many nodes are you seeing on your node list? Busy MQTT channels are know to overload these devices.

No it isn't. NodeDB is almost full or full, but there are like 30 nodes active at a time, sending maybe 1x msg/hour. It's not that.

Power save mode settings are not on by default, if you did not set them, the device will not go to lite sleep as you're suspecting. The only thing that will turn off is the screen and it should not be that either. I believe mqtt is flooding the node with too many nodes. To work around this, I suggest using admin to be able to control your node from other nodes via the app.

So, you're saying that settings like: "Wait for Bluetooth duration" "Light sleep duration" "Minimum wake time" are NOT active, unless "Enable power saving mode" is turned ON? Why aren't they greyed out then?

Like said above, MQTT doesn't have much traffic and nodes, it's not flooded. It's not that.

I still think it's related to some power saving "feature". Or... any other ideas?

Unless you are able to get some logs I think this is going to stay a mystery as most people are assuming it's MQTT related. (I have issues I'm procrastinating on getting logs for)

[bsacharski]

Having similar issue with my Heltec V3 running 2.5.15.

There is very little traffic on the LoRa side and my node is set to ignore the MQTT messages.
Is there anything I could do to help diagnose the issue?

[TheDomanNow]

Not really, this issue is mostly ignored... try FW 2.5.5, it seems somewhat stable with WiFi.

[JaraLowell]

Have same issue with firmware 2.5.10 and up, where IP connection simply drops after x hours; seems be around six for Station G2, issue not there on 2,5,9 where config more or less same. WiFI, MQTT default channel plus extra one. The extra added to push out the neighbor info but not does so... Nor a topic for here as will just do that by software seeing it seems get fade out. Sleep settings on default for all versions. as in disabled. Blue tooth disabled. Heartbeat let disabled

[TheDomanNow]

This issue persists in most of the FW releases and it's ignored all the time. The workaround would be to restart WiFi module after X time or device restart after X time... but no, it's not in plans. So yes, it's a Schrödingers feature/bug and it is/isn't.

[JaraLowell]

This issue is not apparent in 2.5.5.e182ae7, it started after this that after x hours when connected via a program to monitor the node the wifi cuts out and then completely closes. Not tested if serial then still would work but would presume it does. Nor do i per say know how to catch the issue to gather a log of what might of has happened. But for something that for emergencies should always be accessible this is obviously a no go per my view.

[TheDomanNow]

I'm now running Heltec v3 with 2.5.9.93, it was ok on this FW, right? Or should I roll back to 2.5.5?

[JaraLowell]

i my self had no issue in 2.5.9.93 but we had some nodes in neighborhood that showed an halt in relaying packages after x hours. if that is due local db issues or other is yet unknown. so we for past 8 days went to 2.5.5 to see if that keeps running without interfering and or how long so. Our setup being simple two nodes about 10km apart both with same version and both connected to mqtt so to wifi. And basically having a program connected to it near 24/7 (LoraLog from my creation)

[TheDomanNow]

Great info, thanks. I've got 1 node with 2.5.6.d55c08d running for 26 days and WiFi seems stable, it just works, it's TLORA_T3_S3 tho.

If you'll find any new 2.5 FW working stable with WiFi, please let me know here. It's quite important information for me.

Update;
2.5.6 is also bad - WiFi drops... more below...

[TheDomanNow]

A little update; I did 24h tests on few nodes with few FW. WiFi+MQTT+no_smartphone (standalone, CLIENT role), Heltec V3 + T-LoRa T3-S3.
2.5.15 - WiFi went offline after few hours on 3 nodes
2.5.9 - WiFi went offline after few hours on 3 nodes
2.5.6 - WiFi went offline after few hours on 3 nodes (I was mistaken, I thought it works well)
2.5.5 - WiFi was ok after 24h on 2 nodes, 3rd node acted weirdly so I discard results from this one

Conclusions;

• after WiFi drops the node works fine with LoRa, of course MQTT stops working and there is no way to remote manage it... it is strictly an WiFi issue
• everything after FW 2.5.5 is broken, and the issue persists in latest stable releases (last 2.5.15)
• it's hard to get some logs, because after few hours serial freezes, but from what I've seen the WiFi drops and there is no attempt to reconnect... just like that... I think it might be connected to low WiFi activity and/or some power saving features...

...so... for now I recommend sticking with FW 2.5.5.

[fifieldt]

2.5.15 added an improvement to wifi: #5443 -- have you tried 2.5.20 or so?

[TheDomanNow]

I've tried 2.5.15 with no success... BUT, 2.5.18 works great for me.

[leedowthwaite]

Hi, I recently updated to 2.5.20 and I'm having the issue that the device becomes unreachable on wifi. I was hoping to use the Heltec as an indoor second node to act as an MQTT relay for the "real" nodes outside, but of course as soon as I enable wifi the device cannot be reached again. None of the client interfaces seem to work, and when I directly navigate to the IP address via a browser I only see a JSON dump instead of an admin interface. Am I missing something important here?

[TheDomanNow]

Try 2.5.18, it works like a charm for me.

[TheDomanNow]

Hello, I've got an update;
I've found the most stable release with WiFi+MQTT working w/o any issues, it's 2.5.18. I have 3 nodes running it for a few months now and all is fine, there is no issues related to MQTT and/or WiFi, there is almost no issues at all, even when in standalone mode (no phone connected).

Still, I'd like to see a option to auto-reboot the node after X time form start (if not time sync) or at selected hour+day (time sync).

Anyway, the time is slowly coming to switch to 2.6. Does any one have some experience with 2.6 and WiFi+MQTT???

[roperscrossroads]

For the past two weeks I've been running 2.6.4 and 2.6.9 on Heltec v3s and the Heltec e290, using the beta Android app (a couple of different versions, currently 2.6.7). I started using Meshtastic a month ago and it's been very broken the past 2 weeks. I don't see anything coming from mqtt servers and I have a hard time configuring nodes with the app.

[blue-tj01]

Seems like this issue still exists in 2.6.11. Do I seek to downgrade to 2.5.5 or 2.5.18? It a shame that this bug was logged 12 months ago but not much movement on fixing this issue :(

[amasterengineer]

i am here with version 2.6.11.60ec05e Beta and im having same issues with wifi just dropping out, i can use another node and connect to it and still use mqtt so it seems. just can not connect to it with android app or web client. once it has been rebooted then it will work correctly. tomorrow ill try and see about down grading to 2.5.18 and see how that goes

[dblaber]

Unfortunately another met too with this bug. I would love to help troubleshoot, for me the android app will constantly mention its downloading each of the node info, and then if i try to scroll through options on oled, its extremely lagged and unresponsive. Once I hit reset button, it seems to work for a bit... until it doesn't again.

[JaraLowell]

For a long time, i had zero issues 2.5.9.93, till the MQTT broker i was logged in to shut down, and made me have to go back to the main MQTT broker. The old broker had no TLS; While official meschtastic mqtt has. And what do you know, the issue started to happen. So this might simply be because MQTT crashes, and then brings the whole wifi down. Evil chap gpt stated, Run loop (runOnce()), no explicit MQTT PINGREQ handling found. stating it based this on the broker disconnects after 6h idle time; So if we get disconnected from the broker, and not catch this, we seem to loose whole connections interface as a whole. Just food for toughts.

[JaraLowell]

So over all our test shows,. it the mqtt server you connect to; the old 2.5.9.93 stays connected to some servers but disconects after x hours from the main one meshtastic hosts. while it stays connected to some of the pivate mqtt brokers/servers. But the latest works fine on the meshtastic mqtt server, but gets disconected from private brokers/servers. and what mqtt breaks it breaks apparently the whole network stack.

[nasiralamreeki]

Damn