YuckFou node is down hard

Posted 7 months ago by YuckFou

YuckFou node is down hard.
NodeID = 0212fff1e1565713ba134324dcbc4eec59659c167569db6c2ef9797ee1a6e54c63

I was suddenly unable to ssh into it.
Rebooted.  Twice.  Still unable to ssh in.
Connected external display.
This is what I am seeing:
Screenshot from 2023-10-19 11-51-25.png 1.29 MB

I have my SCB and seed words.  Am going to use this opportunity to switch from Citadel to Raspiblitz.
Flashing SDHC card now.
Restoring from SCB should force close all of my existing channels.
I will reopen all channels listed on this spreadsheet ASAP: https://docs.google.com/spreadsheets/d/1XQZR3OX_b0AC1vlyR3WAsKOzzZO3_bi30ZztzO1DlQs

74 Comments

YuckFou

YuckFou wrote 7 months ago

Having problems getting a node stood up.  Possible cable failure.  Ordered replacement cable.  Expected to arrive in the next 24 hours.


LN+

LN+ Admin wrote 7 months ago

Good luck! 🍀


YuckFou

YuckFou wrote 7 months ago

It wasn't a cable problem.
Suspecting pi4 now.
Have replacement pi4 in hand, different/new powersupply.
Flashing raspiblitz to different sdhc card now.
If the flash goes well, I should be booting it in the next 2 hours.


LN+

LN+ Admin wrote 7 months ago

How did it go?


YuckFou

YuckFou wrote 7 months ago

Booting up freshly flashed node right now for the first time.
New everything; new pi4, new cables, new SDHC, and new raspiblitz software to eliminate all possibilities.
Planning on using SCB, so all existing channels will be force closed momentarily.

I will reopen all channels that I had before, and pay all rebalancing fees.


YuckFou

YuckFou wrote 7 months ago

Progess has been made!

Copying of the blockchain data to the new node has begun.  It will be several hours before it's done copying.

Screenshot: https://ibb.co/sChhPKn


YuckFou

YuckFou wrote 7 months ago

Probably another 8 hours or so before LND gets done with the restore...


LN+

LN+ Admin wrote 6 months ago

So, you had to close all the channels or it's the same pubkey?


YuckFou

YuckFou wrote 6 months ago

All channels were automatically force closed.  The force close transactions are all currently in the mempool with about 10 sats/vB fee.  It will be a while before they're even confirmed, then the sats will sit in Limbo status for another 48 hours or so before I will be able to start re-opening channels that I had before.

I thought it would have a new pubkey, but it looks like it will still be the original pubkey 0212fff1e1565713ba134324dcbc4eec59659c167569db6c2ef9797ee1a6e54c63.   Still working on getting this new node permanently set up; static LAN IP address, mounted in a permanent location on my rack in an active-cooled Pi enclosure (whereas the last one was just dangling behind my monitor with no case, no cooling).


Middle Way ₿⚡

Middle Way ₿⚡ wrote 6 months ago

Sad story. So you think it was a hardware fault of Pi4 due to overheating? Did Citadel give you temp reading of the CPU?


SpookyPaymentAtADistance

SpookyPaymentAtADistance wrote 6 months ago

hey, sorry to hear that :/

from the screenshots it seems that the cause might be the SD card. Have you checked it?

 good luck with the transition!


Middle Way ₿⚡

Middle Way ₿⚡ wrote 6 months ago

If the SD card gets corrupted, would a solution be to flash an identical OS on an identical card and try to boot from it? Or once installed, the OS writes some critical data on the card, and the new one will only offer a reinstall?


SpookyPaymentAtADistance

SpookyPaymentAtADistance wrote 6 months ago

Middle, a possible solution could be make a fresh install and try to copy the lnd files. This is a bit risky because you could possibly restore an old channel state and lose all the channel funds.


Middle Way ₿⚡

Middle Way ₿⚡ wrote 6 months ago

Can backups sent to a watchtower help, theoretically, in some future release of LND?


YuckFou

YuckFou wrote 6 months ago

Based on the troubleshooting that I did, I narrowed it down to either the SD card or the Pi that failed.  Have not yet had time to tinker with the carcass to figure out which of the two it was.

The node started acting wacky days before it failed.
When it did, I quickly made an up-to-date backup of the SCB.
However, I left the node running and am positive that at least 5 very small LN transactions occurred since I did the last SCB backup.  That pretty much invalidated the SCB.  I did not want to take a chance of some peer running a watchtower see that I was 5000 sats short on my restore, then penalize me dearly for it.

Also the node that failed was a Citadel (https://github.com/runcitadel/citadel) and finding their node software to download is near impossible.  They're going thru some major changes and rebranding, so finding the package to flash another SDHC with was taking more time than it was worth, so I gave up on that whole idea and switched over to Raspiblitz instead.

@Middle Way ₿⚡ : that idea of sending backups to a watchtower sounds like it should work... but I know little/nothing of watchtowers yet other than the theory of them.  I have yet to run one myself.   Also to answer your question about temp warnings; I never saw any warnings.  Maybe there was one very early on when I first set it up.  I moved it away from hanging over the USB strip (which gives off the tiniest amount of heat), the problem went away, so I left it as it was for it's whole lifetime (about 13-14 months before it died).


Middle Way ₿⚡

Middle Way ₿⚡ wrote 6 months ago

Watchtower client sends every state update to a remote location. I guess with proper authentication it could be doable to restore a node from a crash. But I am dreaming...


YuckFou

YuckFou wrote 6 months ago

This new node failed to come up long enough that I could open any channels on it. Already says it's corrupted.


YuckFou

YuckFou wrote 6 months ago

I'm frustrated at this point.  I am now debating on what to do here.  I have a relatively reliable VirtualBox installation that I could set up a new node with, then try the SCB again.  Already used it once.  Can I use it again, I wonder?   Ugh.  Can get into CLI on the new node, but it's looping where it keeps failing with the "EXCEPTION: 15dbwrapper_error       
Fatal LevelDB error: Corruption: 1 missing files; e.g.: /mnt/hdd/bitcoin/indexes/txindex/053716.ldb       
bitcoin in AppInit() "
Screenshot: https://ibb.co/K60DFGZ


Middle Way ₿⚡

Middle Way ₿⚡ wrote 6 months ago

Maybe your SSD is dying? Did you check it?


YuckFou

YuckFou wrote 6 months ago

Got another Pi power supply.  Trying that now...


YuckFou

YuckFou wrote 6 months ago

The deleted/missing file makes sense; I rm'ed it because it said that file failed the checksum.  I thought that by simply rm'ing it, it would see it as missing then go ask the network for it.  Apparently not.


YuckFou

YuckFou wrote 6 months ago

Ok, got a new power supply hooked up, and also swapped the cable going to to the external SSD with a brand new one as well.

I'll be searching for instructions on how to test the ssd, and unfuck the deletion of that bad file.


YuckFou

YuckFou wrote 6 months ago

While I search and learn, it's still looping;
image.png 28.5 KB


Middle Way ₿⚡

Middle Way ₿⚡ wrote 6 months ago

FWIW, I had a pretty smooth experience so far (1.5 years) with Umbrel running on Pi4 8gb inside this case: https://amzn.eu/d/7q4X4lv. As a UPS I bought this: https://a.co/d/dqazkZB


YuckFou

YuckFou wrote 6 months ago

New node failed shortly after starting the restore via SCB.
I suspect it was the power supply that was the cause.
Replaced power supply with a new one.
Now the raspiblitz node is stuck in a loop since I made a mistake of deleting one of the files it was complaining about that failed a checksum.
Opened issue on raspiblitz: https://github.com/raspiblitz/raspiblitz/issues/4230


YuckFou

YuckFou wrote 6 months ago

Started to reindex by doing:

sudo -u bitcoin /usr/local/bin/bitcoind -reindex


LN+

LN+ Admin wrote 6 months ago

Godspeed!


YuckFou

YuckFou wrote 6 months ago

Progress update: still going... 

2023-10-27T14:28:28Z Loaded 47 blocks from external file in 13166ms
 2023-10-27T14:28:28Z Reindexing block file blk01397.dat...
 2023-10-27T14:28:50Z Loaded 175 blocks from external file in 21253ms
 2023-10-27T14:28:50Z Reindexing block file blk01398.dat...
 2023-10-27T14:29:22Z Loaded 226 blocks from external file in 32099ms
 2023-10-27T14:29:22Z Reindexing block file blk01399.dat...
 2023-10-27T14:29:40Z Loaded 120 blocks from external file in 18813ms
 2023-10-27T14:29:40Z Reindexing block file blk01400.dat...
 2023-10-27T14:30:01Z Loaded 151 blocks from external file in 20839ms
 2023-10-27T14:30:01Z Reindexing block file blk01401.dat...
 2023-10-27T14:30:21Z Loaded 129 blocks from external file in 20138ms
 2023-10-27T14:30:21Z Reindexing block file blk01402.dat...
 2023-10-27T14:30:45Z Loaded 145 blocks from external file in 23109ms
 2023-10-27T14:30:45Z Reindexing block file blk01403.dat...
 2023-10-27T14:31:04Z Loaded 127 blocks from external file in 19657ms
 2023-10-27T14:31:04Z Reindexing block file blk01404.dat...

I did some rough math earlier and I estimated it to be about another 16 hours yet.


YuckFou

YuckFou wrote 6 months ago

Progress report: 83.63% of reindex completed...


SpookyPaymentAtADistance

SpookyPaymentAtADistance wrote 6 months ago

almost there! 


YuckFou

YuckFou wrote 6 months ago

Looks like the reindex has entered a new phase... seeing the debug.log scrolling new messages now.  That's good news.  But the bad news is that it looks like it will be many hours more to wait yet...

2023-10-28T10:12:40Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-10-28T10:12:40Z UpdateTip: new best=0000000000000000016a2c8eed5394531362063578c9318bcffb29e773c815f1 height=353485 version=0x00000002 log2_work=82.673864 tx=66561852 date='2015-04-24T10:56:35Z' progress=0.075099 cache=871.3MiB(5535858txo)
2023-10-28T10:12:40Z UpdateTip: new best=00000000000000000d69bcd4c71b14216dd4f5990773f6c98e08049958b53bbb height=353486 version=0x00000002 log2_work=82.673902 tx=66562632 date='2015-04-24T10:52:35Z' progress=0.075099 cache=871.4MiB(5536530txo)
2023-10-28T10:12:40Z UpdateTip: new best=000000000000000013ceee6d367995c1ea567e0917381a199dec83eef422c800 height=353487 version=0x00000002 log2_work=82.673940 tx=66563325 date='2015-04-24T11:05:48Z' progress=0.075100 cache=871.4MiB(5536837txo)
2023-10-28T10:12:40Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-10-28T10:12:40Z UpdateTip: new best=00000000000000000847e53498147641112aa9b869e9b9cfd36ea172d995c6a5 height=353488 version=0x00000002 log2_work=82.673979 tx=66563615 date='2015-04-24T11:03:45Z' progress=0.075101 cache=871.4MiB(5536813txo)
2023-10-28T10:12:40Z UpdateTip: new best=00000000000000000a3cb11859a4312f2ee1f656f2e2bed66737976e6d412fea height=353489 version=0x00000002 log2_work=82.674017 tx=66563770 date='2015-04-24T11:05:01Z' progress=0.075101 cache=871.4MiB(5536881txo)
2023-10-28T10:12:40Z UpdateTip: new best=000000000000000005e9206a9e694fcef80c5d6c94c2cf9bee3357c23198b0d7 height=353490 version=0x00000003 log2_work=82.674055 tx=66565254 date='2015-04-24T11:28:27Z' progress=0.075102 cache=871.3MiB(5536243txo)
2023-10-28T10:12:41Z UpdateTip: new best=000000000000000004f87ec418d33ac4d698c265a9e35207339559b5bda33016 height=353491 version=0x00000002 log2_work=82.674093 tx=66565766 date='2015-04-24T11:38:59Z' progress=0.075103 cache=871.3MiB(5535784txo)
2023-10-28T10:12:41Z UpdateTip: new best=000000000000000006bc1125cf4438ee2ac6f362b60df90ad7841d53f7a54b47 height=353492 version=0x00000002 log2_work=82.674132 tx=66565767 date='2015-04-24T11:35:50Z' progress=0.075103 cache=871.3MiB(5535785txo)
2023-10-28T10:12:41Z UpdateTip: new best=0000000000000000092c5221fd7c86bfc9cf6860d32eabb721c826bef5511871 height=353493 version=0x00000003 log2_work=82.674170 tx=66566924 date='2015-04-24T11:49:16Z' progress=0.075104 cache=871.2MiB(5535125txo)
2023-10-28T10:12:41Z UpdateTip: new best=0000000000000000039a8f04a49d817a929df60f9e1aec7d8eab2de34d892a34 height=353494 version=0x00000002 log2_work=82.674208 tx=66568099 date='2015-04-24T12:01:43Z' progress=0.075106 cache=871.2MiB(5535324txo)
2023-10-28T10:12:41Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications


Tortuga wrote 6 months ago

it works until it doesn't. then its a whole lot of trying and time. good luck!


YuckFou

YuckFou wrote 6 months ago

Omg.  It froze up again.  Last 2 entries in debug.log;

2023-10-28T19:24:20Z UpdateTip: new best=00000000000000000231088bd23ed91f85cba753c232ef335d68676422dcfb9b height=422319 version=0x20000000 log2_work=85.040819 tx=144859098 date='2016-07-26T08:34:19Z' progress=0.163415 cache=2516.3MiB(19036909txo)
2023-10-28T19:24:21Z Cache size (2638899504) exceeds total space (2638586624)


YuckFou

YuckFou wrote 6 months ago

no wait... it started listing more again....

2023-10-28T19:24:20Z UpdateTip: new best=00000000000000000231088bd23ed91f85cba753c232ef335d68676422dcfb9b height=422319 version=0x20000000 log2_work=85.040819 tx=144859098 date='2016-07-26T08:34:19Z' progress=0.163415 cache=2516.3MiB(19036909txo)
2023-10-28T19:24:21Z Cache size (2638899504) exceeds total space (2638586624)
2023-10-28T19:29:54Z UpdateTip: new best=000000000000000003f2ef3bdf7200f19366743ef9160977ce0bed43d46b1809 height=422320 version=0x20000000 log2_work=85.040852 tx=144861323 date='2016-07-26T08:43:42Z' progress=0.163418 cache=187.7MiB(0txo)
2023-10-28T19:29:54Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-10-28T19:29:54Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-10-28T19:30:28Z UpdateTip: new best=00000000000000000294a1c389d1e1ee4781174333d151d6d5170bec0504066a height=422321 version=0x20000000 log2_work=85.040886 tx=144863207 date='2016-07-26T08:48:09Z' progress=0.163420 cache=188.7MiB(7937txo)
2023-10-28T19:30:53Z UpdateTip: new best=000000000000000002abf533ee9c656c88558d576d4355cd8f4ad9ac3969ebe4 height=422322 version=0x20000000 log2_work=85.040919 tx=144865463 date='2016-07-26T09:11:02Z' progress=0.163422 cache=189.6MiB(15125txo)
2023-10-28T19:30:53Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-10-28T19:30:53Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-10-28T19:30:54Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-10-28T19:30:54Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications


YuckFou

YuckFou wrote 6 months ago

I put together a spreadsheet that lists the channels that I will reopen once my node is synced up.

https://docs.google.com/spreadsheets/d/1XQZR3OX_b0AC1vlyR3WAsKOzzZO3_bi30ZztzO1DlQs


Middle Way ₿⚡

Middle Way ₿⚡ wrote 6 months ago

We had a 5m channel, but I don't insist


YuckFou

YuckFou wrote 6 months ago

@Middle Way ₿⚡ : indeed we did.  I slacked off on compiling that spreadsheet.  Thanks for catching me on that.  I added you and several more.

To everyone else: if you think you had a channel with YuckFou that I should rebuild, please check the spreadsheet at https://docs.google.com/spreadsheets/d/1XQZR3OX_b0AC1vlyR3WAsKOzzZO3_bi30ZztzO1DlQs and let me know if you're not already listed on there.


YuckFou

YuckFou wrote 6 months ago

Worth noting that when the YuckFou node failed, it's power supply was a CanaKit, 
CanaKit 3.5A Raspberry Pi 4 Power Supply (USB-C), ASIN B07TYQRXTK, ordered from Amazon.

I have heard other horror stories about using knock-off Raspberry Pi 4 power supplies, but I thought those were just unlucky people somehow.  Now I'm a believer; use only genuine Raspberry Pi power supplies.  The cost savings when buying a competing product from a different manufacturer is only a few dollars.  This corrupted node failure, whether it was caused by the CanaKit power supply or not, has ended up costing me hundreds of dollars in open/closing channel fees, 30+ hours hours spent troubleshooting, and the end of this node's life.  Lesson learned.

If you ever need to purchase another Raspberry Pi 4 power supply, do yourself a favor; spend the couple extra bucks to get a genuine Raspberry Pi product.


Middle Way ₿⚡

Middle Way ₿⚡ wrote 6 months ago

Also avoid powering off without proper shut down, due to grid power loss. I have this occasionally where I live, so I bought this power bank, that can output USB-C while charging itself: 

ZMI PowerPack No. 20, QB826G


YuckFou

YuckFou wrote 6 months ago

Progress report: 56.1% done, still going...


YuckFou

YuckFou wrote 6 months ago

59.1% now.  Holy.  It will be days yet at this rate...


SpookyPaymentAtADistance

SpookyPaymentAtADistance wrote 6 months ago

yup, IBD can take a long time on a raspberry. 

If you have another synched full node you can manually copy the blockchain on the PI and this could be much faster.


YuckFou

YuckFou wrote 6 months ago

I do have one.  That's how I got the data copied onto this new one.  But then one of the files failed the checksum when it tried to start up LND, so I deleted that offending file.  Wrong move-- that caused it to get stuck in a loop complaining about that missing LevelDB file.  That's when I decided to try a "bitcoind -reindex" hoping that would fix it.

It's up to 62.7% reindexed now.  I'm just gonna ride it out.


YuckFou

YuckFou wrote 6 months ago

Still going...

2023-11-05T20:25:05Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-11-05T20:25:18Z UpdateTip: new best=000000000000000000028e5b1b0872c4155146a8380cbfc8dbd4c4777439d97f height=725622 version=0x20400000 log2_work=93.378656 tx=714605625 date='2022-03-02T18:23:45Z' progress=0.803831 cache=332.0MiB(1173658txo)
2023-11-05T20:25:24Z UpdateTip: new best=000000000000000000015da61ec4c6c42f7bbd127a9c7571bb10f36685b4a9e6 height=725623 version=0x20e00000 log2_work=93.378669 tx=714607404 date='2022-03-02T18:26:07Z' progress=0.803833 cache=331.8MiB(1172182txo)


YuckFou

YuckFou wrote 6 months ago

Still going...

2023-11-07T06:58:16Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-11-07T06:58:17Z UpdateTip: new best=00000000000000000003b952680bf4f0c9d03b6bb8004482fab551aba054f5f5 height=765237 version=0x22110000 log2_work=93.869742 tx=784253577 date='2022-11-30T00:19:49Z' progress=0.881723 cache=2262.1MiB(16808244txo)




YuckFou

YuckFou wrote 6 months ago

Still going...

2023-11-08T17:51:01Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-11-08T17:51:06Z UpdateTip: new best=000000000000000000038fda3291f0e03783037278dab4688e82b09bb8df0c37 height=796764 version=0x30056000 log2_work=94.274871 tx=858906012 date='2023-07-02T00:12:56Z' progress=0.954267 cache=2310.0MiB(15522530txo)


Middle Way ₿⚡

Middle Way ₿⚡ wrote 6 months ago

No rush. Mempool is ugly (


YuckFou

YuckFou wrote 6 months ago

>No rush. Mempool is ugly (
Yah, I saw that.  Yikes.  I guess it's time to figure out how to open multiple channels with one on-chain Tx.  I've never done that before.  If you have, and know what works, plz lmk.

The last 1% of the reindex is crawling;
2023-11-10T08:21:12Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-11-10T08:21:20Z UpdateTip: new best=00000000000000000000cf0ba5f11728ee0a83f149015d5f58532d42a5da1430 height=813661 version=0x2b0dc000 log2_work=94.496277 tx=908611948 date='2023-10-24T18:45:36Z' progress=0.994248 cache=1317.9MiB(9018932txo)
2023-11-10T08:21:30Z UpdateTip: new best=000000000000000000048d96bf4d1d504fd2537c586be0cb200c9e051c963813 height=813662 version=0x20a00000 log2_work=94.496290 tx=908614895 date='2023-10-24T18:50:25Z' progress=0.994249 cache=1318.4MiB(9022786txo)



Middle Way ₿⚡

Middle Way ₿⚡ wrote 6 months ago

In LNDg there is a Batching page. It takes nodes public keys (the part before @ in full address). This software is very useful to run a node.


YuckFou

YuckFou wrote 6 months ago

Cool, thanks for the LNDg tip.  I'll look into it.


YuckFou

YuckFou wrote 6 months ago

Finally caught up with the tip height;

2023-11-10T13:01:46Z UpdateTip: new best=00000000000000000001eea4060fe616e15d0a0748b596a8d07ea68c1230531d height=816145 version=0x22000000 log2_work=94.530035 tx=916952040 date='2023-11-10T12:51:59Z' progress=0.999998 cache=400.6MiB(1588305txo)
2023-11-10T13:01:46Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications

I'm not sure what it's doing now.  
The Raspiblitz status screens still shows "Node is Syncing".
Not seeing any more new messages in /mnt/hdd/bitcoin/debug.log
I'm just going to leave it alone for a while as to not mess up anything it might be doing behind the scenes.


LN+

LN+ Admin wrote 6 months ago

Yeah, just wait. There is a lot it needs to do...


YuckFou

YuckFou wrote 6 months ago

bitcoin/debug.log says progress=1.000000, but the RaspiBlitz screen still shows "Lightning Progress : prepare sync".

2023-11-10T20:33:24Z UpdateTip: new best=000000000000000000022b179caa169d5f90a7d63eb3733a65b0ba07fd30e4c2 height=816198 version=0x23f6e000 log2_work=94.530752 tx=917142427 date='2023-11-10T20:32:52Z' progress=1.000000 cache=459.0MiB(2028002txo)
Screenshot from 2023-11-10 20-43-39.png 33.9 KB

Since it took weeks to get this far, I'm just going to let it stay this way for a while.


YuckFou

YuckFou wrote 6 months ago

It wasn't doing anything for nearly 24 hours so I did a "sudo reboot".

It rebooted.

Things are looking a lot better now.  It's still synchronizing, but at least I am now seeing positive signs in the output of "sudo tail -f /mnt/hdd/lnd/logs/bitcoin/mainnet/lnd.log".

Still going...

2023-11-11 05:42:52.746 [DBG] CNCT: ChannelArbitrator(b10e207b01f426a0d92e3866f79aee58852a5e5c4e54025997bffc9750d11671:0): checking commit chain actions at height=816257, in_htlc_count=0, out_htlc_count=0
2023-11-11 05:42:52.746 [DBG] CNCT: ChannelArbitrator(b10e207b01f426a0d92e3866f79aee58852a5e5c4e54025997bffc9750d11671:0): no actions for chain trigger, terminating
2023-11-11 05:42:52.746 [DBG] CNCT: ChannelArbitrator(b10e207b01f426a0d92e3866f79aee58852a5e5c4e54025997bffc9750d11671:0): terminating at state=StateDefault
2023-11-11 05:42:52.746 [DBG] CNCT: ChannelArbitrator(4148ef628291245a3c087568a898efb22be9ec85bb1d5f59686c2b44b18a3ec3:1): no actions for chain trigger, terminating
2023-11-11 05:42:52.746 [DBG] CNCT: ChannelArbitrator(4148ef628291245a3c087568a898efb22be9ec85bb1d5f59686c2b44b18a3ec3:1): terminating at state=StateDefault
2023-11-11 05:42:52.747 [INF] DISC: Authenticated Gossiper starting
2023-11-11 05:42:52.747 [INF] NTFN: New block epoch subscription
2023-11-11 05:42:52.763 [INF] CRTR: Channel Router starting
2023-11-11 05:42:52.764 [DBG] DISC: Requesting online notification for peer=035e4ff418fc8b5554c5d9eea66396c227bd429a3251c8cbc711002ba215bfc226


YuckFou

YuckFou wrote 6 months ago

Good news: I now have a lightning node online once again.

Bad news: Prior to the node failure I had 65,299,064 sats on the node according to the last screenshot that I had of the node's dashboard before it failed.  Now it shows that I only have 60,446,879 sats.  So somehow I lost 4,852,185 sats, or 0.04852185 BTC, about $1,795.64 at the current BTC price of $37007.

I see that RTL is showing 5 channels in the Pending tab;
Pending Force Closing  (1)
Waiting Close (4)

None of those Pending channels make sense; 
The 1 in Pending Force Closing has an on-chain closing TxID, 77b92d64026251c9404cda48e4d34654bc7266310b7a8b8d3ea626bfea601abd, and that Tx id Confirmed on-chain with 2,398 confirmations.
RTL still shows it as Pending.
Not sure what to make of that, or what I should try to do to fix it.

The other 4 Waiting Close channels are wrong also.

So something is still screwy somewhere.

I'm too tired to figure it out now.  Need sleep.  Will have to resume hacking away at this tomorrow.


Middle Way ₿⚡

Middle Way ₿⚡ wrote 6 months ago

The other 4 Waiting Close channels are wrong also.
In what way? Do you see the closing txs in mempool.space? If not, maybe the fees was too low and your node did not bump them, so they were evicted from mempool. 
1. Check your lnd.conf to have:
max-commit-fee-rate-anchors=100
2. Try fc again with 
lncli closechannel funding_txid --force
Contact me via Telegram if you need more help.


YuckFou

YuckFou wrote 6 months ago

Wow, thanks @Middle Way ₿⚡ - that makes perfect sense as to how/why I could be seeing things this way.

I will give that a shot momentarily.

I'm on Telegram as @YuckFouBTC.


Middle Way ₿⚡

Middle Way ₿⚡ wrote 6 months ago

I'm @MiddleWayNode and I sent you Hello this morning


SpookyPaymentAtADistance

SpookyPaymentAtADistance wrote 5 months ago

hey mate, how is it going?


YuckFou

YuckFou wrote 5 months ago

have not touched this since the mempool has been so full.  too costly to open channels when it's full.


SpookyPaymentAtADistance

SpookyPaymentAtADistance wrote 5 months ago

yes! I understand this! but at the end did it finish the IBD?


YuckFou

YuckFou wrote 5 months ago

Screenshot from 2023-11-27 19-49-04.png 21.6 KB


YuckFou

YuckFou wrote 5 months ago

It shows that when ssh'ing into it.

RTL does not load.

This doesn't look good...


YuckFou

YuckFou wrote 5 months ago

Screenshot from 2023-11-27 20-02-11.png 94.8 KB


YuckFou

YuckFou wrote 5 months ago

Those tail commands worked previously to see what bitcoind and lnd were doing.
They don't work anymore.
Maybe something changed with the raspiblitz software so those are no longer the correct commands to use.
Giving up on it again soon.

Looks like the node has not been rebooted in 16 days;

admin@192.168.1.247:~ ₿ uptime
 23:35:40 up 16 days, 12:54,  5 users,  load average: 7.09, 6.49, 5.33
admin@192.168.1.247:~ ₿ 

But I'm not going to just go and reboot it yet.

Hmm...  It should come up with a menu when ssh'ing in, not just get stuck at that "Node is Syncing" message...


YuckFou

YuckFou wrote 5 months ago

The status screen has changed.  It now shows the raspiblitz menu, as it should.

I'm too tired to do anything more with this tonight.  I will try to pick this up again tomorrow.

Still seeing 4 ("zombie"?) channels showing in LNDg:
image.png 245 KB


Tortuga wrote 5 months ago

congrats on sticking with things. slow and steady wins the sats back. 


YuckFou

YuckFou wrote 3 months ago

This is finally over.

I ended up enlisting the help of another (semi-famous) bitcoiner; Semisol.

Semisol has helped me in the past when I had node problems.

This time he helped me again.

As promised, I split the recovered sats with him 50/50.

I can attest that Semisol is a genuinely helpful and honest person.  He is trustworthy enough to deal with delicate node recovery situations where you may need to share ssh access into the node, or share your seed phrase.  Normally I would advise to never share your seed phrase with anyone, under any circumstances, but here was different; my node was down for 2+ months and I tried everything I could think of and I was only able to recover 1 of the zombie channels myself.

I ended up setting up ssh access over a Tor .onion address, and sharing all passwords (A, B, and C) and seed phrase with Semisol after I had swept all on-chain funds off of the node myself.

Semisol was able to rescue a lot of the sats that I had already considered to be lost.  I believe he used chantools to do his magic.  He sent them to one of his own wallets and then sent me 50% of the sats to a lightning invoice that I produced for him.

If anyone ever needs node recovery services, I would recommend that you contact Semisol directly on Nostr at npub12262qa4uhw7u8gdwlgmntqtv7aye8vdcmvszkqwgs0zchel6mz7s6cgrkj


YuckFou

YuckFou wrote 3 months ago

I am taking a breather away from LN for a while.
Apologies to all of my liquidity swap partners that I had channels to before my node failed.
At first I thought I would build another node and reopen channels to all that were listed on https://docs.google.com/spreadsheets/d/1XQZR3OX_b0AC1vlyR3WAsKOzzZO3_bi30ZztzO1DlQs/edit?usp=sharing, but now things have changed.  I need a break.  I will not be rebuilding another node or reopening any channels anytime soon.
If/when I do set up another node, I will still attempt to reopen channels that I abandoned early here.


LN+

LN+ Admin wrote 3 months ago

Take it easy!


SpookyPaymentAtADistance

SpookyPaymentAtADistance wrote 3 months ago

no need to rush YuckFou. Losing funds is an hard experience that takes time to recover from.
hope you'll get over it soon!


YuckFou

YuckFou wrote 6 days ago

Going thru some old notes and I happened to see something related to this that I thought was worth mentioning here.

0376bd2f87ea8376ae5b451f85a4210e4df5ce81b949054288def4c7867640923c
https://lightningnetwork.plus/nodes/0376bd2f87ea8376ae5b451f85a4210e4df5ce81b949054288def4c7867640923c

Middle Way ₿⚡🌊 helped me get some of the stuck sats back when I interacted with him on Telegram.  He was very helpful and if you ever want to build a channel to one of the good guys, he earned my recommendation.



LN+

LN+ Admin wrote 4 days ago

Thank you for the comment!


Middle Way ₿⚡

Middle Way ₿⚡ wrote 3 days ago

❤️

Please login to post comments.

Lightning Network Node
woody
Rank: 0
Capacity: 33,528,891 SAT
Channels: 2

Latest news