YuckFou node is down hard
Posted about 1 year ago by YuckFou
NodeID = 0212fff1e1565713ba134324dcbc4eec59659c167569db6c2ef9797ee1a6e54c63
I was suddenly unable to ssh into it.
Rebooted. Twice. Still unable to ssh in.
Connected external display.
This is what I am seeing:
I have my SCB and seed words. Am going to use this opportunity to switch from Citadel to Raspiblitz.
Flashing SDHC card now.
Restoring from SCB should force close all of my existing channels.
I will reopen all channels listed on this spreadsheet ASAP: https://docs.google.com/spreadsheets/d/1XQZR3OX_b0AC1vlyR3WAsKOzzZO3_bi30ZztzO1DlQs
74 Comments
YuckFou wrote about 1 year ago
LN+ Admin wrote about 1 year ago
YuckFou wrote about 1 year ago
Suspecting pi4 now.
Have replacement pi4 in hand, different/new powersupply.
Flashing raspiblitz to different sdhc card now.
If the flash goes well, I should be booting it in the next 2 hours.
LN+ Admin wrote about 1 year ago
YuckFou wrote about 1 year ago
New everything; new pi4, new cables, new SDHC, and new raspiblitz software to eliminate all possibilities.
Planning on using SCB, so all existing channels will be force closed momentarily.
I will reopen all channels that I had before, and pay all rebalancing fees.
YuckFou wrote about 1 year ago
Copying of the blockchain data to the new node has begun. It will be several hours before it's done copying.
Screenshot: https://ibb.co/sChhPKn
YuckFou wrote about 1 year ago
LN+ Admin wrote about 1 year ago
YuckFou wrote about 1 year ago
I thought it would have a new pubkey, but it looks like it will still be the original pubkey 0212fff1e1565713ba134324dcbc4eec59659c167569db6c2ef9797ee1a6e54c63. Still working on getting this new node permanently set up; static LAN IP address, mounted in a permanent location on my rack in an active-cooled Pi enclosure (whereas the last one was just dangling behind my monitor with no case, no cooling).
Middle Way ₿⚡ wrote about 1 year ago
SpookyPaymentAtADistance wrote about 1 year ago
from the screenshots it seems that the cause might be the SD card. Have you checked it?
good luck with the transition!
Middle Way ₿⚡ wrote about 1 year ago
SpookyPaymentAtADistance wrote about 1 year ago
Middle Way ₿⚡ wrote about 1 year ago
YuckFou wrote about 1 year ago
The node started acting wacky days before it failed.
When it did, I quickly made an up-to-date backup of the SCB.
However, I left the node running and am positive that at least 5 very small LN transactions occurred since I did the last SCB backup. That pretty much invalidated the SCB. I did not want to take a chance of some peer running a watchtower see that I was 5000 sats short on my restore, then penalize me dearly for it.
Also the node that failed was a Citadel (https://github.com/runcitadel/citadel) and finding their node software to download is near impossible. They're going thru some major changes and rebranding, so finding the package to flash another SDHC with was taking more time than it was worth, so I gave up on that whole idea and switched over to Raspiblitz instead.
@Middle Way ₿⚡ : that idea of sending backups to a watchtower sounds like it should work... but I know little/nothing of watchtowers yet other than the theory of them. I have yet to run one myself. Also to answer your question about temp warnings; I never saw any warnings. Maybe there was one very early on when I first set it up. I moved it away from hanging over the USB strip (which gives off the tiniest amount of heat), the problem went away, so I left it as it was for it's whole lifetime (about 13-14 months before it died).
Middle Way ₿⚡ wrote about 1 year ago
YuckFou wrote about 1 year ago
YuckFou wrote about 1 year ago
Fatal LevelDB error: Corruption: 1 missing files; e.g.: /mnt/hdd/bitcoin/indexes/txindex/053716.ldb
bitcoin in AppInit() "
Screenshot: https://ibb.co/K60DFGZ
Middle Way ₿⚡ wrote about 1 year ago
YuckFou wrote about 1 year ago
YuckFou wrote about 1 year ago
YuckFou wrote about 1 year ago
I'll be searching for instructions on how to test the ssd, and unfuck the deletion of that bad file.
YuckFou wrote about 1 year ago
Middle Way ₿⚡ wrote about 1 year ago
YuckFou wrote about 1 year ago
I suspect it was the power supply that was the cause.
Replaced power supply with a new one.
Now the raspiblitz node is stuck in a loop since I made a mistake of deleting one of the files it was complaining about that failed a checksum.
Opened issue on raspiblitz: https://github.com/raspiblitz/raspiblitz/issues/4230
YuckFou wrote about 1 year ago
sudo -u bitcoin /usr/local/bin/bitcoind -reindex
LN+ Admin wrote about 1 year ago
YuckFou wrote about 1 year ago
2023-10-27T14:28:28Z Loaded 47 blocks from external file in 13166ms
2023-10-27T14:28:28Z Reindexing block file blk01397.dat...
2023-10-27T14:28:50Z Loaded 175 blocks from external file in 21253ms
2023-10-27T14:28:50Z Reindexing block file blk01398.dat...
2023-10-27T14:29:22Z Loaded 226 blocks from external file in 32099ms
2023-10-27T14:29:22Z Reindexing block file blk01399.dat...
2023-10-27T14:29:40Z Loaded 120 blocks from external file in 18813ms
2023-10-27T14:29:40Z Reindexing block file blk01400.dat...
2023-10-27T14:30:01Z Loaded 151 blocks from external file in 20839ms
2023-10-27T14:30:01Z Reindexing block file blk01401.dat...
2023-10-27T14:30:21Z Loaded 129 blocks from external file in 20138ms
2023-10-27T14:30:21Z Reindexing block file blk01402.dat...
2023-10-27T14:30:45Z Loaded 145 blocks from external file in 23109ms
2023-10-27T14:30:45Z Reindexing block file blk01403.dat...
2023-10-27T14:31:04Z Loaded 127 blocks from external file in 19657ms
2023-10-27T14:31:04Z Reindexing block file blk01404.dat...
I did some rough math earlier and I estimated it to be about another 16 hours yet.
YuckFou wrote about 1 year ago
SpookyPaymentAtADistance wrote about 1 year ago
YuckFou wrote about 1 year ago
2023-10-28T10:12:40Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-10-28T10:12:40Z UpdateTip: new best=0000000000000000016a2c8eed5394531362063578c9318bcffb29e773c815f1 height=353485 version=0x00000002 log2_work=82.673864 tx=66561852 date='2015-04-24T10:56:35Z' progress=0.075099 cache=871.3MiB(5535858txo)
2023-10-28T10:12:40Z UpdateTip: new best=00000000000000000d69bcd4c71b14216dd4f5990773f6c98e08049958b53bbb height=353486 version=0x00000002 log2_work=82.673902 tx=66562632 date='2015-04-24T10:52:35Z' progress=0.075099 cache=871.4MiB(5536530txo)
2023-10-28T10:12:40Z UpdateTip: new best=000000000000000013ceee6d367995c1ea567e0917381a199dec83eef422c800 height=353487 version=0x00000002 log2_work=82.673940 tx=66563325 date='2015-04-24T11:05:48Z' progress=0.075100 cache=871.4MiB(5536837txo)
2023-10-28T10:12:40Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-10-28T10:12:40Z UpdateTip: new best=00000000000000000847e53498147641112aa9b869e9b9cfd36ea172d995c6a5 height=353488 version=0x00000002 log2_work=82.673979 tx=66563615 date='2015-04-24T11:03:45Z' progress=0.075101 cache=871.4MiB(5536813txo)
2023-10-28T10:12:40Z UpdateTip: new best=00000000000000000a3cb11859a4312f2ee1f656f2e2bed66737976e6d412fea height=353489 version=0x00000002 log2_work=82.674017 tx=66563770 date='2015-04-24T11:05:01Z' progress=0.075101 cache=871.4MiB(5536881txo)
2023-10-28T10:12:40Z UpdateTip: new best=000000000000000005e9206a9e694fcef80c5d6c94c2cf9bee3357c23198b0d7 height=353490 version=0x00000003 log2_work=82.674055 tx=66565254 date='2015-04-24T11:28:27Z' progress=0.075102 cache=871.3MiB(5536243txo)
2023-10-28T10:12:41Z UpdateTip: new best=000000000000000004f87ec418d33ac4d698c265a9e35207339559b5bda33016 height=353491 version=0x00000002 log2_work=82.674093 tx=66565766 date='2015-04-24T11:38:59Z' progress=0.075103 cache=871.3MiB(5535784txo)
2023-10-28T10:12:41Z UpdateTip: new best=000000000000000006bc1125cf4438ee2ac6f362b60df90ad7841d53f7a54b47 height=353492 version=0x00000002 log2_work=82.674132 tx=66565767 date='2015-04-24T11:35:50Z' progress=0.075103 cache=871.3MiB(5535785txo)
2023-10-28T10:12:41Z UpdateTip: new best=0000000000000000092c5221fd7c86bfc9cf6860d32eabb721c826bef5511871 height=353493 version=0x00000003 log2_work=82.674170 tx=66566924 date='2015-04-24T11:49:16Z' progress=0.075104 cache=871.2MiB(5535125txo)
2023-10-28T10:12:41Z UpdateTip: new best=0000000000000000039a8f04a49d817a929df60f9e1aec7d8eab2de34d892a34 height=353494 version=0x00000002 log2_work=82.674208 tx=66568099 date='2015-04-24T12:01:43Z' progress=0.075106 cache=871.2MiB(5535324txo)
2023-10-28T10:12:41Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
Tortuga wrote about 1 year ago
YuckFou wrote about 1 year ago
2023-10-28T19:24:20Z UpdateTip: new best=00000000000000000231088bd23ed91f85cba753c232ef335d68676422dcfb9b height=422319 version=0x20000000 log2_work=85.040819 tx=144859098 date='2016-07-26T08:34:19Z' progress=0.163415 cache=2516.3MiB(19036909txo)
2023-10-28T19:24:21Z Cache size (2638899504) exceeds total space (2638586624)
YuckFou wrote about 1 year ago
2023-10-28T19:24:20Z UpdateTip: new best=00000000000000000231088bd23ed91f85cba753c232ef335d68676422dcfb9b height=422319 version=0x20000000 log2_work=85.040819 tx=144859098 date='2016-07-26T08:34:19Z' progress=0.163415 cache=2516.3MiB(19036909txo)
2023-10-28T19:24:21Z Cache size (2638899504) exceeds total space (2638586624)
2023-10-28T19:29:54Z UpdateTip: new best=000000000000000003f2ef3bdf7200f19366743ef9160977ce0bed43d46b1809 height=422320 version=0x20000000 log2_work=85.040852 tx=144861323 date='2016-07-26T08:43:42Z' progress=0.163418 cache=187.7MiB(0txo)
2023-10-28T19:29:54Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-10-28T19:29:54Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-10-28T19:30:28Z UpdateTip: new best=00000000000000000294a1c389d1e1ee4781174333d151d6d5170bec0504066a height=422321 version=0x20000000 log2_work=85.040886 tx=144863207 date='2016-07-26T08:48:09Z' progress=0.163420 cache=188.7MiB(7937txo)
2023-10-28T19:30:53Z UpdateTip: new best=000000000000000002abf533ee9c656c88558d576d4355cd8f4ad9ac3969ebe4 height=422322 version=0x20000000 log2_work=85.040919 tx=144865463 date='2016-07-26T09:11:02Z' progress=0.163422 cache=189.6MiB(15125txo)
2023-10-28T19:30:53Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-10-28T19:30:53Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-10-28T19:30:54Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-10-28T19:30:54Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
YuckFou wrote about 1 year ago
https://docs.google.com/spreadsheets/d/1XQZR3OX_b0AC1vlyR3WAsKOzzZO3_bi30ZztzO1DlQs
Middle Way ₿⚡ wrote about 1 year ago
YuckFou wrote about 1 year ago
To everyone else: if you think you had a channel with YuckFou that I should rebuild, please check the spreadsheet at https://docs.google.com/spreadsheets/d/1XQZR3OX_b0AC1vlyR3WAsKOzzZO3_bi30ZztzO1DlQs and let me know if you're not already listed on there.
YuckFou wrote about 1 year ago
CanaKit 3.5A Raspberry Pi 4 Power Supply (USB-C), ASIN B07TYQRXTK, ordered from Amazon.
I have heard other horror stories about using knock-off Raspberry Pi 4 power supplies, but I thought those were just unlucky people somehow. Now I'm a believer; use only genuine Raspberry Pi power supplies. The cost savings when buying a competing product from a different manufacturer is only a few dollars. This corrupted node failure, whether it was caused by the CanaKit power supply or not, has ended up costing me hundreds of dollars in open/closing channel fees, 30+ hours hours spent troubleshooting, and the end of this node's life. Lesson learned.
If you ever need to purchase another Raspberry Pi 4 power supply, do yourself a favor; spend the couple extra bucks to get a genuine Raspberry Pi product.
Middle Way ₿⚡ wrote about 1 year ago
ZMI PowerPack No. 20, QB826G
YuckFou wrote about 1 year ago
YuckFou wrote about 1 year ago
SpookyPaymentAtADistance wrote about 1 year ago
If you have another synched full node you can manually copy the blockchain on the PI and this could be much faster.
YuckFou wrote about 1 year ago
It's up to 62.7% reindexed now. I'm just gonna ride it out.
YuckFou wrote about 1 year ago
2023-11-05T20:25:05Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-11-05T20:25:18Z UpdateTip: new best=000000000000000000028e5b1b0872c4155146a8380cbfc8dbd4c4777439d97f height=725622 version=0x20400000 log2_work=93.378656 tx=714605625 date='2022-03-02T18:23:45Z' progress=0.803831 cache=332.0MiB(1173658txo)
2023-11-05T20:25:24Z UpdateTip: new best=000000000000000000015da61ec4c6c42f7bbd127a9c7571bb10f36685b4a9e6 height=725623 version=0x20e00000 log2_work=93.378669 tx=714607404 date='2022-03-02T18:26:07Z' progress=0.803833 cache=331.8MiB(1172182txo)
YuckFou wrote about 1 year ago
2023-11-07T06:58:16Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-11-07T06:58:17Z UpdateTip: new best=00000000000000000003b952680bf4f0c9d03b6bb8004482fab551aba054f5f5 height=765237 version=0x22110000 log2_work=93.869742 tx=784253577 date='2022-11-30T00:19:49Z' progress=0.881723 cache=2262.1MiB(16808244txo)
YuckFou wrote about 1 year ago
2023-11-08T17:51:01Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-11-08T17:51:06Z UpdateTip: new best=000000000000000000038fda3291f0e03783037278dab4688e82b09bb8df0c37 height=796764 version=0x30056000 log2_work=94.274871 tx=858906012 date='2023-07-02T00:12:56Z' progress=0.954267 cache=2310.0MiB(15522530txo)
Middle Way ₿⚡ wrote about 1 year ago
YuckFou wrote about 1 year ago
Yah, I saw that. Yikes. I guess it's time to figure out how to open multiple channels with one on-chain Tx. I've never done that before. If you have, and know what works, plz lmk.
The last 1% of the reindex is crawling;
2023-11-10T08:21:12Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2023-11-10T08:21:20Z UpdateTip: new best=00000000000000000000cf0ba5f11728ee0a83f149015d5f58532d42a5da1430 height=813661 version=0x2b0dc000 log2_work=94.496277 tx=908611948 date='2023-10-24T18:45:36Z' progress=0.994248 cache=1317.9MiB(9018932txo)
2023-11-10T08:21:30Z UpdateTip: new best=000000000000000000048d96bf4d1d504fd2537c586be0cb200c9e051c963813 height=813662 version=0x20a00000 log2_work=94.496290 tx=908614895 date='2023-10-24T18:50:25Z' progress=0.994249 cache=1318.4MiB(9022786txo)
Middle Way ₿⚡ wrote about 1 year ago
YuckFou wrote about 1 year ago
YuckFou wrote about 1 year ago
2023-11-10T13:01:46Z UpdateTip: new best=00000000000000000001eea4060fe616e15d0a0748b596a8d07ea68c1230531d height=816145 version=0x22000000 log2_work=94.530035 tx=916952040 date='2023-11-10T12:51:59Z' progress=0.999998 cache=400.6MiB(1588305txo)
2023-11-10T13:01:46Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
I'm not sure what it's doing now.
The Raspiblitz status screens still shows "Node is Syncing".
Not seeing any more new messages in /mnt/hdd/bitcoin/debug.log
I'm just going to leave it alone for a while as to not mess up anything it might be doing behind the scenes.
LN+ Admin wrote about 1 year ago
YuckFou wrote about 1 year ago
2023-11-10T20:33:24Z UpdateTip: new best=000000000000000000022b179caa169d5f90a7d63eb3733a65b0ba07fd30e4c2 height=816198 version=0x23f6e000 log2_work=94.530752 tx=917142427 date='2023-11-10T20:32:52Z' progress=1.000000 cache=459.0MiB(2028002txo)
Since it took weeks to get this far, I'm just going to let it stay this way for a while.
YuckFou wrote about 1 year ago
It rebooted.
Things are looking a lot better now. It's still synchronizing, but at least I am now seeing positive signs in the output of "sudo tail -f /mnt/hdd/lnd/logs/bitcoin/mainnet/lnd.log".
Still going...
2023-11-11 05:42:52.746 [DBG] CNCT: ChannelArbitrator(b10e207b01f426a0d92e3866f79aee58852a5e5c4e54025997bffc9750d11671:0): checking commit chain actions at height=816257, in_htlc_count=0, out_htlc_count=0
2023-11-11 05:42:52.746 [DBG] CNCT: ChannelArbitrator(b10e207b01f426a0d92e3866f79aee58852a5e5c4e54025997bffc9750d11671:0): no actions for chain trigger, terminating
2023-11-11 05:42:52.746 [DBG] CNCT: ChannelArbitrator(b10e207b01f426a0d92e3866f79aee58852a5e5c4e54025997bffc9750d11671:0): terminating at state=StateDefault
2023-11-11 05:42:52.746 [DBG] CNCT: ChannelArbitrator(4148ef628291245a3c087568a898efb22be9ec85bb1d5f59686c2b44b18a3ec3:1): no actions for chain trigger, terminating
2023-11-11 05:42:52.746 [DBG] CNCT: ChannelArbitrator(4148ef628291245a3c087568a898efb22be9ec85bb1d5f59686c2b44b18a3ec3:1): terminating at state=StateDefault
2023-11-11 05:42:52.747 [INF] DISC: Authenticated Gossiper starting
2023-11-11 05:42:52.747 [INF] NTFN: New block epoch subscription
2023-11-11 05:42:52.763 [INF] CRTR: Channel Router starting
2023-11-11 05:42:52.764 [DBG] DISC: Requesting online notification for peer=035e4ff418fc8b5554c5d9eea66396c227bd429a3251c8cbc711002ba215bfc226
YuckFou wrote about 1 year ago
Bad news: Prior to the node failure I had 65,299,064 sats on the node according to the last screenshot that I had of the node's dashboard before it failed. Now it shows that I only have 60,446,879 sats. So somehow I lost 4,852,185 sats, or 0.04852185 BTC, about $1,795.64 at the current BTC price of $37007.
I see that RTL is showing 5 channels in the Pending tab;
Pending Force Closing (1)
Waiting Close (4)
None of those Pending channels make sense;
The 1 in Pending Force Closing has an on-chain closing TxID, 77b92d64026251c9404cda48e4d34654bc7266310b7a8b8d3ea626bfea601abd, and that Tx id Confirmed on-chain with 2,398 confirmations.
RTL still shows it as Pending.
Not sure what to make of that, or what I should try to do to fix it.
The other 4 Waiting Close channels are wrong also.
So something is still screwy somewhere.
I'm too tired to figure it out now. Need sleep. Will have to resume hacking away at this tomorrow.
Middle Way ₿⚡ wrote about 1 year ago
The other 4 Waiting Close channels are wrong also.
1. Check your lnd.conf to have:
max-commit-fee-rate-anchors=100
lncli closechannel funding_txid --force
YuckFou wrote about 1 year ago
I will give that a shot momentarily.
I'm on Telegram as @YuckFouBTC.
Middle Way ₿⚡ wrote about 1 year ago
SpookyPaymentAtADistance wrote 12 months ago
YuckFou wrote 12 months ago
SpookyPaymentAtADistance wrote 12 months ago
YuckFou wrote 12 months ago
YuckFou wrote 12 months ago
RTL does not load.
This doesn't look good...
YuckFou wrote 12 months ago
YuckFou wrote 12 months ago
They don't work anymore.
Maybe something changed with the raspiblitz software so those are no longer the correct commands to use.
Giving up on it again soon.
Looks like the node has not been rebooted in 16 days;
admin@192.168.1.247:~ ₿ uptime
23:35:40 up 16 days, 12:54, 5 users, load average: 7.09, 6.49, 5.33
admin@192.168.1.247:~ ₿
But I'm not going to just go and reboot it yet.
Hmm... It should come up with a menu when ssh'ing in, not just get stuck at that "Node is Syncing" message...
YuckFou wrote 12 months ago
I'm too tired to do anything more with this tonight. I will try to pick this up again tomorrow.
Still seeing 4 ("zombie"?) channels showing in LNDg:
Tortuga wrote 12 months ago
YuckFou wrote 10 months ago
I ended up enlisting the help of another (semi-famous) bitcoiner; Semisol.
Semisol has helped me in the past when I had node problems.
This time he helped me again.
As promised, I split the recovered sats with him 50/50.
I can attest that Semisol is a genuinely helpful and honest person. He is trustworthy enough to deal with delicate node recovery situations where you may need to share ssh access into the node, or share your seed phrase. Normally I would advise to never share your seed phrase with anyone, under any circumstances, but here was different; my node was down for 2+ months and I tried everything I could think of and I was only able to recover 1 of the zombie channels myself.
I ended up setting up ssh access over a Tor .onion address, and sharing all passwords (A, B, and C) and seed phrase with Semisol after I had swept all on-chain funds off of the node myself.
Semisol was able to rescue a lot of the sats that I had already considered to be lost. I believe he used chantools to do his magic. He sent them to one of his own wallets and then sent me 50% of the sats to a lightning invoice that I produced for him.
If anyone ever needs node recovery services, I would recommend that you contact Semisol directly on Nostr at npub12262qa4uhw7u8gdwlgmntqtv7aye8vdcmvszkqwgs0zchel6mz7s6cgrkj
YuckFou wrote 10 months ago
Apologies to all of my liquidity swap partners that I had channels to before my node failed.
At first I thought I would build another node and reopen channels to all that were listed on https://docs.google.com/spreadsheets/d/1XQZR3OX_b0AC1vlyR3WAsKOzzZO3_bi30ZztzO1DlQs/edit?usp=sharing, but now things have changed. I need a break. I will not be rebuilding another node or reopening any channels anytime soon.
If/when I do set up another node, I will still attempt to reopen channels that I abandoned early here.
LN+ Admin wrote 10 months ago
SpookyPaymentAtADistance wrote 10 months ago
hope you'll get over it soon!
YuckFou wrote 7 months ago
0376bd2f87ea8376ae5b451f85a4210e4df5ce81b949054288def4c7867640923c
https://lightningnetwork.plus/nodes/0376bd2f87ea8376ae5b451f85a4210e4df5ce81b949054288def4c7867640923c
Middle Way ₿⚡🌊 helped me get some of the stuck sats back when I interacted with him on Telegram. He was very helpful and if you ever want to build a channel to one of the good guys, he earned my recommendation.
LN+ Admin wrote 7 months ago
Middle Way ₿⚡ wrote 6 months ago
Please login to post comments.
Latest news
Use FreeNAS as a way to back up your node
Posted about 2 years ago