sniffit's Posts
Posted 5 months ago
Category Estimate (SAT) USD (@$0.00109/SAT) MYR (@4.2198)
| Lost Channel Funds | ~600,000 | ~$654.00 | ~RM2,759.72
| Wallet Balance Kept | ~157,000 | ~$171.13 | ~RM722.50
| Total On-Node Value Pre-Failure | ~757,000 | ~$825.13 | ~RM3,482.22
| Lessons Learned | - | Priceless | -
⚡ Rebuilding from the Ashes: A Node Operator's Journey ⚡
Posted 5 months ago
In the world of Lightning, things rarely break quietly. Over the past week, our node faced one of its most challenging trials—an unexpected decoupling of its infrastructure stack that spiraled into lost connections, misconfigured Tor services, and disrupted channel visibility.
It started with the physical hardware - randomly flinging errors and blinking amber lights. Mind you, these are Dell PowerEdge R630s - built for production and data center use. But as in Murphy's Law, the worse case that can happen - will happen.
Bitcoind had to be reindex - LND effectively yeeted to the void and everything was silent.
It started with the physical hardware - randomly flinging errors and blinking amber lights. Mind you, these are Dell PowerEdge R630s - built for production and data center use. But as in Murphy's Law, the worse case that can happen - will happen.
Bitcoind had to be reindex - LND effectively yeeted to the void and everything was silent.
But Lightning isn't about giving up when things go dark. It's about finding a way to restore that spark.
Through long nights, countless reconnect attempts, and relentless troubleshooting, I had to:
- Rebuild the Bitcoin and Lightning stack for clearer separation of responsibilities.
- Untangled conflicting Tor hidden services to restore proper node identity on the network.
- Reconnected to thousands of peers to reestablish our presence in the global Lightning graph.
- Watched as gossip propagation slowly restored our channels to the public network view.
The process wasn’t clean, easy, or painless. But it was necessary.
This isn’t a victory lap—it’s a reminder to every node runner: your node will face failures. What matters is how you respond.
We're still standing. Channels may have been lost. Lessons were gained. And we keep building forward.
- sniffit
0263a27989d64b6eca6958cfb60cc05fc641db6a258e91b9274d7042dd19bb8c88
- sniffit
0263a27989d64b6eca6958cfb60cc05fc641db6a258e91b9274d7042dd19bb8c88
Node Maintenance Extended Update #2
Posted 5 months ago
Despite best efforts - there's still some time before the node can be brought online but am currenly work on other tooling scripts and strategies to avoid this in the future. am guesstimating still 24+ hours left before everything can be ready and be brought back online piece by piece.
thank you for your patience.
thank you for your patience.
Node Maintenance Extended Update
Posted 5 months ago
Hey folks, just a quick heads-up — my node is still catching up on IBD. I’ve already crossed 92% sync but the remaining blocks are heavy, so I expect about 3–4 more days before it's online and stable again. Appreciate the patience — I’ll notify once I’m fully synced and back in action. 🙏
Posted 5 months ago
Due to a series of unrelated but cascading events, unplanned node maintenance is happening right now - should be back online in about 48 hours.
Posted 6 months ago
Built this to simplify node ops:
lnd-telemetry — a Python script to collect on-chain + channel stats, fee rates, and BOS data from your Dockerized node.
Collaborated with Duchess AI to get this polished.
Source: bitbucket.org/sniffit/lnd-telemetry
lnd-telemetry — a Python script to collect on-chain + channel stats, fee rates, and BOS data from your Dockerized node.
Collaborated with Duchess AI to get this polished.
Source: bitbucket.org/sniffit/lnd-telemetry
⚡ 30 Days in the Routing Trenches
Posted 6 months ago
🆔 Lightning Node ID: 0263a27989d64b6eca6958cfb60cc05fc641db6a258e91b9274d7042dd19bb8c88
📅 Start Date: May 16, 2025
📊 Report Date: June 19, 2025
👤 Operator: sniffit (SEA 🇲🇾)
🧭 Background
I'm currently a Cloud Infrastructure Engineer rooted in *nix systems and based in Southeast Asia.
With years of experience managing business-critical systems — from Oil & Gas to global procurement and multi-tenant platforms — this node is shaped by operational discipline and uptime-first thinking.
This Lightning node isn’t a side project — it’s the embodiment of a philosophy:
With years of experience managing business-critical systems — from Oil & Gas to global procurement and multi-tenant platforms — this node is shaped by operational discipline and uptime-first thinking.
This Lightning node isn’t a side project — it’s the embodiment of a philosophy:
🔁 We route value. Our sats move with purpose.
🧱 We’re grounded, sovereign, and strategic.
Built atop a lean home lab stack with Proxmox, Docker, and internal orchestration, this node is designed for resilience — not reliance.
📆 30 Days of Deployment: Insights & Learnings
✅ Uptime & Stack
- Running bitcoind, lnd, btc-rpc-proxy (soon), bos, custom monitoring, and DDNS scripts.
- Designed for dynamic IP handling with automated restarts on changes.
📡 Channel Strategy
- Selective peer opens and trims.
- Routing paths, fee flow, and peer quality continuously assessed via Balance of Satoshis (BoS).
- Liquidity is circulated, not hoarded. Our sats serve movement.
🛠 Automation Built
- Lightweight telemetry stack to track sink nodes, flow patterns, and peer behavior.
- Event-driven service handlers for network health and continuity.
🌍 Regional Commitment
- Prioritized SEA-based peers and routing-first operators.
- Aligned with LN+’s spirit: collaborative liquidity, not one-way drains.
🌐 Philosophy
This isn’t about leaderboard clout — it’s about meaningful connectivity and lasting impact.
It’s about helping shape a resilient, peer-aligned Lightning fabric — especially in underrepresented regions like SEA.
It’s about helping shape a resilient, peer-aligned Lightning fabric — especially in underrepresented regions like SEA.
💡 A sat moved is a story told —
a business supported, a freedom preserved,
a border crossed without permission.
We’re here to serve the flow. And the flow matters.
🔭 What’s Next
- Continued analysis of sink node behavior.
- Expansion of internal telemetry and liquidity heuristics.
- Collaborative insights with LN+ admins and SEA partners.
🤝 Let’s Connect
🧭 Are you operating in Southeast Asia?
⚙️ Building tooling for routing infrastructure?
🔁 Focused on purposeful liquidity?
⚙️ Building tooling for routing infrastructure?
🔁 Focused on purposeful liquidity?
Let’s collaborate — and route with intention.
📬 Reach out.
🛰 Let’s build pathways with purpose.
⚡ One mSAT at a time.
🛰 Let’s build pathways with purpose.
⚡ One mSAT at a time.