HODLmeTight's Posts

A Guide for Situations Where Your Lightning Close-Transaction Fails to Close the Channel

Posted 3 months ago

We've all been been there, a channel gets closed, no-matter yet if it was a cooperative or force-closed, but somehow the close transaction hangs around and doesn't get confirmed. You want to clear the house, or need the pending onchain sats, but wonder how to go about it? Follow this guide and you'll hopefully find remedy.

But before, read this article from LND. Then read it again. Once you think you understand it, read it again. It's not super cohesive, since a number of things from the top need to be understood to grasp the section Channel closing, so ensure you spend some time in there.

Situation

This is written for LND, but similar methodologies are applicable for CLN. Eclair folks know their way around anyway. You identified a channel-close, either in one of the GUIs you use, or from a Bot notification. But it doesn't confirm. First thing to do is SSH into your node via ssh user@node-ip and get an overview of all pending (open, and close) with lncli pendingchannels.

For umbrel ☂️, you need to jump through some loops for the command lncli . You need to add this whole string before: ~/umbrel/scripts/app compose lightning exec -it lnd lncli. To simplify things, create a temporary alias with alias lncli="/home/umbrel/umbrel/scripts/app compose lightning exec -it lnd lncli", then all of the above commands should work.

{  
  "channel": {  
	"remote_node_pub": "123456789889786857463636288291929458767688493999392324",  
	"channel_point": "dhksfkeh8458734kldjfksd39393932033fkk3003022244:1",  
	"capacity": "10000000",  
	"local_balance": "9999711",  
	"remote_balance": "70137",  
	"local_chan_reserve_sat": "10053",  
	"remote_chan_reserve_sat": "10053",  
	"initiator": "INITIATOR_REMOTE",  
	"commitment_type": "ANCHORS",  
	"num_forwarding_packages": "2",  
	"chan_status_flags": "ChanStatusBorked|ChanStatusCommitBroadcasted|ChanStatusLocalCloseInitiator"  
  },  
  "limbo_balance": "9999711",  
  "commitments": {  
	"local_txid": "eeof599490930002dcc0db3bfa789eb4afbd89a37794a4c49a5f529ddb1b23e85b3",  
	"remote_txid": "ta48849fkkek39947uuc80371ec96a88297465821ee79ff0b85f8624f83a7455f1cc",  
	"remote_pending_txid": "",  
	"local_commit_fee_sat": "2510",  
	"remote_commit_fee_sat": "2510",  
	"remote_pending_commit_fee_sat": "0"  
  },  
"closing_txid": "eeof599490930002dcc0db3bfa789eb4afbd89a37794a4c49a5f529ddb1b23e85b3"

What we see here is that our LND sees that channel as borked, we have a an opening_tx (channel_point:vout), a local_txand a remote_tx. What's usually the next step, we wonder why it's not closing, so we copy the

"closing_txid": "eeof599490930002dcc0db3bfa789eb4afbd89a37794a4c49a5f529ddb1b23e85b3"

and copy it into our favorite mempool tool. Just for guiding sake, we check with mempool.space, and now it's the time to split into two scenarios:

Scenario 1: mempool.space shows the closing-tx, but it's unconfirmed and the fee is too low

What many runners don't know, mempool.space runs with a mempool-size of 2GB, while most standard node-systems run with just 300MB. This is usually enough, but not in high fee environments. So let's check if your local mempool has it:

Similar to lncli with umbrel ☂️ you need to follow a similar approach for bitcoin-cli:  ~/umbrel/scripts/app compose bitcoin exec -it bitcoind bitcoin-cli for the one off-command, or alias bitcoin-cli="~/umbrel/scripts/app compose bitcoin exec -it bitcoind bitcoin-cli"

bitcoin-cli getrawtransaction eeof599490930002dcc0db3bfa789eb4afbd89a37794a4c49a5f529ddb1b23e85b3
error code: -5
error message:
No such mempool or blockchain transaction. Use gettransaction for wallet transactions.

In this case your mempool probably purged the transaction, or LND never published it due to it's low commitment fee. Let's jump to the next section to see what we can do about it.

Scenario 2: mempool.space shows Transaction not found

Okay, we can assume that your closing-tx probably never hit the public-mempool, we can cut things short here and move to the next section, but just to be safe, let's check if our local bitcoin-mempool has it:

bitcoin-cli getrawtransaction eeof599490930002dcc0db3bfa789eb4afbd89a37794a4c49a5f529ddb1b23e85b3
error code: -5
error message:
No such mempool or blockchain transaction. Use gettransaction for wallet transactions.

Complication

Why are there transactions of channel-closes with too low fee? Can't LND figure out what fee is necessary and get things just out there? I want my funds!

Let's assume best intend, no-one built that protocol to make things intentionally complicated. But since you have two parties involved in the process, and we should assume no-trust between those, we need to install processes to keep things fair, balanced and transparent. Read up on the HTLC-detail guide from Elle Mouton to make yourself more familiar with the details.

In short, we have 2 (out of 3) possible ways for a channel-close:

  1. cooperative-close, where both parties are online, one of the two sends a channel-close signal, and then both nodes get to court and negotiate a closing price. Since only the opener pays for the closing-cost, it's imminent it's not sky high. There are settings in lnd.conf like coop-close-target-confs=100 which help to keep things rational from your side if the other side triggered the close.
  2. force- or uncooperative-close, where either one of the two parties is unresponsive / offline, or other protocol / node-OS disagreements happen (like expiring HTLCs). In either case one of the two parties force-close, there is no court-negotiation, but based on the last channel_commit agreement the two nodes handshaken on, will be used for the closing-fee. Those agreements are also based on a few lnd.conf settings, like max-commit-fee-rate-anchors and bitcoind.estimatemode.

Wonder what your channel-peers current commit-fee is? And whether they use anchors or static channel-fee commitments? Check out this bash-script, among other Node-Tools available as open-source.

Either have their intricacies, but when the channel-close is not in your local mempool, or too low fee in the public mempool, you can't get the channel closed and the onchain-sats stay locked away.

Proposed Solutions

We won't outline every possible scenario, but a couple of most happening ones. Some will work, others not. To avoid this article becoming a master-class, let's go through a couple of those which are doing no-harm, but helping you worse case to get some more information what's happening.

For each of those attempts, prepare a second terminal window to your node, and filter (grep) the channel-point and pubkey of the channel, so you'll be more informed after of what's happening. This terminal will monitor your lnd.log and show only entries where either the Pubkey of your to be closed channel-peer, or your mutual channel-point is triggered.

sudo tail -f .lnd/logs/bitcoin/mainnet/lnd.log | grep -E 'PUBKEYOFCHANNELPARTNER|CHANNELPOINT:VOUT'


While Terminal 2 stays open, let's move to Terminal Window 1 for the next couple of commands. We'll try one after the other, and this guide will be verbose if there's risk or sat-spending involved. It's sorted from lowest risk / spend to highest, top down.

1. Reconnect

Many coop-close attempts fail, because when both nodes go to court, there is a possibility to not find consensus on the closing fee. Let's see if this is the reason, you'll see entries like commit-fee proposal staggering up, and whether you'll find consensus lncli disconnect PUBKEY && lncli connect PUBKEY@ADDRESS:PORT If the closing attempts fail, you'll see that each time, the delta between your node and your channel-peer will get closer. Eventually it'll work, if not, check your lnd.conf settings and eg change bitcoind.estimatemode=ECONOMICAL to bitcoind.estimatemode=CONSERVATIVE.

2. Restart LND

Sometimes closing attempts miss a broadcast due to a missing connection to bitcoind. Restarting LND rebroadcasts every pending channel-close and sweeps, so give this a shot and check your terminal 2 while LND is taking it's time to come back up. Really, you need some patience here.

3. Reinitiate coop-close (and force-close)

Try to close the channel again, in rare cases this would solve something which 1. didn't: lncli closechannel --chan_point CHANPOINT:VOUT --sat_per_vbyte XX. This may offer some more insight in your Terminal 2 window. You can do the same for the force-close, just need to adjust the command slightly. For the record, I never had this situation, but it's not doing any harm in case you know it's a force-close anyway: lncli closechannel --chan_point CHANPOINT:VOUT --force. We can omit the target sat/vbyte setting since force-closes use the last agreed commit-fee of both you and your peer.

4. Increase local Mempool size

This solves for a couple of things, both Scenario 1 and Scenario 2 above, at least as a preparation. Open your bitcoind-configuration with sudo nano .bitcoin/bitcoin.conf to lookup and edit the following entry. Take it to 600 MB to capture all tx of the past two weeks, or up to 2000, (2GB) if not sufficient:

# Increase Mempool size
maxmempool=600


Restart bitcoind (typically sudo systemctl restart bitcoind to take this setting into effect. And either LND already restarted by itself, or trigger a manual restart with sudo systemctl restart lnd. Watch Terminal 2, to see whether your 300MB previous mempool was too limited to accept your LND-closing. With 2GB, that limitation is lifted (a little) and potentially allows your LND to broadcast the closing-tx.

5. Manually broadcast the closing-transaction

Eventually 4. allows your local mempool to pick up the closing transaction. Sometimes it doesn't. Or your mempool has it, but the public mempool didn't. Without either your bitcoind node, or the public mempools knowing about this tx, your channel can't get closed. In this event, let's look at how to broadcast it manually and see what response we get in both Terminals.

  1. Local Mempool check: Follow Scenario 1.
  2. If it's a No, does mempool.space have it? Then open the tx there, and grab the RAW-tx by appending the txid here: https://mempool.space/api/tx/[TXID]/hex. Copy the whole long string, and execute the local broadcast in your Terminal 1 with lncli wallet publishtx RAWTX.
  3. If that's a No too, check your LND, it should have it (only in rare circumstances, like SCBs, LND doesn't have the raw-tx: lncli listchaintxns --start_height 818181 --end_height -1. Substitute 818181 with the current mempool height to filter the transaction output showing only not confirmed pending transactions (utxos). You'll get the raw-tx there, and can use it with lncli wallet publishtx RAWTX as above.
  4. If your local Mempool has it, but mempool.space doesn't, there is no harm broadcasting it there as well. Follow Scenario 1 to retrieve the raw-txid from your local mempool, and copy it in here.

6. Increase the closing-fee

So mempools know about your closing-tx, but it's way too low. Let's assume it hangs at 10sat/vbyte (like many pending closing-tx these days), but let's also assume we won't see <30sat/vbyte anytime soon. Because you'd always have the option to wait for 10sats, your tx will eventually confirm. But you don't want to wait, then you'll need to pay up. Disclaimer: This one is costing you. No matter who the opener of the channel was (who always pays for both un- and cooperative-closings), the next step costs You. So be concious about it.

The LND Guide linked above has a succinct closing channels section. And since you inhaled that guide, you'll know the next steps:

  1. For Coop-Closes, you can lncli wallet bumpfee -h to make yourself familiar with the options. It's pretty self-explanatory, lncli wallet bumpfee --conf_target 6 CHANPOINT:VOUT will ask your local mempool for a sat/vbyte target and set the new closing fee. With lncli wallet bumpfee --sat_per_vbyte 36 CHANPOINT:VOUT, you can set a confident 36 sats/vbyte yourself. Watch Terminal 2 and see what lnd.log tells you.

  2. For Force-Closes, you need a so called CPFP (Child Pays For Parent) transaction, which will become a child of your existing closing fee, and at the same time expensive enough, that both parent + child and their fees will be attractive enough for a miner to pick up and mine it. You can either do some pre-calc with a helper script, or try current mempool medium fee x 2. Not a problem if you don't start high enough, you can follow step 1. above here with this new child-transaction and bump it's fee with RBF any time later: lncli wallet bumpclosefee -h for the options:

  • lncli wallet bumpclosefee --conf_target 6 CHANPOINT:VOUT will ask your local mempool for a sat/vbyte target and set the fee for the child-tx.
  • lncli wallet bumpclosefee --sat_per_vbyte 100 CHANPOINT:VOUT will target 100vByte for the combined tx, so your child will have a much higher (probably 2x) effective rate for both your parent and your child to be mined at a 100.

Note that none of this works if your local mempool doesn't have that txid. So follow steps 1-5 before trying this one. Now but if it worked, you'll be prompted with a new txid for your child, which you can later check in your local mempool, and ideally it'll land in the public mempools to be mined as well. If it ended up too low, bump it with RBF (replace by fee) described in point 1, but targeting your child.

7. Go out of band

Just for completion, there are various miners who offer their mining service for your tx for a hefty cost. This sometimes might be necessary, eg for your channels where the other side force-closed on you, but you only have a static channel-commitment. If your peer is unresponsive or not cooperative to do a CPFP on their side, you'll wait until forever. Check your favorite search engine, ViaBTC or mempool.space for their acceleration services.

Hope you enjoyed this article. Please do share feedback and suggestions for improvement. If this guide was of any help, I'd appreciate if you share the article with others, give me a follow on X  or nostr, perhaps even donating some sats to hakuna@getalby.com. I'm also always grateful for incoming channels to my node: HODLmeTight.

Beta Launched: LND & CLN fully managed Hybrid VPN available for your node today

Posted almost 2 years ago

A challenge many node-runners are facing today is reliability on one anonymous connection point to the lightning network: Tor
It's serving us all well, allowing mostly trustless and hidden connection point, to the payment infrastructure of the future, with little to no costs.

But Lightning payment routing requires more than tat: our needs are about privacy, that's our DNA, but for successful routing, reliability, connectivity and speed come into play. Relying your node connectivity to a single service Tor is a risk, as anyone running a lightning node can testify. For fast communication establishing clearnet connections between nodes is inevitable.

The effort of creating a valuable “clearnet over VPN” node is quite high and intense because it touches several disciplinaries not every node runner is comfortable with. You could go hybrid, we've shared guides to get there in previous posts, but announcing your home IP publicly hampers security and anonymity.

Therefore we came to the conclusion that this process has to be simplified a lot. In the last few weeks we put together all the pieces that we think provide the best of both worlds to make it as easy as possible to go hybrid.
Hybrid connectivity means you're connected via Tor and Clearnet at the same time, allowing more reliability and speed.

Hence, we're launching the Beta of Tunnel Sats.
With this solution, we want to address the growing pain-points running your Lightning Node with Tor only, or exposing your home IP.

What does that look like?
Visit http://tunnelsats.com today & read through the 3 part activation routine:
  1. sign up for a tunnel-server we run for you
  2. download the open-sourced script installing the tunnel
  3. do your own changes in your LND / CLN config

We expect that in ~ 5-10 minutes, you're done and gone hybrid. You will see better connectivity, faster response times and hence more routing, but with no compromises on privacy and security!

Have more questions? Check out the FAQ or join our Telegram channel.

Convinced already?
The workflow is tested with umbrel, mynodebtc and RaspiBlitz, and works on bare-metal raspibolt too.
We have 10+ Alpha customers and our own test-nodes with vivid testing processes.

Join the test-phase with a significant discount for $1 / month today if you like to go hybrid.

Setup LNbits on a VPS, connected to your Lightning Network Node through a secured tunnel

Posted almost 2 years ago

VPS-LNbits


Here's my current setup shared with you, while your objective and intend can be manyfold, you may

  • have a dynamic IP from your Internet Service Provider
  • want to hide your home IP from the world, for whatever reason
  • desire to decrease your Lightning Node HTLC Routing times, so instead of running Tor only, you want Clearnet availability, too
  • want others to leverage the LN Services you want to offer, via LNBits, BTCPay or others
  • get a domain-name or use a free-domain host such as DuckDNS to point to your LNBits instance
  • are just curious and want to tinker around a bit, because it's good to have those skills when demand for experience continues to rise

This is a long post, so please follow this linked Post on Github with better code-formatting, TOC and some casual updates.
Lastly, if you want a simpler LND Hybrid VPS guide, without the LNBits piece, check out the following guide.


Pre-Amble

Objective
Your LNbits instance installed on a cheap, but anonymous Virtual Private Server (VPS), connected to your own, non-custodial Lightning-Network Node running on both Tor and Clearnet (Hybrid-Mode).

Challenge
We want payment options with ₿itcoin to be fast, reliable, non-custodial - but the service should ideally not be easily to be identifiable. LNBits provides a quick and simple setup today, for instance on your Raspiblitz oder Umbrel, however, if you want to build the setup from scratch on your own, you have to bypass a number of technical discovery and hurdles.

Proposed Solution
There are plenty of ways how to solve for this. This creates hesitance to implement, especially when you're not very technical. This guide is supposed to provide one approach, whilst there remain many other ways to Rome. Take your time following this through. It might take you 1-2hrs, depending highly on your skill. So don't go in here in a rush.

Pre-Reads
This guide heavily relies on the intelligence and documentation of others 🙏, but putting those together to one picture creates the last 10% hurdle which is sometimes the highest. Have a careful read through the following articles, to get a deeper understanding on some of the lighter references we'll be using further below

Pre-Requisites
  • running lnd-0.14.2-beta or later. This can either be Umbrel, Raspiblitz, MyNode or even a bare RaspiBolt
  • Technical curiosity and not too shy to use the command-line
  • A domain name or a subdomain registered at DuckDNS
  • An SSH connection to your node, and to the VPS as well. On Windows, use something like putty and get putty-gen, too
  • VPS Account at DigitalOcean or any alternative VPS Solution out there offering similar capabilities (it's critical they offer a public IP for you)
Disclaimer: this is a ref link, gets you $100 in credit over 60 days, while the cheapest option we use here comes at a cost of $5/month.

Preperations
The better we prepare, the more we can deal with blindspots and the unexpected.

Make notes
It's generally advised to document your own steps. Make a bucket-list of things you've done, and a ToDo to go through in case your environment changes. Imagine yourself 18 months from now, you want to setup this new hardware-node: Will you remember all the steps or extra corners you've taken? Suggested Laundry-List, you can tick them off while you go through this guide

  •  IP-Adresses of VPS external, VPS Tunnel, Node Tunnel
  •  Ports which needs forwarding
  •  ToDos
  •  Questions / open items
Visualize
Some of us are visual people. Draw your diagram to get an idea how you want things to flow
Schema we aim for


Secure
It goes without saying, but this guide doesn't go into the necessary security steps in detail, and can't take on liability for any things breaking or losing funds. Ensure you don't get reckless, start with small funds you're ok to lose. Keep an eye on developments or in touch with the active Telegram Groups, to get news and updates with low delays. Also, would recommend to do those steps with a peer, so you follow a second pair of eye review. Lastly, 2fa / yubikeys are your friends!

Let's get started (LFG!)
Well, let's get into it, shall we?!

1) Lightning Node
We will consider you have your Lightning Node up and running, connected via Tor and some funds on it. You also have SSH access to it and administrative privilidges

2) VPS: Setup
In case you don't have a VPS provider already, sign-up with my referal or pick another which provides you with a static IP and cheap costs. Maybe you even prefer one payable with Lightning ⚡. In case you go for DigitalOcean, here are the steps to create a Droplet, shouldn't take longer than a few minutes:

  •  add a new Droplet on the left hand navigation
  •  chose an OS of your preference, I have Ubuntu 20.04 (LTS) x64
  •  take the Basic Plan with a shared CPU, that's enough power. You can upgrade anytime if necessary
  •  Switch the CPU option to "Regular Intel with SSD", which should get you down to $5/month
  •  You don't need an extra volume, but pick a datacenter region of your liking
  •  Authentication: Chose the SSH keys option and follow the next steps to add your public keys in here for secure access. For Windows, with putty and putty-gen referenced above, you should be relatively quick to use those keys instead of a password. For Linux users, you probably know your ways already.
  •  Add backups (costs), Monitoring or IPv6 if you wish to, however this guide won't use any of those items
  •  Lastly, chose a tacky hostname, something which resonates with you, eg myLNBits-VPS
After a few magic cloud things happening, you have your Droplet initiated and it provides you with a public IPv4 Adress. Add it to your notes! In this guide, I'll refer to it as VPS Public IP: 207.154.241.101

3) VPS: Connect to your VPS and tighten it up
Connect to your VPS via SSH root@207.154.241.101 and you will be welcomed on your new, remote server. Next steps are critical to do right away, harden your setup:

  •  Update your packages: apt-get update and apt-get upgrade
  •  Install Docker: apt-get install docker.io tmux
  •  Enable Docker automated start: systemctl start docker.service
  •  Enable Uncomplicated Firewall (UFW) and add ports to be allowed to connected to:
$ apt install ufw
$ ufw default deny incoming
$ ufw default allow outgoing
$ ufw allow OpenSSH
$ ufw allow 80 comment 'Standard Webserver'
$ ufw allow 443 comment 'SSL Webserver'
$ ufw allow 9735 comment 'LND Main Node 1'
$ ufw enable

  •  Follow further hardening steps, eg setting up non-root users for additional security enhancements.
  •  Install fail2ban to protect your SSH user, it runs automatically on it's own sudo apt install fail2ban

4) VPS: Install OpenVPN Server
Now we will get OpenVPN installed, but using a Docker Setup like Krypto4narchista suggests here. It's easier to setup, but needs some tinkering with port forwarding, which we will go into in a bit.

  •  export OVPN_DATA="ovpn-data" which sets a global-name placeholder for your VPN to be used for all the following commands. You can make this permanent by adding this to survive any reboot via nano .bashrc, add it to the very bottom => CTRL-X => Yes.
  •  docker volume create --name $OVPN_DATA notice how the $ indicates picking up the placeholder you have defined above
  •  docker run -v $OVPN_DATA:/etc/openvpn --rm kylemanna/openvpn ovpn_genconfig -u udp://207.154.241.101, whereby you need to adjust the 207.154.241.101 with your own VPS Public IP.
  •  docker run -v $OVPN_DATA:/etc/openvpn --rm -it kylemanna/openvpn ovpn_initpki this generates the necessary VPN certificate password. Take your password manager and create a secure pwd, which you will store safely. It will be needed once we create client-configuration files for your node to connect later.
  •  docker run -v $OVPN_DATA:/etc/openvpn -d -p 1194:1194/udp -p 9735:9735 -p 9735:9735/udp -p 8080:8080 -p 8080:8080/udp --cap-add=NET_ADMIN kylemanna/openvpn this works under two assumptions. If any of those aren't true, you need to adjust your settings, either on your node, or by starting the docker container with different ports:
    1. your current LND Node configuration is listening on port 9735, which you can verify by looking into your cat ~/.lnd/lnd.conf => [Application Options] => listen=0.0.0.0:9735
    2. your LND RestLNDWallet is listening on port 8080, same location under [Application Options] => restlisten=0.0.0.0:8080
Your OpenVPN Server is now running, which means the Internet can now connect to your VPS via ports 80, 443, 9735 (and 22 SSH), and it has a closed tunnel established on port 1194. You need to complement your notes with the IP-Adresses which are essentially added with the running server.

  •  CONTAINER-ID: docker ps to list your docker container. In the first column, you will find the CONTAINER-ID, usually a cryptic 12-digit number/character combination. Copy into the clipboard and make a note of it.
  •  Docker Shell: Get into the container, with docker exec -it <CONTAINER-ID> sh.
  •  VPS Docker IP: Run ifconfig and you typically find 3 devices listed with IPs assigned. Make a note of the one with eth0, which is your own VPS Docker IP: 172.17.0.2. Type exit to get out of the docker-shell.

5) VPS: Install LNBits
Next we will install LNBits on this server, since it'll allow to keep your node independent and light-weight. It also allows to change nodes swiftly in-case you need to move things. We won't install it via Docker (like Umbrel does), but do the implementation based slightly on their Github Installation Guide. You can also follow their own, excellent video walkthrough here. Just don't use Ben's commands, since these are a little dated. Since we assume you have followed the hardening guide above to add additional users, we will now have to use sudo in our commands.

$ sudo apt-get install git
$ git clone https://github.com/lnbits/lnbits-legend
$ sudo apt update
$ sudo apt install pipenv
$ cd lnbits-legend
$ pipenv --python 3.9 shell
$ pipenv run pip install -r requirements.txt
$ pipenv run python -m uvicorn lnbits.__main__:app

Now when this is successfully starting, you can abort with CTRL-C. We will come back to this for further configuration editing LNBits' config-file to our desired setup.

6) VPS: Retrieve the OpenVPN config & certificate
In this section we'll switch our work from setting up the server towards getting your LND node ready to connect to the tunnel. For this, we will retrieve and transfer the configuration file from your VPS to your node.

  •  docker run -v $OVPN_DATA:/etc/openvpn --rm -it kylemanna/openvpn easyrsa build-client-full NODE-NAME nopass whereby NODE-NAME should be changed to a unique identifier you chose. For example, if your LND Node is called "BringMeSomeSats", I suggest to use that - with all lowercase.
  •  docker run -v $OVPN_DATA:/etc/openvpn --rm kylemanna/openvpn ovpn_getclient NODE-NAME > NODE-NAME.ovpn which will prompt you to provide the secure password you have generated earlier. Afterwards, it'll store bringmesomesats.ovpn in the directory you currently are.

Into the Tunnel
We have installed the tunnel through the mountain, but need to get our LND Node to use it.

7) LND Node: Install and test the VPN Tunnel
Now switch to another terminal window, and SSH into your Lightning Node. We want to connect to the VPS and retrieve the VPN-Config file, to be able to establish the tunnel

$ cd ~
$ mkdir VPNcert
$ scp user@207.154.241.101:/home/user/bringmesomesats.ovpn /home/admin/VPNcert/
$ chmod 600 /home/admin/VPNcert/bringmesomesats.ovpn
$ sudo cp -p /home/admin/VPNcert/bringmesomesats.ovpn /etc/openvpn/CERT.conf

Note: You need to adjust user, the VPS Public IP and the absolute directory where the ovpn file is stored. We keep a copy of the cert file in the home directory for backup, but the actual file we use is CERT.conf.

Now we need to install OpenVPN, start it up to see if it works.

Important Warning: Depending on your network-setup, there is a slight chance your LND Node Service gets interrupted. Be aware there might be small down-times of your lightning node, as we will reconfigure things. Be patient!

$ sudo apt-get install openvpn
$ sudo systemctl enable openvpn@CERT
$ sudo systemctl start openvpn@CERT

You should see something similiar to the following output. Note this one line indicating the next important IP Adress VPN Client IP: 192.168.255.6. Make a note of it, we need it for port-configuration at the server, soon.

* openvpn@CERT.service - OpenVPN connection to CERT
     Loaded: loaded (/lib/systemd/system/openvpn@.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-04-06 13:11:13 CEST; 4s ago
       Docs: man:openvpn(8)
             https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
             https://community.openvpn.net/openvpn/wiki/HOWTO
   Main PID: 1514818 (openvpn)
     Status: "Initialization Sequence Completed"
      Tasks: 1 (limit: 18702)
     Memory: 1.0M
        CPU: 49ms
     CGroup: /system.slice/system-openvpn.slice/openvpn@CERT.service
             `-1514818 /usr/sbin/openvpn --daemon ovpn-CERT --status /run/openvpn/CERT.status 10 --cd /etc/openvpn --config /etc/openvpn/CERT.conf --writepid /run/openvpn/CERT.pid

Apr 06 13:11:13 debian-nuc ovpn-CERT[1514818]: WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1541', remote='link-mtu 1542'
Apr 06 13:11:13 debian-nuc ovpn-CERT[1514818]: WARNING: 'comp-lzo' is present in remote config but missing in local config, remote='comp-lzo'
Apr 06 13:11:13 debian-nuc ovpn-CERT[1514818]: [207.154.241.101] Peer Connection Initiated with [AF_INET]207.154.241.101:1194
Apr 06 13:11:14 debian-nuc ovpn-CERT[1514818]: TUN/TAP device tun0 opened
Apr 06 13:11:14 debian-nuc ovpn-CERT[1514818]: net_iface_mtu_set: mtu 1500 for tun0
Apr 06 13:11:14 debian-nuc ovpn-CERT[1514818]: net_iface_up: set tun0 up
Apr 06 13:11:14 debian-nuc ovpn-CERT[1514818]: net_addr_ptp_v4_add: 192.168.255.6 peer 192.168.255.5 dev tun0

The tunnel between your LND Node and your VPS VPN is established. If you need to troubleshoot, call the systemctl journal via sudo journalctl -u openvpn@CERT -f --since "1 hour ago"

8) VPS: Add routing tables configuration into your droplet docker
Back to your terminal window connected to your VPS. We have the VPN Client IP: 192.168.255.6 now, which we need to tell our VPS where it should route those packets to. To achieve that, we'll get back into the docker container and add IPTables rules.

  •  Remember how to get into the container? Arrow-up on your keyboard, or do docker ps and docker exec -it <CONTAINER-ID> sh
  •  Doublecheck your VPN Client IP, and adjust it in the following IPtables commands you enter into the container and confirm with Enter
$ iptables -A PREROUTING -t nat -i eth0 -p tcp -m tcp --dport 9735 -j DNAT --to 192.168.255.6:9735
$ iptables -A PREROUTING -t nat -i eth0 -p udp -m udp --dport 9735 -j DNAT --to 192.168.255.6:9735
$ iptables -A PREROUTING -t nat -i eth0 -p tcp -m tcp --dport 18080 -j DNAT --to 192.168.255.6:8080
$ iptables -t nat -A POSTROUTING -d 192.168.255.0/24 -o tun0 -j MASQUERADE
$ exit

What we basically do here, is assign a ruleset to say: As soon a packet arrives at device eth0 on port 9735/udp and /tcp, forward it to the VPN client at 192.168.255.6:9735, and vice versa everything at device tun0. If you have different ports or IPs, please make adjustments accordingly. What you also see, is a port 8080 preperation for LNBits packets, we'll get to this later.

9) LND Node: LND adjustments to listen and channel via VPS VPN Tunnel
We switch Terminal windows again, going back to your LND Node. A quick disclaimer again, since we are fortunate enough to have plenty of good LND node solutions out there, we cannot cater for every configuration out there. Feel free to leave comments or log issues if you get stuck for your node, we'll be looking at the two most different setups here. But this should work very similar on MyNode, Raspibolt or Citadel.

Be very cautious with your lnd.conf. Make a backup before with cp /mnt/hdd/lnd/lnd.conf /mnt/hdd/lnd/lnd.bak so you can revert back when things don't work out. The brackets below indicate the section where each line needs to be added to. Don't place anything anywhere else, as it will cause your LND constrain from starting properly.

Adjust ports and IPs accordingly!

Raspiblitz / Raspibolt settings
LND.conf adjustments, open with sudo nano /mnt/hdd/lnd/lnd.conf

[Application Options]

Command | Description
externalip=207.154.241.101:9735 | # to add your VPS Public-IP
nat=false | # deactivate NAT
tlsextraip=172.17.0.2 | # allow later LNbits-access to your rest-wallet API

[tor]

Command | Description
tor.active=true | # ensure Tor is active
tor.v3=true | # with the latest version. v2 is going to be deprecated this summer
tor.streamisolation=false | # this needs to be false, otherwise hybrid mode doesn't work
tor.skip-proxy-for-clearnet-targets=true | # activate hybrid mode

CTRL-X => Yes => Enter to save

RASPIBLITZ CONFIG FILE sudo nano /mnt/hdd/raspiblitz.conf since Raspiblitz has some LND pre-check scripts which otherwise overwrite your settings.

Command | Description
publicIP='207.154.241.101' | # add your VPS Public-IP
lndPort='9735' | # define the LND port
lndAddress='207.154.241.101' | # define your LND public IP address

CTRL-X => Yes => Enter to save

LND Systemd Startup adjustment

Command | Description
sudo nano /etc/systemd/system/lnd.service | edit the line 15 where it starts your LND binary, and add the following parameter: ExecStart=/usr/local/bin/lnd ${lndExtraParameter}
sudo systemctl restart lnd.service | apply changes and restart your lnd.service. It will ask you to reload the systemd services, copy the command, and run it with sudo. This can take a while, depends how long your last restart was. Be patient.
sudo tail -n 30 -f /mnt/hdd/lnd/logs/bitcoin/mainnet/lnd.log | to check whether LND is restarting properly
lncli getinfo | to validate that your node is now online with two uris, your pub-id@VPS-IP and pub-id@Tor-onion

"03502e39bb6ebfacf4457da9ef84cf727fbfa37efc7cd255b088de426aa7ccb004@207.154.241.101:9736",
     "03502e39bb6ebfacf4457da9ef84cf727fbfa37efc7cd255b088de426aa7ccb004@vsryyejeizfx4vylexg3qvbtwlecbbtdgh6cka72gnzv5tnvshypyvqd.onion:9735"

  •  Restart your LND Node with sudo reboot

Umbrel / Citadel settings
LND.conf adjustments, open with sudo nano /home/umbrel/umbrel/lnd/lnd.conf

[Application Options]

Command | Description
externalip=207.154.241.101:9735 | # to add your VPS Public-IP
nat=false | # deactivate NAT
tlsextraip=172.17.0.2 | # allow later LNbits-access to your rest-wallet API

[tor]

Command | Description
tor.active=true | # ensure Tor is active
tor.v3=true | # with the latest version. v2 is going to be deprecated this summer
tor.streamisolation=false | # this needs to be false, otherwise hybrid mode doesn't work
tor.skip-proxy-for-clearnet-targets=true | # activate hybrid mode

CTRL-X => Yes => Enter to save

LND Restart to incorporate changes to lnd.conf

Command | Description
cd umbrel && docker-compose restart lnd | this can take a while, depends how long your last restart was. Be patient.
tail -n 30 -f ~/umbrel/lnd/logs/bitcoin/mainnet/lnd.log | check whether LND is restarting properly
~/umbrel/bin/lncli getinfo | validate that your node is now online with two uris, your pub-id@VPS-IP and pub-id@Tor-onion

"03502e39bb6ebfacf4457da9ef84cf727fbfa37efc7cd255b088de426aa7ccb004@207.154.241.101:9736",
        "03502e39bb6ebfacf4457da9ef84cf727fbfa37efc7cd255b088de426aa7ccb004@vsryyejeizfx4vylexg3qvbtwlecbbtdgh6cka72gnzv5tnvshypyvqd.onion:9735"

  •  Restart your LND Node with sudo reboot

Warning: This guide did not verify yet, if and how the docker LND service on Umbrel & Citadel needs to be adjusted to channel clearnet packets via tunnel. We will add peer-reviewed adjustments here from Umbrel / Citadel devs. Until then, consider this highly experimental, it might fail.


Connect VPS LNBits to your LND Node

The traffic line between the two connection points is established. Worth noting that this can be extended: In case you run more than one node, just repeat the steps above for additional clients. Now, let's get LNBits talk to your node.

10) LND Node: provide your VPS LNBits instance read / write access to your LND Wallet
Assuming LND restarted well on your LND Node, your LND is now listening and connectable via VPS Clearnet IP and Tor. That's quite an achievement already. But we want to setup LNBits as well, right? So go grab another beverage, now we'll get LNBits running. For that, let's climb another tricky obstacle; to respect the excellent security feats the LND engineering team has implemented. Since we don't want to rely on a custodial wallet provider, which would be super easy to add into LNBits, we have some more tinkering to do. Follow along to basically provide two things to your VPS from your LND Node.

Note of warning again: Both of those files are highly sensitive. Don't show them to anyone, don't transfer them via Email, just follow the secure channel below and you should be fine, as long you keep the security barriers installed in Section "Secure" intact.

  1. your tls.cert. Only with access to this file, your VPS is going to be allowed to leverage your LND Wallet via Rest-API scp ~/.lnd/tls.cert root@207.154.241.101:/root/ sends your LND Node tls.cert to your VPS, where we will use it in the next section.

  2. your admin.macaroon. Only with that, your VPS can send and receive payments xxd -ps -u -c ~/.lnd/data/chain/bitcoin/mainnet/admin.macaroon will provide you with a long, hex-encoded string. Keep that terminal window open, since we need to copy that code and use it in our next step on the VPS.

11) VPS: Customize and configure LNBits to connect to your LNDRestWallet
Now since we're back in the VPS terminal, keep your LND Node Terminal open. We'll adjust the LNBits environment settings, and we'll distinguish between necessary and optional adjustments. First, send the following commands:

$ cd lnbits-legend
$ mkdir data
$ pwd
$ cp .env.example .env
$ sudo nano .env

Worth noting, that the directory data will hold all your database SQLite3 files. So in case you consider proper backup or migration procedures, this directory is the key to be kept.

Necessary adjustments
Variable | Description
LNBITS_DATA_FOLDER="/user/lnbits-legend/data" | enter the absolute path to the data folder you created above
LNBITS_BACKEND_WALLET_CLASS=LndRestWallet | Specify that we want to use our LND Node Wallet Rest-API
LND_REST_ENDPOINT="https://172.17.0.2:8080" | Add your VPS Docker IP: 172.17.0.2 on port 8080
LND_REST_CERT="/root/tls.cert" | Add the link to the tls.cert file copied over earlier
LND_REST_MACAROON="HEXSTRING" | Copy the hex-encoded snippet from your LND Node Terminal output from Section 11.2 in here

Optional adjustments
Variable | Description
LNBITS_SITE_TITLE="HODLmeTight LNbits" | Give your Website a tacky title
LNBITS_SITE_TAGLINE="free and open-source lightning wallet" | Define the sub-title in the body
LNBITS_SITE_DESCRIPTION="Offering free and easy Lightning Bitcoin Payment options for Friends & Family" | Outline your offering
LNBITS_THEME_OPTIONS="classic, bitcoin, flamingo, mint, autumn, monochrome, salvador" | Provide different color themes, or keep it simple
CTRL-X => Yes => Enter to save | 

12) VPS: Start LNBits and test the LND Node wallet connection
As soon you got here, we got the most complex things done 💪. The next few steps will be a walk in the park. Get another beverage, and then start LNBits again in your tmux-environment

$ tmux new -s lnbits
$ cd ~/lnbits-legend
$ pipenv --python 3.9 shell
$ pipenv run python -m uvicorn lnbits.__main__:app --host 0.0.0.0

Back into the background with CTRL-B + CTRL-D

See further uvicorn startup options listed here, but with our slightly adjusted default settings, LNBits should now be running and listening on all incoming requests on port 8000. If you're impatient, add a temporary1 ufw exception to test it: sudo ufw allow 8000/tcp comment 'temporary lnbits check' and open the corresponding VPS Public IP: 207.154.241.101:8000.

If you see your own LNBits instance, with all your Optional Adjustments added, we'll go to the last, final endboss.

13) Your domain, Webserver and SSL setup
We don't want to share our IP-Adress for others to pay us, a domain name is a much better brand. And we want to keep it secure, so we need to get us an SSL certificate. Good for you, both options are available for free, just needs some further work.

Domain
While there are plenty of domain-name providers out there, we are going to use a free, easy and secure provider: duckdns.org. They do their own elevator pitch why to use them on their site. Feel free to pick another, such as Ahnames, but this guide will use the former for simplicity

  •  make an account on DuckDNS with GH or Email
  •  add 1 of 5 free subdomains, eg. paymeinsats
  •  point this domain to your VPS Public IP: 207.154.241.101
  •  Make a note of your Token
Keep the site open, we'll need it soon

VPS: SSL certificate
You want your secure https:// site to confirm to your visitor's browser that you're legit. For this, we will use Certbot to manage our SSL certificate management, even though LNBits recommends caddy. Use your own preference, we'll walk through certbot here:

$ sudo apt update
$ sudo apt install nginx certbot
$ sudo certbot certonly --manual --preferred-challenges dns

Next to a few other things, Certbot will ask you for your domain, so add your paymeinsats.duckdns.org. Then it'll prompt you to place a TXT record for _acme-challenge.paymeinsats.duckdns.org, which is basically their way to verify whether you really own this domain. To achieve this, leave the certbot alone without touching anything, and follow those steps in parallel:

  •  Open a text editor, and add this URL: https://www.duckdns.org/update?domains={YOURVALUE}&token={YOURVALUE}&txt={YOURVALUE}[&verbose=true]
  •  replace each variable2
    • domains={YOURVALUE} with your subdomain only, in our case domains=paymeinsats
    • token={YOURVALUE} with your token from your duckdns.org overview
    • txt={YOURVALUE} with the random text-snippet certbot provided you to fill in
    • optional: set verbose=true if you want 2 lines more info as a response
  •  Copy that whole string into a new Webbrowser window, and if verbose isn't set as true, it'll be as crisp as OK
  •  In a new Terminal window, install dig sudo apt-get install dnsutils to check if the world knows about you solved the challenge: dig -t txt _acme-challenge.paymeinsats.duckdns.org. Compare the TXT record entry with what Certbot provided you. If both are similar, confirm with Enter in the Certbot Terminal, so it can do it's own verification
  •  Once successful, you got your SSL certificates. Make a note in your calendar when the validation time is over, so you renew early enough. Also take note of the absolute paths of those two certificates you received.

VPS: Webserver NGINX
Uvicorn is working fine, but we'll add a more robust solution, to be able to do some caching and better log-management: nginx (engine-x). We'll add a new configuration file for your website.

Please don't forget to adjust domain names and paths below accordingly

  •  sudo nano /etc/nginx/sites-available/paymeinsats.conf to create and edit your new configuration file nginx will use
Add the following entries

server {
        # Binds the TCP port 80
        listen 80;
        # Defines the domain or subdomain name
        server_name paymeinsats.duckdns.org;
        # Redirect the traffic to the corresponding 
        # HTTPS server block with status code 301
        return 301 https://$host$request_uri;
       }

server {
        listen 443 ssl; # tell nginx to listen on port 443 for SSL connections
        server_name paymeinsats.duckdns.org; # tell nginx the expected domain for requests

        access_log /var/log/nginx/paymeinsats-access.log; # Your first go-to for troubleshooting
        error_log /var/log/nginx/paymeinsats-error.log; # Same as above

        location / {
                proxy_pass http://127.0.0.1:8000; # This is your uvicorn LNbits local host IP and port
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection 'upgrade';
                proxy_set_header X-Forwarded-Proto https;
                proxy_set_header Host $host;
                proxy_http_version 1.1; # headers to ensure replies are coming back and forth through your domain
       }

        ssl_certificate /etc/letsencrypt/live/paymeinsats.duckdns.org/fullchain.pem; # Point to the fullchain.pem from Certbot 
        ssl_certificate_key /etc/letsencrypt/live/paymeinsats.duckdns.org/privkey.pem; # Point to the private key from Certbot
}

CTRL-X => Yes => Enter to save Next we'll test the configuration and enable it by creating a symlink from sites-available to sites-enabled.

$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
$ sudo ln -s /etc/nginx/sites-available/paymeinsats.conf /etc/nginx/sites-enabled/
$ sudo systemctl restart nginx

Now the moment of truth: Go to your Website https://paymeinsats.duckdns.org and either celebrate 🍻 or troubleshoot where things could have gone wrong. If the former: Congratulations - you made it!

Hope you enjoyed this article. Please do share feedback and suggestions for improvement. If this guide was of any help, I'd appreciate if you share the article with others, give me a follow on Twitter
or even donating some sats below

LNBits rocks
I'm also always grateful for incoming channels to my node (2M min): HODLmeTight

Appendix & FAQ
I see anyone can create a wallet on my LNBits service, but I don't want that. How do I change that?
Once you have created your first user wallet, and you want only this to be accessible, go to the user-section in LNBits and notice the user-ID in the URL: /usermanager/?usr=[32-digit-user-ID]. Copy the user-id and add it to your .env file: nano ~/lnbits-legend/.env and add this to the variable LNBITS_ALLOWED_USERS="". You can comma-seperate a list of user-ids.

I'm stuck and have no idea why it's not working. Who can help?
Please add an issue on Github with your question and provide as much detail as possible. Keep it safe though, no macaroon or user-ids!

So I have LNBits running, now what?
Head over to LNBits Website and check out the plethora of options you could do. For instance, I've built a donation wallet, which is shared 50:50 between the main author and my own wallet. All automated.

Why DigitalOcean - can't we pick a VPS where we can pay with Lightning, and anonymously
Consider this guide a work-in-progress. I've picked DigitalOcean since I know what I'm doing there. Heard good things about Luna Node, it's cheaper and you can pay with sats, so will test this out next. Also happy to add further alternatives, leave comments if you think these can accomplish the same results. Fee free to provide suggestions here.

Can I add more nodes connecting to the tunnel? If so, how?
In fact, I have more than one node connected to the tunnel. You need to handle your port-forwarding appropriately, since every node needs their unique LND listen port. Eg Node 1 has 9735, Node 2 9736 and so on. Docker runs need to be called with further -p for publish-options, IPtable rules and UFW needs to be adjusted. But once you got this guide internalised, the principle should be clear. Otherwise, let me know.



LND: Tor & Clearnet - How to setup hybrid-mode

Posted over 2 years ago

Prelude and Objective


The lightning network functions in rapid growing speed as infrastructure for payments across the globe between merchants, creators, consumers, institutions and investors alike. Hence the key pillars of sustained growth are their nodes, by providing reliable, liquid, discoverable and trustless connection points between those parties.

While a growing number of nodes come online every day, a significant share of those are using Tor, which allows them to remain anonymous in a sense that they don't need to reveal their real IP address (aka Clearnet IP). However, this methodology paired with the increased demand for Bitcoin payments will continue to stretch Tor's capacity to cater for continued need of supply. It also hampers existing and new node's metric of success being a reliable and fast peering partner.

To mitigate some of ongoing Tor capacity constraints, a node runner may choose to reconsider (see Chapter Caution: Clearnet! offering both, a Tor as well as a Clearnet IP connection option. Next to the drawbacks outlined in the first section below, it has three main net benefits

  1. allows for alternative discovery, routing and peering in case your own Tor cluster is affected by capacity constraints. Even though mostly temporarily, it cuts into your reliability
  2. provides quicker routing of HTLCs, both for payment and probing. Quite nervous waiting 8 seconds for your transfer confirmation at the supermarket or bar, isn't it?
  3. offers other clearnet-only nodes to connect directly to you. Otherwise it would be required for you as Tor-only to peer-connect to them first, before they can open a channel

With those considerations in mind, have a careful read through the words of caution below, make an educated decision by yourself, and then use our guide below on how to enable a hybrid Tor & Clearnet Node.

Table of Content

  • Prelude and Objective
  • Caution clearnet!
  • Preconditions
  • Configuring hybrid-mode:
    • Static IP
    • Dynamic IP: Solution 1 - NAT/UPnP
    • Dynamic IP: Solution 2 - Dynamic DNS (DDNS)
  • Wrap-Up
  • Special Case: Clearnet through VPN

Caution: Clearnet!

A word of caution: Running a node behind the Tor network offers many advantages (anonymity, security and usability) and therefore it is currently the most recommended way. For nodes maintaining a high number of connected channels and/or have high availability requirements, Tor can be a hindrance. Tor's organic network is prone to censorship of a country's law regulation and internal failures of circuits and relays. LND also allows running clearnet nodes that do not make use of the Tor network but directly connect to peers. This requires node administrators to take care of the underlying system's security policies. At least one port (default: 9735) needs to be forwarded and exposed for remote peers to connect to. Setting up a firewall is highly recommended. Not only security is a topic to be thought about, also the risk of being localized by clearnet IP. Only use hybrid-mode if privacy is not of concern!


Preconditions

For this guide the following is required:

  • You are tech-savvy and know what you do
  • A fully installed and synchronized node (Umbrel / custom)
  • For RaspiBlitz these features will be implemented and available in Release v1.8.
  • lnd-0.14.1-beta or later
  • tor.streamisolation=false must be turned off when using hybrid-mode

Hybrid-mode was brought to life in LND by Lightning Labs in version lnd-0.14.0-beta. A new option was introduced to split connectivity and to separately address Tor-only peers via Tor and clearnet peers via clearnet:
[tor]

; Allow the node to connect to non-onion services directly via clearnet. This
; allows the node operator to use direct connections to peers not running behind
; Tor, thus allowing lower latency and better connection stability.
; WARNING: This option will reveal the source IP address of the node, and should
; be used only if privacy is not a concern.

tor.skip-proxy-for-clearnet-targets=true


Configuring hybrid-mode

Advertising clearnet connectivity LND needs to know the external IP of a node. There are two different cases to investigate: static and dynamic IP connections.

A static IP is rather easy to set in LND. The external IP address has to be applied to LND's option externalip. That's almost it. But most internet providers change IPs on a regular basis or at least on reconnection. Therefore externalip in lnd.conf would have to be changed accordingly each time a new IP was assigned, followed by a restart of lnd.service to reload lnd.conf. This is unsustainable for continuous node running. Two possible solutions to prevent re-editing and restarting LND:

  • Solution 1: NAT/UPnP
  • Solution 2: Dynamic DNS (DDNS)

Static IP
Static IPs are rarely provided for home use internet connections. It is a feature mostly offered to cable or business connections. Having a static IP makes configuring of lnd.conf much easier. In this case option externalip needs a closer look.
; Adding an external IP will advertise your node to the network. This signals
; that your node is available to accept incoming channels. If you don't wish to
; advertise your node, this value doesn't need to be set. Unless specified
; (with host:port notation), the default port (9735) will be added to the
; address.
; externalip=


Dynamic IP: Solution 1 - NAT/UPnP:
Dealing with dynamic IPs tends to be a bit more complex. LND provides an integrated approach to this: NAT. NAT tries to resolve dynamic IPs utilising built-in techniques in order to fetch a node's external IP address. Notable that LND doesn't handle the setting of externalip and nat at the same time well. Choose only one of them, based on your router's UPnP capabilities (nat traversal).
; Instead of explicitly stating your external IP address, you can also enable
; UPnP or NAT-PMP support on the daemon. Both techniques will be tried and
; require proper hardware support. In order to detect this hardware support,
; `lnd` uses a dependency that retrieves the router's gateway address by using
; different built-in binaries in each platform. Therefore, it is possible that
; we are unable to detect the hardware and `lnd` will exit with an error
; indicating this. This option will automatically retrieve your external IP
; address, even after it has changed in the case of dynamic IPs, and advertise
; it to the network using the ports the daemon is listening on. This does not
; support devices behind multiple NATs.
; nat=true


Dynamic IP: Solution 2 - Dynamic DNS (DDNS):
Dynamic DNS (DDNS) is a method of automatically updating a name server in the Domain Name System (DNS), often in real time, with the active DDNS configuration of its configured hostnames, addresses or other information. (src)
A script or an app regularly fetches the client's current IP address which is saved for later requests. LND is able to resolve a given domain / DDNS to the actual IP address as well. Log output of HostAnnouncer listed below:
[DBG] NANN: HostAnnouncer checking for any IP changes...
[DBG] NANN: No IP changes detected for hosts: [ln.example.com]
...
[DBG] NANN: HostAnnouncer checking for any IP changes...
[DBG] NANN: IP change detected! ln.example.com:9735: 111.11.11.11:9735 -> 222.22.22.22:9735


In this case lnd.conf needs to know a reserved DNS domain instead of an external IP. Option externalhosts has to be set:
[Application Options]
# specify DDNS domain (port optional)
externalhosts=ln.example.com:9735

Lightning explorers like 1ml.com and amboss.space show and use IP addresses only. The node itself also only makes use of the resolved IP addresses (see lncli getinfo). Domains can be some fancy giveaway for peering invitations on chat groups or printed on business cards ... who knows what it might be good for in the future.


Wrap-Up

Summing up the introduced LND options in this article, here are some examples of complete configurations:

Static IP:
[Application Options]
# specify an external IP address e.g. 222.22.22.22:9735
externalip=222.22.22.22:9735
# specify an interface (IPv4/IPv6) and port (default 9735) to listen on
listen=0.0.0.0:9735 # listen on IPv4 interface or listen=[::1]:9735 for IPv6 interface

[tor]
tor.active=true
tor.v3=true
# deactivate streamisolation for hybrid-mode
tor.streamisolation=false
# activate split connectivity
tor.skip-proxy-for-clearnet-targets=true


Dynamic IP - NAT:
[Application Options]
# specify an interface (IPv4/IPv6) and port (default 9735) to listen on
listen=0.0.0.0:9735 # listen on IPv4 interface or listen=[::1]:9735 for IPv6 interface
nat=true

[tor]
tor.active=true
tor.v3=true
# deactivate streamisolation for hybrid-mode
tor.streamisolation=false
# activate split connectivity
tor.skip-proxy-for-clearnet-targets=true


Dynamic IP - DDNS:
[Application Options]
# specify an interface (IPv4/IPv6) and port (default 9735) to listen on
listen=0.0.0.0:9735 # listen on IPv4 interface or listen=[::1]:9735 for IPv6 interface
externalhosts=ln.example.com:9735

[tor]
tor.active=true
tor.v3=true
# deactivate streamisolation for hybrid-mode
tor.streamisolation=false
# activate split connectivity
tor.skip-proxy-for-clearnet-targets=true


After restarting LND, it is now offering two addresses (URIs). These can be verified by typing lncli getinfo:
"uris": [
        "<pubkey>@<onion-address>.onion:9735",
        "<pubkey>@222.22.22.22:9735"
    ],


Special Case: VPN Setup

Clearnet through VPN
To prevent exposure of a node's real IP address connecting through VPN is an approach if anonymity is crucial. To achieve this, some preconditions must be checked and met:

  • ✅ VPN server or provider is able to forward ports.
  • ✅ VPN setup is able to split-tunnel processes (killswitch).
  • ✅ Home setup is able to forward specific ports (router/modem).
  • ✅ Home setup is able to allow incoming traffic (firewall).
Check? Let's go!

0. Declarations
internal_port = Internal LND listening port (for easy setup: internal_port = port-forwarded-VPN_port, but does not necessarily have to be)
port_forwarded_VPN_port = VPN assigned forwarding port
static_VPN_IP = IP of your VPN service/provider
ddns_domain = DDNS (DNS domain) for IP resolution


1. Firewall: allowing incoming port
sudo ufw allow <internal_port> comment 'lnd-vpn-port'
sudo ufw reload


2. Router/Modem: forwarding / mapping internal port to VPN assigned port (check first if this step is necessary)
This step is managed very individually due to high amount of routers and modems out there. Usually GUI-based webinterfaces let define ports to be forwarded for specific devices within a local network.

3. LND: configuring lnd.conf for VPN setup

If VPN provides a static IP:
...
[Application Options]
externalip=<static_VPN_IP>[:<port_forwarded_VPN_port>]
listen=0.0.0.0:<internal_port> // listen on IPv4 interface
#listen=[::1]:<internal_port> // listen on IPv6 interface, if used

[tor]
tor.streamisolation=false
tor.skip-proxy-for-clearnet-targets=true
...


If VPN provides dynamic IPs and a DDNS was claimed:
...
[Application Options]
externalhosts=<ddns_domain>[:<port_forwarded_VPN_port>]
listen=0.0.0.0:<internal_port> // listen on IPv4 interface
#listen=[::1]:<internal_port> // listen on IPv6 interface, if used

[tor]
tor.streamisolation=false
tor.skip-proxy-for-clearnet-targets=true
...

Note: Internal port and assigned VPN port are not necessarily the same. A router/modem may be configured to map any internal to any external port.

4. VPN: Configure VPN connection and check port reachability
Set up a VPN connection with whatever your VPN provider recommends (individual step). Check if the opened port is reachable from the outside by running nc (on Linux) and ping from the internet e.g. with dnstools.ch.
1. run: nc -l -p 9999 (9999 is port_forwarded_VPN_port)
2. ping port 9999 from the internet


5. Killswitch (depends on VPN client): Exclude Tor process from VPN traffic by VPN client or UFW
Most VPNs route all traffic through their network to protect against data leakage. In this case Tor traffic should be excluded from the VPN network because it is anonymized per se plus we want to add redundancy of connectivity and make use of lower clearnet responding times for faster htlc processing. Killswitch can be applied using UFW as well. To do so, please follow this guide.If your VPN client supports command line input, excluding the Tor process could be handled like this:
pgrep -x tor // returns pid of tor process
<vpn cli split-tunnel command> pid add $(pgrep -x tor) // optional step: if VPN provider supports CLI this step can be automated in a script, e.g. after Tor or node restart


6. Restart LND and watch logs for errors (adjust to your setup)
tail -f ~/.lnd/logs/bitcoin/mainnet/lnd.log


7. Lookup node addresses:
If everything is set, two URI addresses will be displayed.
$ lncli getinfo

"uris": [
        "<pubkey>@<onion-address>.onion:9735",
        "<pubkey>@222.22.22.22:9999"
    ],


Alternatively check listening ports with netstat:
netstat -tulpen | grep lnd


Result:
tcp6       0      0 :::9999                :::*                    LISTEN      1000       11111111   1111111/lnd


8. Check connectivity with clearnet peers
To test clearnet connectivity find and ask other clearnet peers to connect to your node, e.g.: lncli connect <pubkey>@222.22.22.22:9999 Successful connection:
lncli connect <pubkey>@222.22.22.22:9999
{

}


Please leave a comment on your setup experience or feedback what can be improved here. We appreciate to hear from you.

Written by osito, Co-Authored & Reviewed by Hakuna.

If this guide was of help and you want to share some ♥ and contribution, please feel free to send a ⚡ tip to our ⚡ addresses: 0x382f9cf667447bb8@ln.tips (osito)  or send some sats, which will be shared between us both via LNURL
LNBits is fun
or via keysend (e.g. 500sats) to
lncli sendpayment --amt=500 --keysend --dest=025a429655f37844f1247fac32fc2cade8af60ae276b300280ffb796917f812e37
lncli sendpayment --amt=500 --keysend --dest=037f66e84e38fc2787d578599dfe1fcb7b71f9de4fb1e453c5ab85c05f5ce8c2e3
Lightning Network Node
HODLmeTight
Capacity: 1,238,825,200 SAT
Channels: 139