It is with great sadness that I announce that my two R&D Bitcoin Lightning Network Nodes CHILLYCALENDAR and CHILLYCALENDAR2 have been hacked and I am taking them offline until further notice.
This morning I woke up to several emails from node operators from this website asking me why I had closed channels to their nodes. I had not closed any channels. I attempted to login to this lightningnetwork.plus website and my password didn't work. This was the first sign that something was seriously wrong. So I requested a new password and I was then able to login and start responding to node operators who had messaged me.
What do I suspect happened?
I suspect that I stupidly used the same password on my main web host server and my two bitcoin nodes as the password I used on this website. The hacker I suspect may have somehow managed to gain my password from this website and then used that password to login to my primary web host where they entered these commands without cleaning up after themselves:
image.png109 KB The hacker was looking for bitcoin wallets on my primary web host. They also looked through KYC Data (I have no real world active users KYC data so they got nothing there. My systems are primarily for R&D and proof of concept purposes).
I ran the last command to see if I could find where the hacker came from:
image.png5.79 KBimage.png70.7 KB
The hackers IP address is: 185.213.155.215
LN+ Alan - you might want to check your servers and make sure this IP address has not infiltrated your systems and to put a block on it anyway as a precaution.
Fellow node operators - I suggest changing your password you use to access the LN+ website as a precaution and do not use the same password on this website or others that you use on your bitcoin nodes. I was not cautious enough and I believe I used the same password on this website as my primary nodes. I'm kicking myself for my stupidity. If anyone else wants to kick/slap me do feel free I deserve it. In my defence - I have been very distracted with my health which has been very poor due to two torn meniscus in both knees which I have been doing a lot of rehab on thus I have been away a lot from my servers and highly distracted learning to walk again in the local swimming pool.
After finishing looking around on my primary web host the hacker gained access to my primary node CHILLYCALENDAR. From there they somehow managed to gain access to my BTCPay Server instance without generating any history in the shell. They turned off 2FA and then they closed all my channels I had open to other nodes. They did the same thing on my secondary node CHILLYCALENDAR2 and they also changed my password on this server of my BTCPay Server instance. After closing out all my channels they then proceeded to steal approx $10,000 AUD of Bitcoin funds I had on both the servers. These funds were made up of on chain funds and funds in channels I had initiated to other nodes via my RTL instance on my BTCPay Server. Luckily for me I had recently removed a lot of funds from my nodes in order to help my family so my losses were limited and it could have been a lot worse.
Whilst the attack itself was on the whole not sophisticated there were parts of it that were. Managing to close off channels and take funds without leaving much of a trace via my BTCPay Server log files for example. It would seem that the hacker was only interested in stealing Bitcoin. Whilst the hacker did gain access to some of my other nodes unrelated to Bitcoin they didn't do anything on those nodes other than take a look around.
I have gone through and changed all my passwords on all of my hosts and my personal passwords for everything else as well.
Obviously I blame myself for my poor security practices. I should have and I did know better.
Thank you to my fellow node operators who alerted me this morning to issues surrounding closed channels. It's unlikely I will be able to respond to all of you personally but thank you for messaging me and I'm very sorry for what has happened. Feel free to message me if you have any information related to this hack that may help other node operators or just to slap me around a bit for my stupidity.
Logs on my primary web host indicate the hacker has been through my email as well unfortunately. I have changed this password as well.
Both CHILLYCALENDAR and CHILLYCALENDAR2 were R&D platforms. The purpose of which was to run bitcoin lightning network nodes and see how far I could get as a sole operator. CHILLYCALENDAR was the 2nd highest ranked node in all of Australia and had processed / forwarded over $1 million worth of lightning transactions and i'm very proud to have achieved this working completely on my own. I have been running lightning nodes since 2021 and I almost got to the 4 year mark of running these servers in the wild without any incidents. This work has enabled me to create a lot of documentation and obviously I have learned a great deal.
Will I be back? I'm not sure yet. I'm still processing everything that has happened. But rest assured that the output of everything I have done, learned etc will go into something in the future if not as a sole operator then likely as part of a dedicated team working for a larger corporation that has the resources and people required to enable me not to have to do literally everything on my own and it has helped the bitcoin lightning network to mature having my nodes online since 2021.
Kind Regards,
Clarke
Clarke Towson, BCMS (Bachelor of Computer & Mathematical Science)
CEO
INTJ Billing
m: +61 432 359 166
a: 7 Cullen Court Spotswood Victoria 3015 AUSTRALIA
Chillycalendar and Chillycalendar 2 are back online after the successful completion of electrical works.
The upgrade of the electrical systems at INTJ Billing's premises is now complete and all services have been successfully restored. INTJ Billing now has 2 x Tesla Powerwall 2's and an upgraded electrical breaker in the mains switch box which will hopefully resolve the issue of unscheduled outages whenever there is an issue with Jemena our electrical distributor.
We have also updated the firmware on our main server's Ecoflow Battery which may be placed back into service in the future after additional testing (depending on if there are any additional unscheduled outages due to Jemena and the issues that then occur on our mains switchboard) and we have also purchased and installed a new Netgear core switch to replace a failed D-link switch.
Outage start time: 9:34 am
Services restored time: 1:32 pm
Outage time: Approx 4 hours.
Kind Regards,
Clarke
Clarke Towson, BCMS (Bachelor of Computer & Mathematical Science)
CEO
INTJ Billing
m: +61 432 359 166
a: 7 Cullen Court Spotswood Victoria 3015 AUSTRALIA
Please be advised that there will be some interruption to my nodes Chillycalendar and Chillycalendar2 on Thursday 7th November 2024. At 7:30 am TCK Solar will be on site to install a 2nd Tesla Powerwall 2 which will be used to provide additional battery power and backup to my Bitcoin Lightning Network Nodes. This work will involve the mains power being switched off to the premises and work on the specific phase that provides power to INTJ Billing's servers.
This electrical work will help ensure fewer unscheduled node outages due to power issues caused over the past few years by Jemena our electrical distributor.
Kind Regards,
Clarke
Clarke Towson, BCMS (Bachelor of Computer & Mathematical Science)
CEO
INTJ Billing
m: +61 432 359 166
a: 7 Cullen Court Spotswood Victoria 3015 AUSTRALIA
I have managed to get my bitcoin lightning servers back up and operational again. I have rerouted power and they're getting up to date with the latest blocks as I type this. This was a really close call but they're back up! UPS and batteries are still however not fully operational.
There may be some instability over the next 5 days as power works are scheduled for the 28th August to try and permanently resolve the power problems. There will be some more downtime on the 28th but shouldn't be more than 2 hours.
Today there was a storm in Melbourne which managed to cause a power surge at my premises which knocked out Chillycalendar and Chillycalendar 2. Despite UPS, Battery, Voltage line regulator and additional power surge protection it wasn't enough. The server is stuck on the boot screen and I'm unable to get it to boot.
I'm not sure exactly what i'm going to do at the moment. A new motherboard may be a quick fix and get me back up and running again. The surge looks like it damaged the motherboard. The power supply is OK but the machine just wont boot no matter what I do.
I will update you all in a few days after considering what actions i'm going to take.
Kind Regards,
Clarke
Clarke Towson, BCMS (Bachelor of Computer & Mathematical Science)
CEO
INTJ Billing
m: +61 432 359 166
a: 7 Cullen Court Spotswood Victoria 3015 AUSTRALIA
I wanted to let you all know about some work I have been doing recently with Grafana which may be of interest to you as fellow node runners.
Using Grafana and the forwarding logs from my RTL instance I have created a public dashboard of my primary Bitcoin Lightning Network node CHILLYCALENDAR (which is currently the 3rd highest ranked node in terms of capacity in Australia and the highest ranked in Melbourne thanks to you, my amazing fellow node runners and this fantastic website). My dashboard contains a connect now button which links to a lightning page on my website where other node runners can easily connect to my node and open channels. I might add an additional button that links to this website as well.
As most of you will know, creating dashboards like this for Bitcoin and the Bitcoin Lightning Network at the moment is quite difficult. It will get easier as more developers build tools which automate a lot of the programming, configuration and integration work I had to do to get all this working.
I used Grafana with a MariaDB. I exported the forwarding data from RTL, populated the database, connected it up to Grafana as a data source and then created the dashboards. I utilised ChatGPT as well for some of the queries. At the moment it's a manual process to export the forwarding logs from RTL and import them into MariaDB but I have reached out to the developers of RTL to find out the MySQL queries they use on the underlying lightning sqlite database to form the view and CSV file from RTL. I did look at the underlying schema of the sqlite db but I thought it would be better to reach out to the RTL devs directly and find out exactly how they do it. My Grafana instance has been set up on another VPS and not on CHILLYCALENDAR. I created a new A record on my webhost for grafana.intjbilling.com and pointed it to my VPS where I have Grafana running. That VPS is running Apache with a number of virtual hosts one of which is Grafana.
As we know Bitcoin is a pseudo anonymous open public ledger. Public dashboards like these I think can give regulators more confidence in those of you who like myself are running lightning network digital exchanges.
If any of you as my fellow node runners would like more details about how I did this and would like to set up a similar dashboard for your nodes or would like to collaborate with me on improving and refining my dashboard so we can come up with an even better one by working together on it please do feel free to reach out to me and I'll provide you with all the code and details.
I have also been hard at work recently populating links at the bottom section of my website https://intjbilling.com/index.html some of those links will be of interest to you as fellow node runners as they make it easier for us to run our nodes keeping access to all the important websites in the one place for quick access.
Almost everything I'm doing with lightning is open source and free for you all. Just reach out to me if you see anything I have done that you like and I'll provide it to you.
With Very Best Regards,
Clarke
Clarke Towson, BCMS (Bachelor of Computer & Mathematical Science)
CEO
INTJ Billing
m: +61 432 359 166
a: 7 Cullen Court Spotswood Victoria 3015 AUSTRALIA
INTJ Billing's primary and secondary lightning network nodes were down for maintenance this morning for approx 24 minutes. This was to implement a new more reliable power configuration. An APC Line-R voltage regulator and a new CyberPower UPS were added to the setup for even more reliability and server uptime. The setup also has an Ecoflow Delta battery connected to it so in the event of a power outage the UPS will kick in and then if the power doesn't come back on in quick fashion the Ecoflow battery will then kick in and provide another 3 hours of uptime which by then the power would hopefully be restored.
The reasoning for this new setup is due to a number of recent brownouts by my power distributor which were causing my main UPS to spike and go into an overload condition. Less load has also been placed on this UPS.
I hope it's OK for me to announce this here. I have created a new community over on Amboss Space for Bitcoin Lightning Network node operators that would like to help support the payments of businesses involved in providing sex related products and services or adult entertainment including sex work, strip clubs, host and hostess clubs, online adult entertainment and so on.
I want to create a network of lightning network nodes that are committed to achieving censorship free payments for the sex industry worldwide to permanently solve the issue of debanking and payments censorship for the industry and to use this as a test case for other industries that have similar issues with payments censorship and banks.
The overarching goal of the community is to create a network of sex industry and sex industry friendly lightning nodes connected to each other with payment channels open between nodes so the sex industry globally can process payments 100% free from censorship. Payments that cannot be blocked, censored or frozen.
My ultimate goal: A Bitcoin Lightning Network node located (or providing services to) every major red light district in the world.
I maintain a website where I have created an interactive map of lightning nodes that have joined me in this quest:
My node CHILLYCALENDAR was down for the past 22 minutes approx. This was caused by a brownout and a power spike in my area Spotswood Victoria which caused one of the connected UPS to cut out and alarm. The server and associated infrastructure needed to be restarted. This is the first time such an incident has occurred on my infrastructure where a UPS has spiked and alarmed.
The infrastructure has been restarted and my node is back online.
Just a quick note to let you know that my node is currently down. It's due to an issue with my BTCPay Server NBXplorer is stuck syncing meaning my node is stuck on 99%.
I believe this is what's happening: https://github.com/btcpayserver/btcpayserver-docker/issues/519
Hopefully I should have it fixed in the next day or so. I may take the downtime opportunity to upgrade my BTCPay Server to the latest 1.7.5 version as well.
My node Chillycalendar has been uncontactable for around 4 hours due to a fault with my web host hostpapa which handles the DNS for my BTCPay Server. Sorry about that to the peers connected with me that I have open channels with that I found via this website. Hopefully the issue should be resolved shortly https://hostpapastatus.com/
2022-04-17 16:00 EST
Our server health team is still working on a permanent fix for the networking issue.
We will post further updates on this page until the issue is fully resolved.
2022-04-17 14:30 EST
Some servers (sXXX, ws01-ws09, c100/c101, and VPS) are experiencing some connectivity issues that may affect the ability of clients to access services/accounts on these servers.
This has been identified by our server health team, and they are working to resolve the issue as soon as possible.
Thank you for your patience and cooperation in regard to troubleshooting this issue. We will post further updates on this page until the issue is resolved.
