22 Nov 2024
In Lightning, whenever you send a payment, you must first pick a path over which to send it. If you pick that path well, the payment will complete immediately and for (hopefully) a low fee. If you pick poorly, a node in the path may not have enough funds available for your payment and will have to fail the payment back to you, at which point your node will need to pick a new path and try the payment again, resulting in slow payments. If you pick poorly several times, the payment may time out and ultimately fail, even if there may have been sufficient capacity for your payment over some path.
In practice, most pathfinding algorithms in Lightning lean heavily on memory of past payment or probe attempts in order to select paths which are likely to succeed. This allows a Lightning node to learn which channels in the network are usually saturated (and in which direction) and allow it to avoid common logjams in the network. Pathfinding algorithms without memory of past attempts appear to perform quite poor, especially in paying a handful of hard-to-pay nodes which operate as “sinks” - generally receiving more than they send.
Testing new Lightning pathfinding algorithms is quite easy when you aren’t concerned with memory - you simply write your algorithm and try to pay many nodes on the network. The more nodes you manage to pay, the better your algorithm is. However, when you’re dealing with memory you have to test an algorithm over a long time horizon, testing lots of payments and comparing the success over time. This makes evaluating pathfinding changes difficult.
Today, I’m announcing a simple pathfinding evaluator (and accompanying dataset) which allows researchers and developers to test a pathfinding algorithm by replaying many historical probes from the perspective of my Lightning node, comparing the real results with the path probabilities their pathfinding model predicts.
While the use of paths LDK’s router picked biases the framework somewhat, it should be general enough with enough data to still provide a good indication of whether a model change is positive or negative.
You can find the replayer and can learn more from its README in its git repo.
11 May 2024
These next few years are as existential for bitcoin as the Blocksize Wars.
Back then it was about who got to decide what bitcoin was, now it’s about what bitcoin is.
To most of us, bitcoin was always a tool for freedom - freedom to transact with who you wanted without trusting any third-party who could prevent you from doing so. But this is now a question.
While Bitcoin’s price skyrocketed, developers focused on building tools to ensure this goal was met - building a robust, stable foundation in Bitcoin Core, building out the Lightning network to enable cheaper payments with better privacy than “tell everyone everything you’re doing”, exploring (and implementing) tons of ideas for other scaling from sidechains to statechains to timeout trees to Ark to…
Sadly, all the ideas for making bitcoin (or any cryptocurrency) actually useful for transacting trend towards having some untrusted party in the flow of funds - whether its a rollup or sidechain with one or ten chain operators, mobile lightning with an LSP, a statechain operator, ASP, etc, we just haven’t cracked building cryptocurrency payment rails without an (untrusted!) party being involved.
With regulators cracking down and more and more participants in the bitcoin ecosystem only interested in a 21M coins limit and seeing any form of non-KYC payment rails as hostile to the value of their investment, these centralized parties are increasingly going to be targets, and over the past few months we’ve even seen proactive closures to avoid regulatory scrutiny.
While there may have been a chance at clarifying (US) regulations that an untrusted party in the flow of funds isn’t required to perform the usual slate of KYC nonsense that strangles the traditional finance world, we squandered it expending all of crypto’s political capital pushing for securities reform to ensure token issuance is (maybe) legal rather than trying to ensure people can meaningfully transact without the entire world learning what they’re doing.
Worse, with mining (pools) as centralized as ever and relatively little desire to change from most miners, every layer of bitcoin is centralized and ripe for regulatory capture. While Sv2 or p2pool revival has a chance at improving this, the push for increased expressiveness on bitcoin and the MEVil that may come along with it may all but close the door to decentralized mining.
With where bitcoin is today its hard not to see a bleak vision of the future. Building a system that enables trustless global transacting was always going to be an uphill battle, but I always anticipated most bitcoiners would remain focused on this goal. Instead, today, the bitcoin community is focused on petty squabbles and infighting.
Sadly I don’t have any great solutions here. If bitcoiners want to preserve what we’ve built and fight for it the focus needs to be on drastic improvements to default wallet privacy across the ecosystem, aggressive investment in regulatory change (and not through lobbyists focused on securities regulation for crypto token issuance), and operation of scalability solutions across the world, not just in the US. Sadly, all of these areas are horribly underinvested in, and invested in at only a tiny fraction of the amount of energy that has gone into other areas of bitcoin.