Leaving the Goldilocks zone
Some time ago, Varun from Farcaster released an article titled The Goldilocks consensus problem where he outlined how Farcaster’s deltagraph algorithm was engineered “just right” for the problem of building a decentralized social network on top as federated consensus is too weak and since blockchain consensus is too strong. I think the post he published is correct in that it doesn’t make any mistakes in argumentation or through its claims. Still, in the following, having implemented that same algorithm in Kiwi News and running it for a year now, I believe we have left the “Goldilocks zone.” I will elaborate.
A quick disclaimer, though: I didn’t care to properly explain my thinking from the baseline of many’s perceptions. You’ll have to deal with the fact that I’m pretty deep down a rabbit hole. But feel free to ask questions!
The Farcaster monopoly of Warpcast
Warpcast’s code base being closed source gives the Merkle Manufactory team a monopoly over the Farcaster protocol. Considering that 99% of all fresh casts are fed from Warpcast into the MM hub, that hub is essentially just a really complicated REST API without rate limiting. Twitter and Facebook also had these types of APIs before they shut them down and in back and forths with Dan, the last resort argument is that MM has no motive to shut down the dissemination of messages through the hub at any time. This, of course, is an argument about trusting the MM leadership, not about trusting to run code. It also means that Warpcast has an effective monopoly over which Farcaster protocol variant will be prevalent on the network. Twitter works by giving you the latest in news and gossip, which means the timeliness of tweets is much more important than preserving their history. Unfortunately, though, this gives Warpcast an advantage over all other hub operators. When Warpcast upgrades its hub’s Farcaster protocol, all other hubs are forced to upgrade as they otherwise forgo the content liquidity of the majority client. Varun’s post concludes that Farcaster’s deltagraph is in the Goldilocks zone of algorithms for a decentralized social network. Yet, I think it would be far better if the protocol produced more credible neutrality and decentralization. Only 51% of miners can upgrade the protocol in Proof of Work. Miners don’t gain monopoly power from aggregating user-write demand. Ethereum’s Proof of Stake even has an implicit incentive built-in to further consensus client diversity.
With Kiwi News, we’ve tried to address the problem in a simple but also dissatisfying way. We’re constantly open-sourcing all code, both the p2p algorithm and product code. This comes with its own challenges, namely that probably anyone could fork us at any time and launch a scam coin on top of the code base. Yet, at least for us, if we upgraded the protocol in a bad way someone could actually easily compete and pressure us through forking. This cannot be done with Warpcast as it’s an accumulation of three years of highly paid top talent engineers working on it, polishing it to a platonic ideal. Good luck competing with them on product!
The storage rent solution produces a shit outcome
One of Ethereum's coolest properties is that the retrievability of its storage is actually a public good that is continuously paid for through writes. Someone could have launched an NFT in 2018, and this data is still accessible on the mainnet for free. All that was necessary was for the NFT launcher to pay the write. That’s an amazing property!
Kiwi News and Farcaster are basically data unions of users labeling data and making it freely available for processing for anyone. Elon and others are very concerned about X et al. being scraped by the data-hungry LLM data centers. Farcaster and Kiwi News have turned this model on its head by giving everyone full access to the basic data model. We’ve said: “For the sake of neutrality, we’re giving away control and ownership over our data such that anyone can build a competing client.” Kiwi News is now a repository of 20 thousand links, all rated by a small group of Ethereum addresses. Farcaster is a data set of 60 thousand people writing casts constantly. While much of this relevancy is currently still downplayed by many, these protocols have become exceptionally valuable data sources for those wanting to build artificial intelligence or mine user data. Through their future growth, this value will grow so much bigger. Yet, not only is this data given away for free, but Farcaster’s storage rent also asks the user to pay to preserve their own data. This produces a bad outcome. If you search for old casts on Farcaster, you will find that all the likes of these casts are gone. You might also find that many of the casts you’re looking for aren’t findable anymore. This is because Farcaster not only deletes old data after some time, it’s also because not everyone decides to pay to preserve their data.
We haven’t copied the storage rent model for Kiwi News because we don’t want to go down this route of giving users an individual choice on whether they want their data to stick around. In our case, this is a bad choice the Farcaster team has made. In fact, we think that rating links by their relevancy and then making this data available, signed, timestamped, and verifiable is becoming an incredible data repository for the crypto world. It would be such a waste if this data were deleted! We strongly prefer to keep data around, and we actually think that the protocol must demand the users to pay collectively for it being a public good that anyone can read from, but only those who pay can write to. Our implementation is far from ideal either in this case, though: There are no hard protocol limits on how many upvotes a Kiwi Pass holder can store on Kiwi News. It’s unclear how we’ll deal with individual users spamming the network etc.
Blockchains deal with congestion by pricing each individual transaction at the time of the write intent. When there is a market crash, the chain can curb demand by increasing write prices, asking everyone to re-consider if they really want to update their position right now or if they can also wait a little. Yet, as I outlined in a talk, the issue with this for a social network is that we don’t want to punish people for upvoting breaking news when a lot is happening in the world. In fact, during those times, users should be encouraged to bring their signal to our network. So clearly, we, Farcaster and Kiwi News, cannot ask users to pay on a per-interaction basis. This wouldn’t make sense.
The ecosystem has moved on
Finally, there is now a question of whether the algorithms that Farcaster and Kiwi News have built actually still serve a problem with DAS networks being online and cheap. Both were built during a time when block space demand was extremely high and when these systems hadn’t been launched yet. Admittedly, building a protocol by oneself is great as custom rules can be added to enforce qualities that we want to pass on to the users. Still, I do feel like there is now the known unknown existential threat of having to understand if that particular implementation of set reconciliation is still actually useful on a business level. Eigenlayer is going to come online, and so are many other solutions promising even further to modularize and network-ify what we’ve painstakingly built using libp2p-gossipsub and other packages. And our outcomes are far from ideal. Most importantly, the algorithms have not caught the interest of the crypto space because there are no answers when they ask us, “Is this censorship resistant?” or “What happens when I post porn to it?”
Vision
Other than that, I think these networks (Farcaster and Kiwi) are actually plagued by a severe lack of imagination. When Bitcoin and Ethereum came of age, the air was ripe with people speculating about these projects’ fantastic potential for the future of humanity. And to be quite frank, there are a bunch of outcomes which have been enabled by them and which are as awesome as we had imagined them. The same cannot be said for decentralized social networks. These visions, in fact, haven’t been properly vocalized yet, which also makes me conclude that the “Goldilocks zone” cannot have possibly been discovered yet.
In fact, I think the rise of our networks, Farcaster and Kiwi News, must not be told without somehow also clarifying their economics and their inevitable commoditization of the basic curation data. For decades now, the best-kept secrets of the big tech companies have been their users' curation and metadata—a free-for-all on-set reconciliation delta graphs. Looking at Kiwi News’s server logs, it’s shocking how much scraping and crawling is constantly going on by the big LLM providers. Despite us even making this data available through a beautiful set reconciliation algorithm, it’s clear that their thirst and greed to harvest all possible user data is inevitable. So how can this be different for X or other social networks, and has it maybe just become a self-deceiving lie to try to stop anyone from crawling a platform’s content? If so, Kiwi News and Farcaster are anyways not doing something fundamentally innovative; we’re merely seeing reality for what it is and coping with it.
Still, I do think that rather than individualizing the burden of paying for storing data on the network, for Kiwi News at least, it makes more sense to collectivize this cost and to come together as a data union of link archivers and curators rather than as individual bloggers looking for webspace. Yet, I then also want to understand how the Kiwi News protocol can further capture the value it generates not only for those actively participating but also for those who are merely reading the site and extracting the result of the curation from it. Curation is the laborious part, and so reading for free feels like leaving money on the table. While I wish there were a way to punish the free riders, what gives me hope is that Ethereum embraced free riding, and it is nevertheless successful.