tired: using CDNs for your fediverse instance

wired: turning the fediverse into a global CDN itself

with @cwebber's datashards proposal, it's possible. we plan to implement datashards in Pleroma 2.0 to accomplish this.
Follow

@kaniini @cwebber is this the "sharing bandwidth rather than rehosting everything" solution or something?

@abby @cwebber

datashards in combination with the Kademlia-based DHT work we are already doing for Pleroma 2.0 will allow for a distributed CDN. this is a combination of bandwidth sharing, and also selective mirroring of encrypted chunks.

the DHT provides the ability to know what nodes have what data shards, the data shards themselves provide a cryptographically verifiable source of truth

@kaniini Is there a particular reason you choose Kademlia as a DHT?
Because due to its probabilistic properties I couldn't use it for my hashtag federation work and went with Chord instead.
git.orlives.de/schmittlauch/pa

Would be a shame if instances have to implement different DHTs if not absolutely necessary.

@schmittlauch we haven't gone with kademlia yet, but kademlia looked attractive in the beginning. Chord works for us, too, but i'm not personally sold on hashtag federation yet -- federating all hashtags in a store-and-forward environment will max out a lot of people's disk space ;)
@kaniini @schmittlauch if you do plan to implement anything that grows storage usage a lot, please 1) have a flag to turn it off, and 2) make a big announcement about it so I don't forget to turn it off before updating :p

@kaniini my approach will definitively increase storage requirements, but hopefully not too badly as storage nodes just have to store the post IDs of some of the hashtags.

@schmittlauch the problem is that no software is equipped to work in that way. we would have to significantly overhaul Pleroma IR to store things that way.

@kaniini What's problematic about the required storage structure in that regard?

@schmittlauch nothing is problematic with it in particular. i'm just saying that Pleroma IR can't handle that yet :)

it's "simple" in theory, but in practice, it will take some work to do.
@schmittlauch @kaniini we didn't yet decide on anything, this will all still be properly planned, tested, evaluated.
@lain @schmittlauch yeah, we are still in the proof of concept stage, it's just that datashards solves a significant chunk of the problem already :)

@kaniini @abby @cwebber
May I suggest talking with the bittorrent people before implementing a DHT ?

My opinion of DHTs at this point is they basically only work as long as there are no bad actors.

@cjd @kaniini @abby Could you elaborate on why? I have my own suspicion of your answer: for content-addressed content, I think they're probably often fine, but if it's a "varying bucket" full of content, it's easier to intentionally drop content from it.

BTW, for the mutable datashards component (which I think Pleroma probably wants for moveable actor profiles) my suspicion is that a gossip network may be more useful than a DHT for informing about updates. Not sure about immutable chunks tho

@cwebber @kaniini @abby
I wrote a modified version of kademlia (it had an additional constraint) and I also ended up knowing a few of the bittorrent people. My opinion of bittorrent DHT is it's mostly a mouse that roars.
When you put data in the DHT, it gets pulled back to a real tracker by DHT scrapers and the same DHT scrapers are in there answering your requests. BUT it's Just Decentralized Enough to be opaque to the lawyers. @astro what's your opinion?

@cjd @kaniini @abby Also I think @lain has written a bittorrent DHT at some point, probably has some opinions

Sign in to participate in the conversation
LGBTQ.cool

The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!