Yes, I’m certain I could final answers to all these questions via research, but I’m coming here as part of the Reddit diaspora. My guess is that there’s a benefit to others like me to have this discussion.
I can vaguely understand the federation concept, the idea that my account is hosted at an individual Lemmy server and that other servers trust that one to validate my account. What’s the network flow like? I’m posting this to the lemmy.ml /asklemmy community, but I’m composing it on the sh.itjust.works interface. I’m assuming sh.itjust.works hands this over to lemmy.ml. How does my browsing work? Is all of my traffic routed through sh.itjust.works?
Assuming there’s a mass influx of redditors, what does it look like as things fail? I’m assuming some servers can keep up under the load and some can’t. If sh.itjust.works goes down under the load, can I still browse other servers? Or, do those servers think I should have some token from sh.itjust.works, because my cookies say I’m still logged in, and I can’t even do that?
Are there easy mechanisms to allow me to grab my post history?
I’m assuming most (all?) Lemmy servers are hosted in home labs? The idea of Lemmy excites me, but the growth pain that could be coming scares me. Anybody using a CDN in front of their servers? That could be good, but with unconstrained growth, that could be costly, which is very bad.
I can imagine lots of different worse case scenarios, but I’m curious what those of you who run servers imagine for the best case scenario? A manageable growth that just gets more vibrant communities, which can’t ever lead to the breadth and variety of Reddit?
Also, for those running servers, have any of you experienced issues during this growth? What scares you?
You must log in or register to comment.
What’s the network flow like? I’m posting this to the lemmy.ml /asklemmy community, but I’m composing it on the sh.itjust.works interface. I’m assuming sh.itjust.works hands this over to lemmy.ml. How does my browsing work? Is all of my traffic routed through sh.itjust.works?
- You register your account on
sh.itjust.works, that’s where all the info you care about resides. Your list of subscribed communities resides there. When you read a post, it gets fetched out of the db onsh.itjust.works(irrespective of where the home instance for that post’s community is… when you read it it comes out of the database on your home instance), and when you comment on a post, that gets written to the db on your home instance. Your home instance a standalone fully functioning thing. - When you subscribe to a remote community like this one, you tell your home instance "keep up to date with posts and comments for this community and let me know about them. Your home instance asynchronously gets all those updates while you’re asleep or whatever so it can show them to you out of its local database when you come back. If more users on
sh.itjust.workssubscribe to the same community… there’s no incremental overhead. All ya’lls instance is ALREADY subscribed to that sub. So other users on your instance can sub to it for free, it’s already in the instance’s database.
Assuming there’s a mass influx of redditors, what does it look like as things fail?
- If
lemmy.ml(where this community is homed) falls over from being overloaded or just is broken for whatever reason, your instance is unaffected. You can still read posts and make comments. This community however… is affected. New posts and comments for this community might come through intermitently or not at all for you (and everyone in the lemmyverse) because the community’s home server isn’t working well enough to reliably deliver them over federated replication. You can still read older posts and comments that have already been synced to your home instance, but new ones might not arrive. You might also see weird stuff like being able to see new comments from othersh.itjust.worksusers on this community, since those get written to your db before getting federated back to the community’s home server. But mostly updates from other instances stop or get unreliable. - If
sh.itjust.worksfalls over for some reason… well… that sucks for you. You can’t log in or browse anything on it. You can still visit this sub at https://lemmy.ml/c/asklemmy/ as long aslemmy.mlis working and you’ll be able to see the posts and comments that other accounts make. But you’ll be an anonymous read-only browser, you won’t be able to post or comment untilsh.itjust.workscomes back online (or you make a new account elsewhere and lose all your comment history and subscription list).
Are there easy mechanisms to allow me to grab my post history?
There’s a github issue for this, but it’s not done yet: https://github.com/LemmyNet/lemmy/issues/506.
I’m assuming most (all?) Lemmy servers are hosted in home labs?
I don’t think that’s a good assumption.
lemmy.mlis hosted on OVH, a cloud provider. My home instance onlemmy.worldis hosted by admins that run something like a 32 CPU mastodon instance. Most instances with over 100 users are running on some kind of probably modest but “real” cloud instance. The admins are volunteers, but often smart technical folks paying for small but real compute infrastructure.The idea of Lemmy excites me, but the growth pain that could be coming scares me. Anybody using a CDN in front of their servers? That could be good, but with unconstrained growth, that could be costly, which is very bad.
Anticipating growing pains isn’t wrong, it’s probably gonna happen. But the devs are gonna find and work on the biggest performance problems so that people can viably run bigger instances, and instance admins are gonna run bigger hardware and ask for donations or run patreons to cover the cost. In my opinion, the bigger worry is that Lemmy will fizzle… not that it will spectacularly explode. As long as people join and contribute and are interested, we’ll find a way to improve scalability and performance. The death knell would be if people get bored and leave, but compute capacity won’t be the problem in that scenario.
- You register your account on
the idea that my account is hosted at an individual Lemmy server and that other servers trust that one to validate my account
I can’t stress highly enough how much this isn’t how it works.
You basically never directly interact with other servers. Instead, when someone on your host site first subscribes to a community hosted on a other site, your instance pulls in some recent posts from that remote site and then requests that all future content from that group be forwarded along to it. Then, people on your local site interact with that mirrored content, and your local site sends local additions back to the original host for syncing.
Your account only exists locally. You’re always reading locally, and you’re always acting locally. Everything else is servers mirroring and forwarding content.
Thanks. Based on some of the other answers, particularly in https://sh.itjust.works/comment/12511, I know understand better.
I appreciate everyone helping to explain some pretty basic questions in such detail.
We’re all learning. And I want to stress – and maybe should have in my original reply – that I wasn’t really trying to criticize. My reply was just going to be so far down, I figured it’d only be noticed by a handful of people.
The way you intuited things is the way a lot of people have intuited things on the Fediverse. There were big blow-ups on Mastodon about this 6 months ago, as users spoke out about instance admins defederating from other Mastodon instances who hosted accounts that chronically broke local rules.
It became clear that people interpreted these defederations as denying new local users’ access to “parts of Mastodon”, as if there were was some central object called “Mastodon” and local admins were being nannies about letting some of their users wander over to some parts of it, when in reality it was a case of the local admins refusing to host content from other websites that gave shelter to individuals who were abusive or otherwise breaking rules that existed on the local site.
If that makes sense.
If you can grok that we’re basically all on different, independent web forums, and there’s just an implicit agreement between the forums to cross-post and share content, you can better grok why somethings that will happen here happen.
If you can grok that we’re basically all on different, independent web forums, and there’s just an implicit agreement between the forums to cross-post and share content, you can better grok why somethings that will happen here happen.
I’m old enough to have participated in Usenet before web servers existed, with the idea that different Usenet servers carried different feeds. Now that I better understand it, that model is closer than my original understanding. I also realize it’s not a completely accurate model, since there’s no central hierarchy to the fediverse like Usenet had, but at least it works to get me to understand the idea that all interactions are going through the server I’m pointed at and that posts originating from other servers across the fediverse are being replicated to my server so I can interact.
Noice. It’s really nice seeing the Reddit migration bring people to the fediverse who experienced the Internet prior to, like, 2012, rather than shunned it, like the Twitter migration brought.
The type of user that is alienated enough by the API changes and capable of actually creating a Lemmy account surely has significant overlap with internet veterans and generally intelligent people. Which is fantastic news, I remember trying voat.co and it was just overrun by nazis and complete idiots. So far on Lemmy my experience has been the opposite. I’m not old enough to be a Usenet veteran, but I did experience the age of the BBS. I remember spending a lot of time on Totse back in the day.
Also coming from the sh.itjust.works instance.
My question is how the broader/larger communities are going to be handled. Which server has the responsibility of hosting large general purpose discussion communities? Because it seems like servers are actively incentivized to limit their size and you end up with many smaller mirrored communities but no realistic path to a reddit style overarching open forum about movies, or soccer, for example.
