The cause is unclear to me, but the processes that manage incoming and outgoing federation were "stuck" until I manually intervened. It's running now. I'm adding to my to-do list for after my job ends in 10 days to implement some sort of detection and hopefully automated restart for when this happens again.
ok - I took a bit different approach. Since I know what error in rabbitmq's log file is associated with things coming to a stop on fedia.io, I installed swatchdog and set it up to look for that word (which is, btw, "timeout"). I created a script that stops all the messengers, then stops php-fpm, keydb, and rabbitmq. Then it start rabbit, keydb, and php-fpm in order. Finally, it restarts the messengers.
I will be surprised if it works first time, so it may still crash again but I'll be watching
I know it's not ideal, but I fully understand the whole situation. Let's focus on making Mbin better for the existing users who are now experiencing CSRF or log-out problems. Hopefully after that, we can focus on improving anti-spam (since hcaptcha is not preventing any spam accounts for some unknown reason).
Maybe even considering additional an optional question? With only 1 correct answer. Or maybe even enforce 2FA.. I dunno.. But spam is getting out of control. Coincidence due to the rise of LLMs? Who knows. But anti-spam like hCaptcha, even set to "difficult" doesn't seem to cut it anymore...
What works for me on both mastodon and Lemmy is a free text question: why do you want to join?
The user enters whatever they like and it goes into a moderation queue. Both lemmy and mastodon send me an email when a new account is ready to review.
I read the response and choose to whether to approve their account. At the moment, spammers are really bad at answering the “why do you want to join” questions.
The main thing I experience since the CDN update is that voting often ends up in an error page, similar to how adding comments sometimes directs you to a secondary page (luckily with the comment intact). Going back and trying to vote again may or may not work.
I did not have that 3 weeks ago though, and they're not 500s - at least they're not displayed as that. I get pages like this one: https://fedia.io/ecv/7319083/-1
The recent version seems to cause PHP to off into space somewhat frequently. I am working to mitigate it - which will likely, at least for a while, require me to restart the services every few hours until we can figure out what the actual cause is.
Can you describe the steps you're going through to attempt to block it and what you observe? For instance, if I go to https://fedia.io/d/hexbear.net and click block on the side, it instantly appears to work without any errors. I tried from other mbin instances as well and didn't see any errors.
Thanks, was able to get the logs for the error happening with some help. Seems to be related to their "allowlist" changes to lemmy (at least that's the error being thrown back). I'll try to track down in the code, as I believe I mentioned before I believe it sends a request to unfollow to remotes even when you aren't following, and make a patch soon(tm). (For jerry, this is an mbin bug, so feel free to move this to the mdev mag if you want or it can hang out here but isn't specific to your instance)
If you use ublock origin adding the following to the filter list should help.
fedia.io##article.entry:has(aside.meta > a.magazine-inline[href$="@hexbear.netl"])
As an update, this should be fixed as of this PR
I was able to verify that I couldn't block before, and could after the change
So it should get into the next tagged release version (which will likely be 1.5.0) and eventually be working on this instance
Blocking instances on kbin / mbin does not work though. You will still see the communities threads & users (and getting to that page in the first place is a hassle too since there's no link to it).
I've noticed that a large number of communities on Lemmy.world aren't visible here. You only get a 404. I've noticed this when looking at US state communities:
Ok. I don’t know why this works, but if you search for the magazine (ie search for oregon@lemmy.world) it’ll return a result. After that, the links you posted starts working.
It should receive data from any activity at the other community. The problem seems to be Fedia not receiving data from certain communities.
It's not just the state communities. There's a lot of communities that are missing posts. Even ones that happened recently. For instance: https://fedia.io/m/netsec@lemmy.world
I am not sure what is going on with some of the lemmy.world communities - some do seem to be having issues like netsec, while others like https://fedia.io/m/news@lemmy.world are humming along very well. I'll keep poking at it.
I mentioned in the adjacent post but if you do see magazines acting oddly, it could be there are no local subscribers. Hopefully we get a UI fix in soon with the PR I had mentioned, because it's a bit of a bad user experience now. (I noticed world from lemmy world on debounced's instance was getting no posts, or rarely a post with no likes, and subscribed, and suddenly everything came in fine.) So probably a good idea for people, if you want to see posts from somewhere, make sure to subscribe.
(This was technically always true, but a while back admins were auto subscribed so a magazine would always have 1 subscriber. That changed, and that change was even backported to kbin by ernest, so it should be like that back in kbin too for new magazines.)
Make sure to subscribe to communities you want. It's not just searching for it and loading it, but if there are no local subscribers it won't get updates (and the last subscriber unsubbing would put it in this situation).
This became a bit harder to tell ever since mbin switched to showing real subscriber numbers rather than local like kbin has. There is an active PR to try to address this issue so users are able to tell when data isn't coming in
Thanks. That seems to resolve the issue. Though, for a lot of smaller communities, no one is going to subscribe to them here, effectively making them invisible here.
I don't completely disagree, hopefully the PR I linked, once complete, will make it a lot more obvious to people when this scenario occurs. I think from an instance owner perspective, they wouldn't want to constantly be paying the overhead on communities no one has an actual interest in viewing (edit: this isn't worded well, so forgive me, obviously people may want to view it but not have it show up in their sub feed, so perhaps another action item is a way to split subscription lists, which I think was already requested), as it has a very real financial cost to them. But I will keep this in mind, I meant to investigate how lemmy works (whether they also require a subscriber, I mean they do because this is how AP works but they might make a fake user or something, I never had a chance to look, but I'd be curious what tools they have to stop incoming messaging for when an instance owner wants to save bandwidth)
It looks like someone from the remote liked it recently, which, and I'm channeling Benti here because I'm not as good at the AP side of things, announced to subscribers which brought it in here. At least that's my best guess
Growing pains are to be expected. You're probably aware that some people (myself included) are shifting here [from|in addition to] kbin.social; that extra load probably doesn't help.
Ernest made a post today, yes, but kbin.social has reached a point which demands a next level of administration (from both technical and non-technical perspectives). While I want that project to thrive, there is writing on the wall which unfortunately cannot be ignored.
Ok. Maintenance completed. I've moved fedia.io to a container-based setup that is now running on my main infrastructure - a 48 core/96 thread AMD epyc zen4 genoa with 256GB of DDR5 ECC ram and 2x4TB nvme SSDs, backed by a dedicated database server with the exact same specs, on a 10gbps network.
fedia
Top
This magazine is from a federated server and may be incomplete. Browse more on the original instance.