@jerry@fedia.io avatar

jerry

@jerry@fedia.io

This profile is from a federated server and may be incomplete. Browse more on the original instance.

jerry,
@jerry@fedia.io avatar

yes, it's working on catching up now.

jerry,
@jerry@fedia.io avatar

Which lemmy instance were you not seeing it on? One of the challenges with all fediverse applications (mbin, lemmy, mastodon, etc) is that posts and comments don’t federate with other instances unless someone on the other instance follows the person or the magazine.

I do not believe existing posts and comments will federate to a different instance after someone on that instance subscribes to the magazine. Only new posts and comments will.

jerry,
@jerry@fedia.io avatar

The cause is unclear to me, but the processes that manage incoming and outgoing federation were "stuck" until I manually intervened. It's running now. I'm adding to my to-do list for after my job ends in 10 days to implement some sort of detection and hopefully automated restart for when this happens again.

jerry,
@jerry@fedia.io avatar

That GitHub tracker is for problems with mbin code, not with instances running Mbin software. The problem that Fedia experienced is very likely caused by a bug of some sort in mbin (or something that Mbin is depending on), but we don’t know for sure what it is yet. Once I’m unemployed in a week, I hope to have more time to debug issues like this and get them resolved.

jerry,
@jerry@fedia.io avatar

It's working through the backlog now. Another obscure bug in an incoming message that stopped message processing.

jerry,
@jerry@fedia.io avatar

Thank you for asking. It's on our about page here: https://fedia.io/about

Yet more stability issues with fedia.io (fedia.io)

Yesterday, the fedia.io server locked up. I was able to reboot it remotely and it came up clean. After less than an hour, the server froze again. This happened several more times throughout the day. Unfortunately, there were no logs recording what happened, and nothing on the console - just frozen hardware....

jerry,
@jerry@fedia.io avatar

To be clear, I have used a lot of Hetzner servers and I’ve only had issues with this particular model of bare metal server.

jerry,
@jerry@fedia.io avatar

I'll open the back up. An issue with mbin is that there's no way to moderate new accounts, and so I have been battling massive amounts of spam, and the moderation interface for mbin is not super efficient, so it creates a LOT of work.

jerry,
@jerry@fedia.io avatar

Hi all. I am looking into why this is happening.

Apologies for the recent downtime (fedia.io)

The server fedia.io had been running on started developing stability problems overnight from Thursday April 4 to Friday April 5. By Saturday (today), the system was completely unbootable. After attempting to resolve the hardware issue with Hetzner (the ISP) for about 6 hours, I gave up and moved the site to a new server. All...

Apologies for the recent downtime (fedia.io)

The server fedia.io had been running on started developing stability problems overnight from Thursday April 4 to Friday April 5. By Saturday (today), the system was completely unbootable. After attempting to resolve the hardware issue with Hetzner (the ISP) for about 6 hours, I gave up and moved the site to a new server. All...

jerry,
@jerry@fedia.io avatar

Report at least one of the posts from any spam account you see. I try to suspend and delete them as I see them, but alas I am not omniscient and don’t know where all the spam accounts are

jerry,
@jerry@fedia.io avatar

thanks. I think I've gotten rid of all the spam accounts, posts, and replies from the recent issue, but there may be more I am not finding.

jerry,
@jerry@fedia.io avatar

it is working for me as well. I'll check to see if there was anything in the error logs.

jerry,
@jerry@fedia.io avatar

ok - just looked through logs and I am not seeing anything that looks related to this issue. There are a good number of error 500's coming from kbin.social, but none appear to be around the time of these posts, nor associated with the referenced magazine.

jerry,
@jerry@fedia.io avatar

As a bonus, the most excellent mbin folk pointed out a problem in the database and that has additionally helped dramatically reduce the load

jerry,
@jerry@fedia.io avatar

Hello. It is not. I think it is the largest. I am not sure there is an official instance for mbin the way there is for kbin or mastodon, etc

jerry,
@jerry@fedia.io avatar

I am not - I have tried to help, and have been helped by the developers though

Fedia.io Downtime Saturday, March 9 (fedia.io)

I am going to be rehoming fedia.io to a new server. Today, fedia.io runs on shared app server and shared database server. I will be moving fedia.io to a dedicated 48 core/96 thread AMD epyc server with 256GB of RAM, which will house both the app and the database. This is less about fedia.io and more about the other apps running...

jerry,
@jerry@fedia.io avatar

it may be related to the ongoing federation problems. https://fedia.io/m/french@sopuli.xyz is working now for me. there are no posts in it yet on our instance because no one had subscribed to it until now.

jerry,
@jerry@fedia.io avatar

I’ll take a look

jerry,
@jerry@fedia.io avatar

Ok. I don’t know why this works, but if you search for the magazine (ie search for oregon@lemmy.world) it’ll return a result. After that, the links you posted starts working.

jerry,
@jerry@fedia.io avatar

The Florida one does because I followed it and there has been posts since then

jerry,
@jerry@fedia.io avatar

Also, I checked the source and there have been no new posts since. I'll keep an eye on it.

jerry,
@jerry@fedia.io avatar

I am not sure what is going on with some of the lemmy.world communities - some do seem to be having issues like netsec, while others like https://fedia.io/m/news@lemmy.world are humming along very well. I'll keep poking at it.

jerry,
@jerry@fedia.io avatar

actually, I'm wrong - the posts that appeared after I followed it are weeks old. I can't explain that.

jerry,
@jerry@fedia.io avatar

The server is busily processing the 1,200,000 messages that queued up over the past 20 hours. It’s died 3 times in the past few minutes, so I’m not optimistic about how long this will take

jerry,
@jerry@fedia.io avatar

Ah - that is what we’re here for. I know kbin has had a cloud of uncertainty around it. Did something recently happen on kbin.social?

jerry,
@jerry@fedia.io avatar

Up to 1,700,000 in the queue 😱

jerry,
@jerry@fedia.io avatar

The good news is that I think I figured out where the problematic messages are coming from. Now I have to figure out what it is about them.

jerry,
@jerry@fedia.io avatar

it took 3 days to process the backlog, but it's caught up now and I've not seen any re-occurrence of the prior problem.

Request for errors on fedia.io (fedia.io)

Hi all. I was recently made aware that people have been getting error 429's and other error 500's when visiting fedia.io. My hope/expectation is that those will no longer happen now that I've moved back to a bare metal install, but if you do experience that, please comment below, or if that doesn't work, send me an emai to...

jerry,
@jerry@fedia.io avatar

Thank you. I will investigate

jerry,
@jerry@fedia.io avatar

Thanks. I missed that and will go apply the upgrade now.

jerry,
@jerry@fedia.io avatar

ok, fix applied.

jerry,
@jerry@fedia.io avatar

I applied a fix that (hopefully) resolves this new problem. Please let me know if you continue to see it.

jerry,
@jerry@fedia.io avatar

yes, there was a problem with rabbitmq that I found/resolved.

jerry,
@jerry@fedia.io avatar

With any luck, this will take care of the 429 errors too. I actually didn’t realize that was happening until I read the thread you linked to above.

jerry,
@jerry@fedia.io avatar

Thank you for this. It is behind nginx, so I’ll likely have to do this

jerry,
@jerry@fedia.io avatar

It's been a continued source of child pornography.

jerry,
@jerry@fedia.io avatar

I will lift the ban again. This has happened twice in the past, and each time within ~12 hours, I've had more child porn landing on the server from them. If it happens again though, I will likely leave it that way.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • random
  • meta
  • Macbeth
  • All magazines