@jerry@fedia.io avatar

jerry

@jerry@fedia.io

This profile is from a federated server and may be incomplete. Browse more on the original instance.

jerry,
@jerry@fedia.io avatar

Hi all. I am looking into why this is happening.

Apologies for the recent downtime (fedia.io)

The server fedia.io had been running on started developing stability problems overnight from Thursday April 4 to Friday April 5. By Saturday (today), the system was completely unbootable. After attempting to resolve the hardware issue with Hetzner (the ISP) for about 6 hours, I gave up and moved the site to a new server. All...

Apologies for the recent downtime (fedia.io)

The server fedia.io had been running on started developing stability problems overnight from Thursday April 4 to Friday April 5. By Saturday (today), the system was completely unbootable. After attempting to resolve the hardware issue with Hetzner (the ISP) for about 6 hours, I gave up and moved the site to a new server. All...

jerry,
@jerry@fedia.io avatar

Report at least one of the posts from any spam account you see. I try to suspend and delete them as I see them, but alas I am not omniscient and don’t know where all the spam accounts are

jerry,
@jerry@fedia.io avatar

thanks. I think I've gotten rid of all the spam accounts, posts, and replies from the recent issue, but there may be more I am not finding.

jerry,
@jerry@fedia.io avatar

it is working for me as well. I'll check to see if there was anything in the error logs.

jerry,
@jerry@fedia.io avatar

ok - just looked through logs and I am not seeing anything that looks related to this issue. There are a good number of error 500's coming from kbin.social, but none appear to be around the time of these posts, nor associated with the referenced magazine.

jerry,
@jerry@fedia.io avatar

As a bonus, the most excellent mbin folk pointed out a problem in the database and that has additionally helped dramatically reduce the load

jerry,
@jerry@fedia.io avatar

Hello. It is not. I think it is the largest. I am not sure there is an official instance for mbin the way there is for kbin or mastodon, etc

jerry,
@jerry@fedia.io avatar

I am not - I have tried to help, and have been helped by the developers though

Fedia.io Downtime Saturday, March 9 (fedia.io)

I am going to be rehoming fedia.io to a new server. Today, fedia.io runs on shared app server and shared database server. I will be moving fedia.io to a dedicated 48 core/96 thread AMD epyc server with 256GB of RAM, which will house both the app and the database. This is less about fedia.io and more about the other apps running...

jerry,
@jerry@fedia.io avatar

it may be related to the ongoing federation problems. https://fedia.io/m/french@sopuli.xyz is working now for me. there are no posts in it yet on our instance because no one had subscribed to it until now.

jerry,
@jerry@fedia.io avatar

I’ll take a look

jerry,
@jerry@fedia.io avatar

Ok. I don’t know why this works, but if you search for the magazine (ie search for oregon@lemmy.world) it’ll return a result. After that, the links you posted starts working.

jerry,
@jerry@fedia.io avatar

The Florida one does because I followed it and there has been posts since then

jerry,
@jerry@fedia.io avatar

Also, I checked the source and there have been no new posts since. I'll keep an eye on it.

jerry,
@jerry@fedia.io avatar

I am not sure what is going on with some of the lemmy.world communities - some do seem to be having issues like netsec, while others like https://fedia.io/m/news@lemmy.world are humming along very well. I'll keep poking at it.

jerry,
@jerry@fedia.io avatar

actually, I'm wrong - the posts that appeared after I followed it are weeks old. I can't explain that.

jerry,
@jerry@fedia.io avatar

The server is busily processing the 1,200,000 messages that queued up over the past 20 hours. It’s died 3 times in the past few minutes, so I’m not optimistic about how long this will take

jerry,
@jerry@fedia.io avatar

Ah - that is what we’re here for. I know kbin has had a cloud of uncertainty around it. Did something recently happen on kbin.social?

jerry,
@jerry@fedia.io avatar

Up to 1,700,000 in the queue 😱

jerry,
@jerry@fedia.io avatar

The good news is that I think I figured out where the problematic messages are coming from. Now I have to figure out what it is about them.

jerry,
@jerry@fedia.io avatar

it took 3 days to process the backlog, but it's caught up now and I've not seen any re-occurrence of the prior problem.

Request for errors on fedia.io (fedia.io)

Hi all. I was recently made aware that people have been getting error 429's and other error 500's when visiting fedia.io. My hope/expectation is that those will no longer happen now that I've moved back to a bare metal install, but if you do experience that, please comment below, or if that doesn't work, send me an emai to...

jerry,
@jerry@fedia.io avatar

Thank you. I will investigate

jerry,
@jerry@fedia.io avatar

Thanks. I missed that and will go apply the upgrade now.

jerry,
@jerry@fedia.io avatar

ok, fix applied.

jerry,
@jerry@fedia.io avatar

I applied a fix that (hopefully) resolves this new problem. Please let me know if you continue to see it.

jerry,
@jerry@fedia.io avatar

yes, there was a problem with rabbitmq that I found/resolved.

jerry,
@jerry@fedia.io avatar

With any luck, this will take care of the 429 errors too. I actually didn’t realize that was happening until I read the thread you linked to above.

jerry,
@jerry@fedia.io avatar

Thank you for this. It is behind nginx, so I’ll likely have to do this

jerry,
@jerry@fedia.io avatar

It's been a continued source of child pornography.

jerry,
@jerry@fedia.io avatar

I will lift the ban again. This has happened twice in the past, and each time within ~12 hours, I've had more child porn landing on the server from them. If it happens again though, I will likely leave it that way.

jerry,
@jerry@fedia.io avatar

I'm going to rebuild fedia on a non-container server which should fix the problem

jerry,
@jerry@fedia.io avatar

The problem is that federation keeps dying every few hours - well, it doesn't entirely die, but becomes incredibly slow.

jerry,
@jerry@fedia.io avatar

Hi all. I am working on it. I am trying to figure out what the problem is this time rather than just restarting the containers.

jerry,
@jerry@fedia.io avatar

I am going to apply this upgrade this weekend. I spent a few hours looking through logs this morning but didn't see anything obvious (there are a breathtaking amount of logs) so gave up and restarted the containers. Will look again when it fails after I make log storage persistent

jerry,
@jerry@fedia.io avatar

lemmy.world is having some issues

jerry,
@jerry@fedia.io avatar

thanks - I wasn't aware of the issue. I will try to figure out why.

jerry,
@jerry@fedia.io avatar

How large are the gifs you're posting? So far, my test uploads have been successful and I'm not able to get it to fail...

jerry,
@jerry@fedia.io avatar

That gives me something to go on. Thank you.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • random
  • meta
  • Macbeth
  • All magazines