@jerry@fedia.io avatar

jerry

@jerry@fedia.io

This profile is from a federated server and may be incomplete. Browse more on the original instance.

Request for errors on fedia.io (fedia.io)

Hi all. I was recently made aware that people have been getting error 429's and other error 500's when visiting fedia.io. My hope/expectation is that those will no longer happen now that I've moved back to a bare metal install, but if you do experience that, please comment below, or if that doesn't work, send me an emai to...

jerry,
@jerry@fedia.io avatar

With any luck, this will take care of the 429 errors too. I actually didn’t realize that was happening until I read the thread you linked to above.

jerry,
@jerry@fedia.io avatar

Thank you for this. It is behind nginx, so I’ll likely have to do this

jerry,
@jerry@fedia.io avatar

It's been a continued source of child pornography.

jerry,
@jerry@fedia.io avatar

I'm going to rebuild fedia on a non-container server which should fix the problem

jerry,
@jerry@fedia.io avatar

The problem is that federation keeps dying every few hours - well, it doesn't entirely die, but becomes incredibly slow.

jerry,
@jerry@fedia.io avatar

Hi all. I am working on it. I am trying to figure out what the problem is this time rather than just restarting the containers.

jerry,
@jerry@fedia.io avatar

I am going to apply this upgrade this weekend. I spent a few hours looking through logs this morning but didn't see anything obvious (there are a breathtaking amount of logs) so gave up and restarted the containers. Will look again when it fails after I make log storage persistent

jerry,
@jerry@fedia.io avatar

lemmy.world is having some issues

jerry,
@jerry@fedia.io avatar

thanks - I wasn't aware of the issue. I will try to figure out why.

jerry,
@jerry@fedia.io avatar

How large are the gifs you're posting? So far, my test uploads have been successful and I'm not able to get it to fail...

jerry,
@jerry@fedia.io avatar

That gives me something to go on. Thank you.

jerry,
@jerry@fedia.io avatar

if/when you see things like this, please tag my @jerry account - the notifications on fedia are a bit lacking.

jerry,
@jerry@fedia.io avatar

Kbin/mbin is still very much a work in progress and there are certain features that aren’t implemented yet, unfortunately

jerry,
@jerry@fedia.io avatar

Most were suspended for child porn at various points - though I think it was not necessarily something they condone, but lemmy has damn near no ability to moderate or remove images once they land on a server

jerry,
@jerry@fedia.io avatar

Ok. Maintenance completed. I've moved fedia.io to a container-based setup that is now running on my main infrastructure - a 48 core/96 thread AMD epyc zen4 genoa with 256GB of DDR5 ECC ram and 2x4TB nvme SSDs, backed by a dedicated database server with the exact same specs, on a 10gbps network.

It's always something... [sorry for the downtime] (fedia.io)

I was happily working on a different set of servers when apparently the database server for fedia.io became unresponsive. Once I realized it, I forced a reboot and everything is back to normal. Now that the code and database are stable(ish), I will be moving the site to actual server hardware in the coming days while I have some...

jerry,
@jerry@fedia.io avatar

I am working with the most excellent mbin team trying to figure out why this is happening...

jerry,
@jerry@fedia.io avatar

ok - it's fixed now. My apologies for the problems.

jerry,
@jerry@fedia.io avatar

My best recommendation is to create issues in their github repo here: https://github.com/MbinOrg/mbin/issues

jerry,
@jerry@fedia.io avatar

I think the issue now is with the containerized version of mbin. I tried saving money by running it in my container host, but it's clear to me that is not going to work out. So I will be rolling it back to a bare metal server :/

jerry,
@jerry@fedia.io avatar

I am going to wait till it happens again and try to debug. I really don’t want another $100/month server bill if I can avoid it

jerry,
@jerry@fedia.io avatar

That would be great. Thank you!

jerry,
@jerry@fedia.io avatar

Maintenance completed

jerry,
@jerry@fedia.io avatar

Is your account @socialspirit?

jerry,
@jerry@fedia.io avatar

thanks. I am working on it

jerry,
@jerry@fedia.io avatar

Can you give it another try? It looks to be correlated with php running out of memory (not sure why your session would require more that 512MB of ram, but lets confirm that is the case first.

jerry,
@jerry@fedia.io avatar

ok - I bumped up the memory limit in php to 1gb and that appears to have fixed the problem. I have no idea yet why it takes so much memory, though. But I'm glad it worked

jerry,
@jerry@fedia.io avatar

no - this is the first time - and as far as I can tell, the only time it's happened according to my logs...

jerry,
@jerry@fedia.io avatar

I’ll take a look at that

jerry,
@jerry@fedia.io avatar

Many thanks!

jerry,
@jerry@fedia.io avatar

I will run an update tonight. There are some other updates I want to pull in, like removing noise from logs.

jerry,
@jerry@fedia.io avatar

I did not update last night - there are many in flight code changes and I’d prefer not to cause more problems, so letting that settle down for a bit before I update.

jerry,
@jerry@fedia.io avatar

This should be working now.

jerry,
@jerry@fedia.io avatar

Is it still happening?

jerry,
@jerry@fedia.io avatar

Awesome! It took a lot to clean things up, and I am not going to be surprised if I missed something

jerry,
@jerry@fedia.io avatar

I am aware that visiting the notifications link gives an error 500. I will work on it.

jerry,
@jerry@fedia.io avatar

Ok, this problem is fixed now as well

jerry,
@jerry@fedia.io avatar

Thanks. I feel bad that it took this long to get here, though

jerry,
@jerry@fedia.io avatar

They should be. I’ve not seen any since I changed the nginx config the prevent that domain from pointing to the mbin instance.

jerry,
@jerry@fedia.io avatar

As part of the database clean up, I had to remove many corrupt records, so the answer is yes there are missing posts and comments, however I do have backups have I have been attempting to find ways to reinsert them without recreating the problems. I’ve not had much luck with that yet.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • random
  • meta
  • Macbeth
  • All magazines