@jerry@fedia.io avatar

jerry

@jerry@fedia.io

This profile is from a federated server and may be incomplete. Browse more on the original instance.

jerry,
@jerry@fedia.io avatar

I see it - it's working again and I'm trying to figure out why rabbitmq died yet again.

jerry,
@jerry@fedia.io avatar

thank you again for your help! I was at the end of my rope and you gave me the idea that solved it

jerry,
@jerry@fedia.io avatar

I don’t know yet - it’s definitely not expected, so my guess is an unintentional bug in mbin somewhere. I am hoping to find a way run a profiler or something similar to see what it’s doing.

jerry,
@jerry@fedia.io avatar

Thanks. It’s a strange problem that only happens when trying to post directly to a community/magazine that resides on another instance. So this works fine because this community it hosted locally. I think I understand when but hoping to get some clarification.

jerry,
@jerry@fedia.io avatar

Not entirely. It looks like the rabbit issue was only impacting one of the queues (“deliver”), though I would have expected that to impact things like microblog too. All I can say with clarity is that the instance was operating in a very unhealthy state.

The queue appears like it’ll take several hours to flush, but it’s working.

jerry,
@jerry@fedia.io avatar

Once it is more reliable, then I’ll agree, but thank you

jerry,
@jerry@fedia.io avatar

I will check.

jerry,
@jerry@fedia.io avatar

ok - rabbitmq started having prroblems with the delivery queue again. I got it going again. Those messages should be delivered soon.

jerry,
@jerry@fedia.io avatar

Working on it

jerry,
@jerry@fedia.io avatar

ok - the queues are processing again. I will work on a more permanent fix after dinner

jerry,
@jerry@fedia.io avatar

:( I'm working on it

jerry,
@jerry@fedia.io avatar

It’s almost caught up. My apologies. I am trying to get it fixed permanently.

jerry,
@jerry@fedia.io avatar

Note that even once I get this fixed, there will inevitably be another problem crop up. Posting here is fine, but an email to jerry@infosec.exchange or a ping to @jerry (my mastodon account) would probably be faster (note, when federation breaks here, messages to @jerry wouldn't get through from fedia.io...

jerry,
@jerry@fedia.io avatar

be aware that this is fixed now

jerry,
@jerry@fedia.io avatar

be aware that this is fixed now

jerry,
@jerry@fedia.io avatar

be aware that this is fixed now

jerry, (edited )
@jerry@fedia.io avatar

Do you know whether there are any users on kbin.melroy.org that subscribe to your magazine? If not, that will explain why your post did not show up until you searched for it.

jerry,
@jerry@fedia.io avatar

Thank you for the problem report. I will be looking into this later today.

jerry,
@jerry@fedia.io avatar

I found the problem and the queues are running now. I will make a thread about it.

jerry,
@jerry@fedia.io avatar

I am not 100% sure. Fedia.io is running on a beast of a server, and so long as it’s working correctly, it should be able to deliver it instantly. But that doesn’t mean that the receiving servers are able to consume and render them that fast.

jerry,
@jerry@fedia.io avatar

It's a known issue - I have been working with @melroy for a while now to resolve. I think we now understand what is happening under the hood, but not yet why it is happening.

jerry,
@jerry@fedia.io avatar

Which magazines/communities are you seeing problems with?

jerry,
@jerry@fedia.io avatar

Outbound federation was indeed broken. I fixed it, but there was a huge backlog the server had to work through, which took about 12 hours to complete. I just checked and everything appears to be working ok. I am going to create some automation that will detect and alert on this (or other issues) happening in the future.

jerry,
@jerry@fedia.io avatar

there isn't an inteval per se - I don't yet know why it's not immediate. It's possible that the delay is on the receiving side - the server that fedia.io is quite substantial and unless there is some sort of bug, the processing should happen immediately.

jerry,
@jerry@fedia.io avatar

Howdy! Mbin (and lemmy) are very different things. It’s sort of like the difference between Twitter and Reddit. You can sort of interact back and forth, but to get the full experience, you have to either be on a lemmy or mbin (or piefed) instance.

jerry,
@jerry@fedia.io avatar

What works for me on both mastodon and Lemmy is a free text question: why do you want to join?

The user enters whatever they like and it goes into a moderation queue. Both lemmy and mastodon send me an email when a new account is ready to review.

I read the response and choose to whether to approve their account. At the moment, spammers are really bad at answering the “why do you want to join” questions.

jerry,
@jerry@fedia.io avatar

There were lots of changes around the same time. I removed fedia.io from the CDN a few days ago though didn't announce it, yet the errors continue.

Account creation issue (lemmy.world)

When trying to register, I fill out a username, email, and password (copy/pasted from a password manager, so no chance of mistyping). I complete the captcha, check the consent box, click register, and... nothing. Page refreshes, but there's no errors displayed or anything. I've tried resetting my password, but I never get an...

jerry,
@jerry@fedia.io avatar

If you can, please try again. If you still have problems, shoot me an email to jerry@infosec.exchange and I'll troubleshoot the issue

jerry,
@jerry@fedia.io avatar

I did end up disabling registration due to spam. I can either open them up at a time you’re free to try again or I can manually create an account for you.

jerry,
@jerry@fedia.io avatar

and there was much rejoicing!

jerry,
@jerry@fedia.io avatar

Ahh - thank you!

jerry,
@jerry@fedia.io avatar

thank for pointing that out. Its on my list to fix

jerry,
@jerry@fedia.io avatar

I moved fedia.io away from fastly. I have a nagging feeling it has something to do with fastly. Can you let me know if you continue to see this?

jerry,
@jerry@fedia.io avatar

This annoys me about the fediverse - people take a chance on coming here and then repeatedly get left in the dark when their instance is shut down. That's why I was so very happy when you and others helped me get fedia.io back to healthy.

jerry,
@jerry@fedia.io avatar

Most interesting: the problem had only been happening on MS Edge on my laptop. I have been using safari on my phone without issue. Just a bit ago, i refreshed the page and now every time I revisit the site, I have to log back in, just like on Edge. It’s like the old session expired and the new ones aren’t sticking. I’ll try FF on my phone.

Note: even in the time I started typing this reply to when I hit the “add comment” button, I got logged out

jerry,
@jerry@fedia.io avatar

I have so many errors in prod.log that it's hard to tell for certain, but when I try to filter out those that are associated with failed federation events, that seems to be when I'm left with. I am trying again to see if I can confirm

jerry,
@jerry@fedia.io avatar

I do not have 2fa active at the moment

jerry,
@jerry@fedia.io avatar

I do not have 2fa turned on right now.

jerry,
@jerry@fedia.io avatar

Indeed. I am trying to get it to happen again now that I’ve got the logs filtered down to a manageable level.

jerry,
@jerry@fedia.io avatar

ok - I just had it happen again while looking at logs. interestingly, there was NOT a CSRF log when that happened. There were a bunch of other errors, but enough that I could look through all of them and see that they were all related to activitypub issues - signaturevalidator and the like

jerry,
@jerry@fedia.io avatar

it's hard to make a blanket statement, because it depends on the details of the application. CSRF attacks are definitely real and common, but using csrf tokens isn't critical in every application. For example, I think we have CORS headers enabled, I don't think we have functionality that allows embedded iframes, but we do allow links - if we have administrative functions that can be triggered solely with GET parameters, then someone could trick an administrator into doing something that caused damage by clicking on a link in a post. The only one that would obviously work that I can see is "logout", which would be annoying, but not world ending, and would work for everyone, not just administrators.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • random
  • meta
  • Macbeth
  • All magazines