Another day, another update.

More troubleshooting was done today. What did we do:

  • Yesterday evening @phiresky@[email protected] did some SQL troubleshooting with some of the lemmy.world admins. After that, phiresky submitted some PRs to github.
  • @[email protected] created a docker image containing 3PR’s: Disable retry queue, Get follower Inbox Fix, Admin Index Fix
  • We started using this image, and saw a big drop in CPU usage and disk load.
  • We saw thousands of errors per minute in the nginx log for old clients trying to access the websockets (which were removed in 0.18), so we added a return 404 in nginx conf for /api/v3/ws.
  • We updated lemmy-ui from RC7 to RC10 which fixed a lot, among which the issue with replying to DMs
  • We found that the many 502-errors were caused by an issue in Lemmy/markdown-it.actix or whatever, causing nginx to temporarily mark an upstream to be dead. As a workaround we can either 1.) Only use 1 container or 2.) set proxy_next_upstream timeout; max_fails=5 in nginx.

Currently we’re running with 1 lemmy container, so the 502-errors are completely gone so far, and because of the fixes in the Lemmy code everything seems to be running smooth. If needed we could spin up a second lemmy container using the proxy_next_upstream timeout; max_fails=5 workaround but for now it seems to hold with 1.

Thanks to @[email protected] , @[email protected] , @[email protected], @[email protected] , @[email protected] , @[email protected] for their help!

And not to forget, thanks to @[email protected] and @[email protected] for their continuing hard work on Lemmy!

And thank you all for your patience, we’ll keep working on it!

Oh, and as bonus, an image (thanks Phiresky!) of the change in bandwidth after implementing the new Lemmy docker image with the PRs.

Edit So as soon as the US folks wake up (hi!) we seem to need the second Lemmy container for performance. So that’s now started, and I noticed the proxy_next_upstream timeout setting didn’t work (or I didn’t set it properly) so I used max_fails=5 for each upstream, that does actually work.

  • phiresky@lemmy.world
    9075·
    2 years ago

    server load is too low, everyone upvote more stuff so i can optimize more

    edit: guess there is some more work to be done 😁

    • Carnelian@lemmy.world
      971·
      2 years ago

      I agree. Felt it immediately when I started browsing. Everything is faster and more responsive, on top of the error messages disappearing

  • Gnothi@lemmy.world
    1283·
    2 years ago

    This is why having a big popular instance isn’t all bad. It helps detect and fix the scaling problems and inefficiencies for all the other 1000s of instances out there!

    • AlmightySnoo 🐢🇮🇱🇺🇦@lemmy.world
      622·
      2 years ago

      This, if everyone kept just spreading out to smaller instances as suggested in the beginning, while still a sensible thing to do, no one would have noticed these performance issues. We need to think a few years out, assuming Lemmy succeeds and Reddit dies, and expect that “small instance” will mean 50k users.

    • Bilb!@lem.monster
      18·
      2 years ago

      I’m actually kinda waiting a few releases to start promoting my instance anywhere, letting some other brave instance admins work the kinks out a bit first.

    • rockstarpirate@lemmy.world
      14·
      2 years ago

      Agreed. I decided to keep my community on lemmy.world specifically because of the community investment I see being put into it.

  • kionay@lemmy.world
    114·
    2 years ago

    You guys had better quit it with all this amazing transparency or it’s going to completely ruin every other service for me. Seriously though amazing work and amazing communication.

  • sv1sjp@lemmy.world
    92·
    2 years ago

    Thank you guys for your awesome work!

    Also to other people: DONATE TO FOSS PROJECTS. If 50.000 people donate only 0.5€, we have 25.000€ for funding the servers, coding, motivating/ people etc. Just don’t take a cup of coffee for 1 day. We are already 2 millions in Lemmy instances. We can build a decentralized world together!!

  • Marxine@lemmy.world
    81·
    2 years ago

    Boy does it feel good to have those reports and understand the work you guys do. It’s really inspiring! Thanks for your hard work, everything has been silk smooth! This instance is really great, Lemmy and its devs are really amazing and I feel at home in a nice, cozy community.

  • Spectator@lemmy.worldEnglish
    80·
    2 years ago

    I’m not sure wtf you just said, but lemmy.world feels very smooth today, so thank you for your continued hard work!

  • sirnak@lemmy.world
    79·
    2 years ago

    Am I getting this correct: the whole lemmy.world instance run in one single container on one single host?

  • ericjmorey@lemmy.world
    65·
    2 years ago

    Submitting PRs is literally the most effective response that helps everyone who uses Lemmy. Thanks to you all.

    • clutchmatic@lemmy.world
      24·
      2 years ago

      Some managers of the devs are not that interested in significant optimizations… Depends on what incentives and company culture drives them

    • Zzombiee2361@lemmy.world
      12·
      2 years ago

      Some company would rather throw more hardware at the problem and make the devs work on another useless feature no one use

      • Kristho@lemm.ee
        1·
        2 years ago

        Yes! And that’s a very short seighted solution. And it feels so good to improve performance in code! That’s extra performance for “free” 😎