This doesn't work to avoid thrashing. The kernel may invoke the OOM killer slightly quicker if you have no swap, so I guess that can sort of help, but it doesn't properly solve the problem.

On Linux, there's a thing called the page cache (aka disk cache): Every time (part of) a file gets read to or written from, that (part of) the file gets copied to RAM. The file is then kept there unless that RAM is needed for something more important. It is cached in RAM. But since it is also on disk, the kernel can drop the file from RAM anytime it wants.

If you're low on RAM, the kernel therefore evicts all of the disk cache, because it can, because those pages can be reloaded from disk if needed. This means it will drop all the programs you're running, the binary code. So any program you're running is constantly interrupted, because its code is not in RAM.

So it runs a couple of instructions, but oh no! Call to function foo() from glibc, but guess what? That's on disk. Queue wait for the kernel to load that. Oh now it wants function bar() from zlib, shit! Need to load that. Since loading stuff from disk is about as slow as running like a gazillion instructions, all your programs are like 1000x slower now.

This happens even with zero swap.

The correct advice is the one from @RedWeasel@lemmy.world: install/enable systemd-oomd or earlyoom.

I wouldn't recommend disabling swap completely and you do need it

https://chrisdown.name/2018/01/02/in-defence-of-swap.html

Unfortunately, there is no guarantee that the leaking process will be the next process to try to allocate memory after you run out. It might actually be your window manager, for example.

The OOM killer is a last-ditch attempt by the OS to keep running, but it is very likely to leave your system in an unstable state.

I think that systemd oomd is pretty slow. I get a quite a few minutes of unresponsive system while waiting for oomd to do something. Honestly rebooting is faster than waiting for it to work.

I recently installed and enabled earlyoomd on fedora, and on an initial test, I still got an unresponsive system with the default settings, but after around 20 seconds or so, it killed the responsible process and I was able to continue working. Not perfect, but much better.