β² 385 βΌ Who is using my file? (lemmy.ml) submitted 1 month ago by DaGammla@lemmy.ml to c/linuxmemes@lemmy.world 48 comments fedilink hide all child comments
[β] mogoh@lemmy.ml 64 points 1 month ago (1 child) I think "consider it done" puts it well. permalink fedilink source parent hideshow 2 child comments replies: [β] oats@piefed.zip 12 points 1 month ago (1 child) "I'll get to it, eventually" would ruin the meme but be more fitting, in my opinion. Had multiple occasions where people fought against filling disks and just couldn't see why. Well, that 10 gig log file you deleted two weeks ago? It's 20 gig now, and still being written to. lsof shows stuff like that. permalink fedilink source parent hideshow 2 child comments replies: [β] Redjard@reddthat.com 7 points 1 month ago* (2 children) Nothing new can open it immediately. So it's effectively deleted with old references slowly phasing out. Zombie files are an issue though. A while back I had a huge zombie file on a tmpfs which was filling all my ram. So I built a tool to track it down and traced it to a konsole instance with a killed tab that previously had billions of lines of stdout history. https://github.com/redjard/zombie-file-list zombie-file-list Lists Linux files that are still opened in a process but were deleted. These "zombie files" use up space and inodes but are hard to find. I wrote this because my /tmp tmpfs was taking up 32GB of ram despite the files inside summing to only 3MB. Usage: zombie-file-list <path to filesystem> Note: This command is designed to be run on filesystem roots, not paths in general. Sizes are apparent sizes, e.g. on ext4 the actual sizes are rounded up to the next 4KiB permalink fedilink source parent hideshow 4 child comments replies: [β] tal@lemmy.today 2 points 1 month ago* (last edited 1 month ago) (1 child) I wrote this because my /tmp tmpfs was taking up 32GB of ram despite the files inside summing to only 3MB. Note that tmpfs doesn't force its contents to remain in memory the kernel can move stuff there to swap space if it needs to do so. Ramfs is the filesystem that keeps things locked in memory: https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt With ramfs, there is no backing store. Files written into ramfs allocate dentries and page cache as usual, but there's nowhere to write them to. This means the pages are never marked clean, so they can't be freed by the VM when it's looking to recycle memory. ramfs and tmpfs: ---------------- One downside of ramfs is you can keep writing data into it until you fill up all memory, and the VM can't free it because the VM thinks that files should get written to backing store (rather than swap space), but ramfs hasn't got any backing store. Because of this, only root (or a trusted user) should be allowed write access to a ramfs mount. A ramfs derivative called tmpfs was created to add size limits, and the ability to write the data to swap space. Normal users can be allowed write access to tmpfs mounts. See Documentation/filesystems/tmpfs.txt for more information. permalink fedilink source parent hideshow 2 child comments replies: [β] Redjard@reddthat.com 1 point 1 month ago Maybe. It's been a while so I don't know 100% this was put to the test, but I wanna say the system has a weird kernel which leads to it not swapping out tmpfs properly. But ordinarily you should be right, this would simply ruin the stats visually until something forced it to swap out, since konsole shouldn't be accessing it. permalink fedilink source parent [β] Dave@lemmy.nz 1 point 1 month ago* (1 child) Could you have restarted to allow the OS to clean it up? permalink fedilink source parent hideshow 2 child comments replies: [β] Redjard@reddthat.com 7 points 1 month ago (1 child) I could have, but the system wasn't set up to restart without downtime, and the server was also remote and not easily accessible. It did acutally die due to a poweroutage some months later and took 2 days to get restarted. So yeah sometimes restarting is way more undesirable than loosing access to 32GB of ram. I would have just eaten that cost otherwise until a more opportune chance to restart. Besides, restarting to fix a problem is equivalent to giving up on understanding the issue, learning new stuff, and maybe finding a way better solution or preventing the type of error entirely. I get not finding the motivation when your software is working against you and learning is ultimately fruitless like on windows, or not having the time in the moment to figure it out properly, but a perfectly good bug on a linux system when you have time is prime real-estate to grow your skills and find fulfillment. permalink fedilink source parent hideshow 2 child comments replies: [β] Dave@lemmy.nz 1 point 1 month ago (1 child) Ah that makes sense. I had considered it might be a server but you mentioned a Konsole tab so my mind decided it must have been local machine. Crazy it took 2 days to restart the server! permalink fedilink source parent hideshow 2 child comments replies: [β] Redjard@reddthat.com 2 points 1 month ago There was a dedicated person on call, but it happened to be when they were away. The Konsole was left running from a local access, with a while true loop of a service status command. When that service was stopped later, the while loop started rerunning the script every second, filling the buffer with error messages. The tab was then killed remotely, but the Konsole window left running. Process ram usage went down but the file remained on tmpfs, which is not counted as ram usage so wasn't noticed. Then it took some time to notice the ram usage mismatch so noone thought of that konsole incident. permalink fedilink source parent
[β] oats@piefed.zip 12 points 1 month ago (1 child) "I'll get to it, eventually" would ruin the meme but be more fitting, in my opinion. Had multiple occasions where people fought against filling disks and just couldn't see why. Well, that 10 gig log file you deleted two weeks ago? It's 20 gig now, and still being written to. lsof shows stuff like that. permalink fedilink source parent hideshow 2 child comments replies: [β] Redjard@reddthat.com 7 points 1 month ago* (2 children) Nothing new can open it immediately. So it's effectively deleted with old references slowly phasing out. Zombie files are an issue though. A while back I had a huge zombie file on a tmpfs which was filling all my ram. So I built a tool to track it down and traced it to a konsole instance with a killed tab that previously had billions of lines of stdout history. https://github.com/redjard/zombie-file-list zombie-file-list Lists Linux files that are still opened in a process but were deleted. These "zombie files" use up space and inodes but are hard to find. I wrote this because my /tmp tmpfs was taking up 32GB of ram despite the files inside summing to only 3MB. Usage: zombie-file-list <path to filesystem> Note: This command is designed to be run on filesystem roots, not paths in general. Sizes are apparent sizes, e.g. on ext4 the actual sizes are rounded up to the next 4KiB permalink fedilink source parent hideshow 4 child comments replies: [β] tal@lemmy.today 2 points 1 month ago* (last edited 1 month ago) (1 child) I wrote this because my /tmp tmpfs was taking up 32GB of ram despite the files inside summing to only 3MB. Note that tmpfs doesn't force its contents to remain in memory the kernel can move stuff there to swap space if it needs to do so. Ramfs is the filesystem that keeps things locked in memory: https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt With ramfs, there is no backing store. Files written into ramfs allocate dentries and page cache as usual, but there's nowhere to write them to. This means the pages are never marked clean, so they can't be freed by the VM when it's looking to recycle memory. ramfs and tmpfs: ---------------- One downside of ramfs is you can keep writing data into it until you fill up all memory, and the VM can't free it because the VM thinks that files should get written to backing store (rather than swap space), but ramfs hasn't got any backing store. Because of this, only root (or a trusted user) should be allowed write access to a ramfs mount. A ramfs derivative called tmpfs was created to add size limits, and the ability to write the data to swap space. Normal users can be allowed write access to tmpfs mounts. See Documentation/filesystems/tmpfs.txt for more information. permalink fedilink source parent hideshow 2 child comments replies: [β] Redjard@reddthat.com 1 point 1 month ago Maybe. It's been a while so I don't know 100% this was put to the test, but I wanna say the system has a weird kernel which leads to it not swapping out tmpfs properly. But ordinarily you should be right, this would simply ruin the stats visually until something forced it to swap out, since konsole shouldn't be accessing it. permalink fedilink source parent [β] Dave@lemmy.nz 1 point 1 month ago* (1 child) Could you have restarted to allow the OS to clean it up? permalink fedilink source parent hideshow 2 child comments replies: [β] Redjard@reddthat.com 7 points 1 month ago (1 child) I could have, but the system wasn't set up to restart without downtime, and the server was also remote and not easily accessible. It did acutally die due to a poweroutage some months later and took 2 days to get restarted. So yeah sometimes restarting is way more undesirable than loosing access to 32GB of ram. I would have just eaten that cost otherwise until a more opportune chance to restart. Besides, restarting to fix a problem is equivalent to giving up on understanding the issue, learning new stuff, and maybe finding a way better solution or preventing the type of error entirely. I get not finding the motivation when your software is working against you and learning is ultimately fruitless like on windows, or not having the time in the moment to figure it out properly, but a perfectly good bug on a linux system when you have time is prime real-estate to grow your skills and find fulfillment. permalink fedilink source parent hideshow 2 child comments replies: [β] Dave@lemmy.nz 1 point 1 month ago (1 child) Ah that makes sense. I had considered it might be a server but you mentioned a Konsole tab so my mind decided it must have been local machine. Crazy it took 2 days to restart the server! permalink fedilink source parent hideshow 2 child comments replies: [β] Redjard@reddthat.com 2 points 1 month ago There was a dedicated person on call, but it happened to be when they were away. The Konsole was left running from a local access, with a while true loop of a service status command. When that service was stopped later, the while loop started rerunning the script every second, filling the buffer with error messages. The tab was then killed remotely, but the Konsole window left running. Process ram usage went down but the file remained on tmpfs, which is not counted as ram usage so wasn't noticed. Then it took some time to notice the ram usage mismatch so noone thought of that konsole incident. permalink fedilink source parent
[β] Redjard@reddthat.com 7 points 1 month ago* (2 children) Nothing new can open it immediately. So it's effectively deleted with old references slowly phasing out. Zombie files are an issue though. A while back I had a huge zombie file on a tmpfs which was filling all my ram. So I built a tool to track it down and traced it to a konsole instance with a killed tab that previously had billions of lines of stdout history. https://github.com/redjard/zombie-file-list zombie-file-list Lists Linux files that are still opened in a process but were deleted. These "zombie files" use up space and inodes but are hard to find. I wrote this because my /tmp tmpfs was taking up 32GB of ram despite the files inside summing to only 3MB. Usage: zombie-file-list <path to filesystem> Note: This command is designed to be run on filesystem roots, not paths in general. Sizes are apparent sizes, e.g. on ext4 the actual sizes are rounded up to the next 4KiB permalink fedilink source parent hideshow 4 child comments replies: [β] tal@lemmy.today 2 points 1 month ago* (last edited 1 month ago) (1 child) I wrote this because my /tmp tmpfs was taking up 32GB of ram despite the files inside summing to only 3MB. Note that tmpfs doesn't force its contents to remain in memory the kernel can move stuff there to swap space if it needs to do so. Ramfs is the filesystem that keeps things locked in memory: https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt With ramfs, there is no backing store. Files written into ramfs allocate dentries and page cache as usual, but there's nowhere to write them to. This means the pages are never marked clean, so they can't be freed by the VM when it's looking to recycle memory. ramfs and tmpfs: ---------------- One downside of ramfs is you can keep writing data into it until you fill up all memory, and the VM can't free it because the VM thinks that files should get written to backing store (rather than swap space), but ramfs hasn't got any backing store. Because of this, only root (or a trusted user) should be allowed write access to a ramfs mount. A ramfs derivative called tmpfs was created to add size limits, and the ability to write the data to swap space. Normal users can be allowed write access to tmpfs mounts. See Documentation/filesystems/tmpfs.txt for more information. permalink fedilink source parent hideshow 2 child comments replies: [β] Redjard@reddthat.com 1 point 1 month ago Maybe. It's been a while so I don't know 100% this was put to the test, but I wanna say the system has a weird kernel which leads to it not swapping out tmpfs properly. But ordinarily you should be right, this would simply ruin the stats visually until something forced it to swap out, since konsole shouldn't be accessing it. permalink fedilink source parent [β] Dave@lemmy.nz 1 point 1 month ago* (1 child) Could you have restarted to allow the OS to clean it up? permalink fedilink source parent hideshow 2 child comments replies: [β] Redjard@reddthat.com 7 points 1 month ago (1 child) I could have, but the system wasn't set up to restart without downtime, and the server was also remote and not easily accessible. It did acutally die due to a poweroutage some months later and took 2 days to get restarted. So yeah sometimes restarting is way more undesirable than loosing access to 32GB of ram. I would have just eaten that cost otherwise until a more opportune chance to restart. Besides, restarting to fix a problem is equivalent to giving up on understanding the issue, learning new stuff, and maybe finding a way better solution or preventing the type of error entirely. I get not finding the motivation when your software is working against you and learning is ultimately fruitless like on windows, or not having the time in the moment to figure it out properly, but a perfectly good bug on a linux system when you have time is prime real-estate to grow your skills and find fulfillment. permalink fedilink source parent hideshow 2 child comments replies: [β] Dave@lemmy.nz 1 point 1 month ago (1 child) Ah that makes sense. I had considered it might be a server but you mentioned a Konsole tab so my mind decided it must have been local machine. Crazy it took 2 days to restart the server! permalink fedilink source parent hideshow 2 child comments replies: [β] Redjard@reddthat.com 2 points 1 month ago There was a dedicated person on call, but it happened to be when they were away. The Konsole was left running from a local access, with a while true loop of a service status command. When that service was stopped later, the while loop started rerunning the script every second, filling the buffer with error messages. The tab was then killed remotely, but the Konsole window left running. Process ram usage went down but the file remained on tmpfs, which is not counted as ram usage so wasn't noticed. Then it took some time to notice the ram usage mismatch so noone thought of that konsole incident. permalink fedilink source parent
[β] tal@lemmy.today 2 points 1 month ago* (last edited 1 month ago) (1 child) I wrote this because my /tmp tmpfs was taking up 32GB of ram despite the files inside summing to only 3MB. Note that tmpfs doesn't force its contents to remain in memory the kernel can move stuff there to swap space if it needs to do so. Ramfs is the filesystem that keeps things locked in memory: https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt With ramfs, there is no backing store. Files written into ramfs allocate dentries and page cache as usual, but there's nowhere to write them to. This means the pages are never marked clean, so they can't be freed by the VM when it's looking to recycle memory. ramfs and tmpfs: ---------------- One downside of ramfs is you can keep writing data into it until you fill up all memory, and the VM can't free it because the VM thinks that files should get written to backing store (rather than swap space), but ramfs hasn't got any backing store. Because of this, only root (or a trusted user) should be allowed write access to a ramfs mount. A ramfs derivative called tmpfs was created to add size limits, and the ability to write the data to swap space. Normal users can be allowed write access to tmpfs mounts. See Documentation/filesystems/tmpfs.txt for more information. permalink fedilink source parent hideshow 2 child comments replies: [β] Redjard@reddthat.com 1 point 1 month ago Maybe. It's been a while so I don't know 100% this was put to the test, but I wanna say the system has a weird kernel which leads to it not swapping out tmpfs properly. But ordinarily you should be right, this would simply ruin the stats visually until something forced it to swap out, since konsole shouldn't be accessing it. permalink fedilink source parent
[β] Redjard@reddthat.com 1 point 1 month ago Maybe. It's been a while so I don't know 100% this was put to the test, but I wanna say the system has a weird kernel which leads to it not swapping out tmpfs properly. But ordinarily you should be right, this would simply ruin the stats visually until something forced it to swap out, since konsole shouldn't be accessing it. permalink fedilink source parent
[β] Dave@lemmy.nz 1 point 1 month ago* (1 child) Could you have restarted to allow the OS to clean it up? permalink fedilink source parent hideshow 2 child comments replies: [β] Redjard@reddthat.com 7 points 1 month ago (1 child) I could have, but the system wasn't set up to restart without downtime, and the server was also remote and not easily accessible. It did acutally die due to a poweroutage some months later and took 2 days to get restarted. So yeah sometimes restarting is way more undesirable than loosing access to 32GB of ram. I would have just eaten that cost otherwise until a more opportune chance to restart. Besides, restarting to fix a problem is equivalent to giving up on understanding the issue, learning new stuff, and maybe finding a way better solution or preventing the type of error entirely. I get not finding the motivation when your software is working against you and learning is ultimately fruitless like on windows, or not having the time in the moment to figure it out properly, but a perfectly good bug on a linux system when you have time is prime real-estate to grow your skills and find fulfillment. permalink fedilink source parent hideshow 2 child comments replies: [β] Dave@lemmy.nz 1 point 1 month ago (1 child) Ah that makes sense. I had considered it might be a server but you mentioned a Konsole tab so my mind decided it must have been local machine. Crazy it took 2 days to restart the server! permalink fedilink source parent hideshow 2 child comments replies: [β] Redjard@reddthat.com 2 points 1 month ago There was a dedicated person on call, but it happened to be when they were away. The Konsole was left running from a local access, with a while true loop of a service status command. When that service was stopped later, the while loop started rerunning the script every second, filling the buffer with error messages. The tab was then killed remotely, but the Konsole window left running. Process ram usage went down but the file remained on tmpfs, which is not counted as ram usage so wasn't noticed. Then it took some time to notice the ram usage mismatch so noone thought of that konsole incident. permalink fedilink source parent
[β] Redjard@reddthat.com 7 points 1 month ago (1 child) I could have, but the system wasn't set up to restart without downtime, and the server was also remote and not easily accessible. It did acutally die due to a poweroutage some months later and took 2 days to get restarted. So yeah sometimes restarting is way more undesirable than loosing access to 32GB of ram. I would have just eaten that cost otherwise until a more opportune chance to restart. Besides, restarting to fix a problem is equivalent to giving up on understanding the issue, learning new stuff, and maybe finding a way better solution or preventing the type of error entirely. I get not finding the motivation when your software is working against you and learning is ultimately fruitless like on windows, or not having the time in the moment to figure it out properly, but a perfectly good bug on a linux system when you have time is prime real-estate to grow your skills and find fulfillment. permalink fedilink source parent hideshow 2 child comments replies: [β] Dave@lemmy.nz 1 point 1 month ago (1 child) Ah that makes sense. I had considered it might be a server but you mentioned a Konsole tab so my mind decided it must have been local machine. Crazy it took 2 days to restart the server! permalink fedilink source parent hideshow 2 child comments replies: [β] Redjard@reddthat.com 2 points 1 month ago There was a dedicated person on call, but it happened to be when they were away. The Konsole was left running from a local access, with a while true loop of a service status command. When that service was stopped later, the while loop started rerunning the script every second, filling the buffer with error messages. The tab was then killed remotely, but the Konsole window left running. Process ram usage went down but the file remained on tmpfs, which is not counted as ram usage so wasn't noticed. Then it took some time to notice the ram usage mismatch so noone thought of that konsole incident. permalink fedilink source parent
[β] Dave@lemmy.nz 1 point 1 month ago (1 child) Ah that makes sense. I had considered it might be a server but you mentioned a Konsole tab so my mind decided it must have been local machine. Crazy it took 2 days to restart the server! permalink fedilink source parent hideshow 2 child comments replies: [β] Redjard@reddthat.com 2 points 1 month ago There was a dedicated person on call, but it happened to be when they were away. The Konsole was left running from a local access, with a while true loop of a service status command. When that service was stopped later, the while loop started rerunning the script every second, filling the buffer with error messages. The tab was then killed remotely, but the Konsole window left running. Process ram usage went down but the file remained on tmpfs, which is not counted as ram usage so wasn't noticed. Then it took some time to notice the ram usage mismatch so noone thought of that konsole incident. permalink fedilink source parent
[β] Redjard@reddthat.com 2 points 1 month ago There was a dedicated person on call, but it happened to be when they were away. The Konsole was left running from a local access, with a while true loop of a service status command. When that service was stopped later, the while loop started rerunning the script every second, filling the buffer with error messages. The tab was then killed remotely, but the Konsole window left running. Process ram usage went down but the file remained on tmpfs, which is not counted as ram usage so wasn't noticed. Then it took some time to notice the ram usage mismatch so noone thought of that konsole incident. permalink fedilink source parent
