Yes, I always use `du . -h 2>/dev/null | sort -k1hr | head -nN` where that last N is however many directories I want to show. I run this regularly and have for a very long time precisely because of the issue that the post talks about.
I just ran it for the first time... I've run windirstat many times including in the last two weeks and yes, this is very fast. According to its timing it took 3.63 seconds to scan my drive... I'm not sure if it really took almost 3 minutes for windirstat, but it did take a long time, so it's plausible. More importantly, ~ 4 seconds is much easier to wait for.
windirstat currently doesn't take 3min to index 500k+ files on an 8x8tb drive raidz2 that's 90% full. When I stopped using windoratat 3ish years ago it took a few minutes to index the same array.
I think you might he using an old version, or have an issue with your storage.
I'm running it now... it is an old version 1.1.2.80, but it didn't tell me there's an update, and the functionality I need stayed the same, so why would I update?
It took almost 5 minutes and reports almost 2M items, 1.45M files and 465k subdirs...
I don't know that there's anything wrong with my disk, windirstat has always been slow when I ran it on my windows installation. Since it doesn't cheat, it goes slow. I will say, while it was reading, task manager showed a high read rate and a high write rate, and I dunno why, it's Windows (10); not that going through a convoluted filesystem is particularly fast anywhere else.
It is - it ‘cheats’ by reading off the NTFS MFT, similarly to another indispensable piece of Windows software Everything Search, both of which I miss keenly on my underprovisioned work Mac.
I recently ran `df` and was surprised to see that I'd used more than half of the two terabyte SSD in my laptop. So, being apparently atypically proactive, my first action was delete all of the target directories in my Rust projects and was equally surprised to see that it freed up nearly 10% of my disk usage (~200 GB)!
On a SanDisk portable SSD I own, the cluster size is even 4MiB. If you put a node modules directory in there with thousands of tiny files, it can "take up" several GiB of space.
This can be an important reason to enable compression (on some file systems it “solves” the problem) and why virtual machine file format can be important.
I think about this problem often in life because of my tendency to procrastinate. Only the long term effects that cause me serious paranoia get any real attention (failing to meet social expectations, major health issues, large-figure finance problems). It’s absolutely necessary to define the constraints before they define you. This requires a lot of forward thinking and effort but considering how little time we get on this earth… worth it
Mine is mostly full and it's not stuff I've saved but stuff software has stored there without asking me. It's quite hard to figure what's going on. I once got a message from Chrome saying it couldn't go on because it had stored like 200gb for websites, would I like to delete it all, and I said yes and noticed no difference.
It's annoying because I'd rather store say family photos than have it full of random junk that means nothing to me. Dunno the answer - better os control? Partition the drive?
It's not just hard drive problem. If you buy a new storage rack at your home, it will be full in no time. No matter how many racks you buy, you will always find things fill those racks. This is like rain water filling up the lakes. if there is a lake, it gets filled. Or maybe it is called a lake because it gets filled. So, one can say that the disks create or cause the data. If there is no disk, there is no data.
If you're on MacOS and suspect a lot of space is taken by multiple copies of the same files, use fclones to turn them into APFS clones. (You can also use this on Linux, as ext4 supports hard links, but only if you're sure the duplicates won't diverge.)
I have less than one terabyte of disk space, and less than 10% of the full capacity is in use, so it is closer to being empty. (The hard drives they sell have more capacity than I require.) And, there are files that I do not use (some of which come with the operating system, such as PulseAudio).
It is so wild that someone would let Claude index all of the files on their computers for such a bane request. We are really turning into the people on WALL·E
> 20+ years of software engineering experience across full-stack, backend, infrastructure, Site Reliability Enigneering, and AI engineering. Past work includes Uber, Airbnb, Canva, and a wide variety of startups - several of which I co-founded.
> I'd never have never found these myself.
You are either the most experienced most incompetent software engineer in the world, or you are lying.
And i am wondering what do people even fill their storage with, and why do they need so much. In my case it is ~200-300 out of 1TB on 2 laptops, 37 out of 128gb on phone. And i don't even actively manage it.
I have one single video game I play that takes up around 650GB, so I have a dedicated drive for it. And between 1 and 2 TB of movies, TV, and photos. Everything else is ephemeral/scratch space for whatever I'm working on.
I used to be into amateur movie making and editing, and back then I needed many TB of space for the stuff I pulled off my cameras before it got edited down.
I don't know what it is for GP, but in my case it's Flight Simulator 2020 with over 700 GB. Not the standard installation though: it's community mods, mostly photogrammetric data for additional realism. There's a whole ecosystem around the base game that could easily fill several TB if you go all in.
I have 2TB filled and I need more. I make music so I need a lot of storage for my instruments and samples. This is measured in tens and often hundreds of gigabytes. I also play games. Games' sizes range wildly from 10 to 200 GBs. I also do gamedev. The textures get big and get duplicated between projects. Programming (npm modules for example), add to this personal data, media archives and 2TB is not enough
The SSD firmware is smart enough to rotate the hot sectors over the entire space gradually.
It is a problem to have a full SSD from a performance perspective. Modern SSDs use pSLC (psuedo-single-level-cells) because they can write much faster. When you write to an SSD, you generally write in pSLC mode first, and once the data goes cold, it will get written back to MLC/TLC storage.
Once you are very full, pSLC gets disabled or becomes less effective, affecting write speeds.
There is also an effect known as write amplification that is exacerbated when a disk is nearly full. You want to write 4kb, but the available erasure zones are all full (either valid or invalidated sectors), so now the SSD has to erase a 4MB zone and compact the data there (along with your new 4kb) to write your 4kb.
This gets even worse when you have a filesystem that isn't SSD optimized, such as EXT4. It will cause further write amplication.
Keep your SSD's <80% full if you want good performance.
Slack has value, though. Your full disk is likely to be very fragmented, making file access slow. A CPU running a full throttle cannot deal with spikes in demand. full RAM means less cache space available. And these issues get asymptotically worse as you approach 100% utilization (which can have catastrophic effects in organizations which are too heavily optimized for high utilization).
Delete is expensive. "Am I positive I'll never need this again?"
As a consequence, "cheap" actions (saves) accumulate, "expensive" ones (deletes) accrue until cheap becomes expensive as space becomes tight.
Ultimately, options become:
- Find criteria for rapidly assessing what data can be deleted.
- Purchase cheap archival storage and migrate old data to that.
- A media or device crash induces loss of arbitrarily-selected data.