[for_xyz]

> the bottleneck is that NTFS is a crappy filesystem.

Care to explain why do you think NTFS is crap?

From my experience it's usually bad assumptions about files under windows and every application or library tries to stick to posix interface when dealing with files (open, read/write, close) which tends to block for longer periods on windows than on linux counterparts which results in significant perfomance loss .

Linux first software will always outperform windows implementation and Windows first software will outperform Linux implementations unless you provide separate code paths to properly handle underlying OS architecture and assumptions.

On Windows closing the file handle is extremely costly operation due AV checks and file content indexing [1]

[1] https://www.youtube.com/watch?v=qbKGw8MQ0i8

[jmnicolas]

> Care to explain why do you think NTFS is crap?

I don't know if it's crap but it's much much slower than EXT4.

I remember reading a comment here that Windows in a VM on a Linux host was faster than bare metal.

Probably not true but I decided to make a test. I have a .net core app that insert data in a sqlite db (the resulting db is about 300 GB).

So I benchmarked this app on Linux (it was previously running on Windows) and IIRC it ran about 4 times faster.

[eitland]

An interesting observation I did was that even Hello World is about 100x faster on Linux than on Windows.

In my Linux VM I had to use the time command to even have an idea of how long it took, as it seemed to return immediately. I think 5-15ms but it already a while ago.

On the Windows machine where the Linux VM ran it took several hundred ms.

[for_xyz]

Which VM software did you use under windows?

[eitland]

Probably VirtualBox, could have been Hyper-V.

[m_mueller]

so what's the 'Windows way' then? keeping a file open forever and let it lock out other programs from using that file? I'm starting to get an idea why Windows is getting on my nerves so much with file locks....

[pjmlp]

Here is the thing, UNIX is the strange dude with advisory file locks, every other multiuser OS does the right thing to avoid file corruption.

[for_xyz]

> so what's the 'Windows way' then?

Using either memory mapped files of overlapped io (IOCP).

It's tricky to use when you want to write the content since you must preallocate the file before you start with the writing. Appending to file just doesn't work under NT kernel since WriteFile blocks even if you use overlapped io.

Devs just need different mentality when it comes to Windows programming compared to Linux. Due the fact that everything under NT kernel is operated asynchronously you'll have to adapt your code to such concept. Meanwhile under Linux you had no other alternative for nearly 30 years (io_uring and friends) so if you wanted to be portable with minimum OS specific code then you had to implement things in synchronous way or write two separate code paths for each OS.

Guess which one is used in practice.

[mehrdadn]

You can just use FILE_SHARE_WRITE if you don't want to lock out other programs...

[VBprogrammer]

If I know one thing about files it's that any fix beginning with "You can just..." is probably wrong.

https://danluu.com/file-consistency/

[mehrdadn]

I didn't recommend you do this, I just said it's there if you really want to.