[mghackerlady]

Has anyone thought about making the linux kernel roughly compatible with NT? Like how FreeBSD is compatible with Linux? I know it'd definitely be harder as NT is proprietary but syscalls (in my very uninformed opinion) seem all that difficult to implement, even without a userland

[augusto-moura]

At what level do you mean that? Kernel level? Driver level?

Wine[1] is the de facto compatibility layer with NT executables. Driver compatibility is too complex and obscure to worth the while. Often information is undocumented or hard to get.

There are a few implementations of windows behaviors at kernel level for a few subsystems features, ntsync, samba, ntfs, etc. they can be used by wine to improve compatibility or performance

[1]: https://www.winehq.org/

[treyd]

FreeBSD is not "compatible with Linux", it provides a way to run Linux applications under a Linux-like syscall environment. What you're suggesting is as if you could load Linux kernel modules into the FreeBSD kernel.

The issue with NT is the driver ecosystem. You'd have to reimplement a lot of under-documented NT behavior for NT drivers to behave themselves, and making that work within the Linux kernel would require major architectural changes. The userland is also where a lot of magic happens for application compatibility.

[BoredomIsFun]

> What you're suggesting is as if you could load Linux kernel modules into the FreeBSD kernel.

Afaik, you partially can.

[bryanlarsen]

It's far from "roughly compatible with NT", but the Linux kernel does accept changes to make supporting Windows applications more efficient.

example: ntsync

[jchw]

Me as a kid thought this would be a great idea, and started implementing a PE binfmt. I actually did make a rudimentary PE binfmt, though it started to occur to me how different Windows and Linux really were as I progressed.

For example, with ELF/UNIX, the basic ELF binfmt is barely any more complex than what you'd probably expect the a.out binfmt to be: it maps sections into memory and then executes. Dynamic linking isn't implemented; instead, similar to the interpreter of a shell script, an ELF binary can have an interpreter (PT_INTERP) which is loaded in lieu of the actual binary. This way, the PT_INTERP can be set to the well-known path of the dynamic linker of your libc, which itself is a static ELF binary. It is executed with the appropriate arguments loaded onto the stack and the dynamic linker starts loading the actual binary and its dependencies.

Windows is totally different here. I mean, as far as I know, the dynamic linker is still in userland, known as the Windows Loader. However, the barrier between the userland and kernel land is not stable for Windows NT. Syscall numbers can change during major updates. And, sometimes, implementation details are split between the kernel and userland. Now, in order to be able to contribute to Wine and other projects, I've had to be very careful how I discover how Windows internals works, often by reading other's writings and doing careful black box analysis (for some of this I have work I can show to show how I figured it out.) But for example, the PEB/TIB structures that store information about processes/threads seems to be something that both the userland and kernel components both read and modify. For dynamic linking in particular, there are some linked lists in the PEB that store the modules loaded into the process, and I believe these are used by both the Windows loader and the kernel in some cases.

The Windows NT kernel also just takes on a lot more responsibilities. For example, input. I can literally identify some of the syscalls that go into input handling and observe how they change behavior depending on the last result of PeekMessage. The kernel also appears to be the part of the system that handles event coalescing and priority. It's nothing absurd (the Wine project has already figured out how a lot of this works) but it is a Huge difference from Linux where there's no concept of "messages" and probably shouldn't be.

So the equivalent of the Windows NT kernel services would often be more appropriate to put in userland on Linux anyways, and Wine already does that.

It would still be interesting to attempt to get a Windows XP userland to boot directly on a Linux kernel, but I don't think you'd ever end up with anything that could ever be upstreamed :)

Maybe we should do the PE binfmt though. I am no longer a fan of ELF with it's symbol conflicts and whatnot. Let's make Linux PE-based so we can finally get icons for binaries without needing to append a filesystem to the end of it :)

[direwolf20]

You can already use binfmt_misc to instruct the kernel to execute PE binaries with Wine.

[jchw]

I mean something a bit different. I mean using PE binaries to store Linux programs, no Wine loader.

Of course, this is a little silly. It would require massively rethinking many aspects of the Linux userland, like the libc design. However, I honestly would be OK with this future. I don't really care that much for ELF or its consequences, and there are PE32+ binaries all over the place anyways, so may as well embrace it. Linux itself is often a PE32+ binary, for the sake of EFI stub/UKI.

(You could also implement this with binfmt_misc, sure, but then you'd still need at least an ELF binary for the init binary and/or a loader.)

(edit: But like I said, it's a little silly. It breaks all kinds of shit. Symbol interposition stops working. The libdl API breaks. You can't LD_PRELOAD. The libpthread trick to make errno a thread local breaks. Etc, etc.)

[direwolf20]

Wine has no problem loading Linux programs in PE format. It doesn't enforce that you actually call any Windows functions and it doesn't stop you making Linux system calls directly.

[jchw]

Well yes, but you'd be spawning a wineserver and running wineboot and all kinds of baggage on top, all for the very simple task of mapping and executing a PE binary, and of course you would still wind up needing ELF... for the Wine loader and all of the dependencies that it has (like a libc, though you could maybe use a statically-linked musl or something to try to minimize it.)

Meanwhile the actual process of loading a PE binary is relatively trivial. It's trivial enough that it has been implemented numerous times in different forms by many people. Hell, I've done it numerous times myself, once for game hacking and once in pure Go[1] as a stubborn workaround for another problem.

Importing an entire Wine install, or even putting the effort into stripping Wine down for this purpose, seems silly.

But I suppose the entire premise is a little silly to begin with, so I guess it's not that unreasonable, it's just not what I am imagining. I'm imagining a Linux userland with simply no ELF at all.

[1]: https://github.com/jchv/go-winloader - though it doesn't do linking recursively, since for this particular problem simply calling LoadLibrary is good enough.

[anthk]

TBH the Wine loader and the Win32 subsystem for NT aren't that different design wise.

[saghm]

I recently learned that Windows binaries contain metadata for what version they are (among other things, presumably). I was discussing in-progress work on making a mod manager for a popular game work on Linux with the author of the tool, and they mentioned that one of the things that surprised them was not being able to rely on inspection of a native library used by most mods to determine what version they had installed on Linux like they could on Windows. It had never occurred to them that this wasn't a first-class feature of Linux binary formats, and I was equally surprised to find out that it was a thing on Windows given that I haven't regularly used Windows since before I really had much of a concept of what "metadata in a binary format" would even mean.

[1718627440]

Are you talking about the "Linux version" it targets or the version of the library? If its the latter, then it is the case, that versioning works per symbol instead of per library, so that a newer library can still contain the old symbols. If you want the latest version a library implements, you could search all symbols and look for the newest symbol version.

If you want it the other way around you could look at the newest symbol the library wants.

[saghm]

I probably could be more clear about what I'm trying to convey. Tool A is written to manage mods for game B, and lots of mods for that game utilize library C. Tool A does not directly load or link to library C, but it does inspect the version of library C that currently exists alongside game B so that it can detect if mods depend on a newer version of it and notify the user that it needs to be updated.

I'm realizing now that I forgot an important detail in all of this: the metadata of the library existed as part of the metadata that the filesystem itself tracked rather than something in the contents of the file itself. This metadata doesn't seem to exist on Linux (which library C only supports if running via Proton rather than any native Linux version of the game). I could imagine it might be possible for this to be set as some sort of extended attribute on a Unix filesystem, but in practice it seems that the library will have this extended filesystem metadata when downloading the DLL onto a Windows machine (presumably with an NTFS filesystem) but not a Linux one.

[1718627440]

> so that it can detect if mods depend on a newer version of it and notify the user that it needs to be updated.

The dynamic linker will literally tell you this, if you ask it.

> the metadata of the library existed as part of the metadata that the filesystem itself tracked rather than something in the contents of the file itself.

So does this metadata has anything to do with the file at all, or could I also attach it to e.g. an MP4 file? If that is the case than the difference is that the distributor for MS Windows did add the attribute and the distributor for GNU/Linux did not, it doesn't have anything to do with the platform.

EDIT:

> (which library C only supports if running via Proton rather than any native Linux version of the game

So library C isn't even an ELF shared object, but a PE/COFF DLL? Then that complaint makes even less sense.

[jamesfinlayson]

I assume it's the ability to tag a .dll as version 0.0.0.1 or whatever (it shows up under the file name in Windows Explorer). I think company name is another one that Windows Explorer displays but there are probably a few other supported attributes as well.

[1718627440]

Well, then the answer is that shared objects on GNU/Linux do not have a finite version, they just implement them. This is because they are expected to implement more than one version, so that you can just update the shared object, and it will still work with programs that expect to call the old ABI. You can however get the latest version. This is also what is in the filename. Note, that there are two versioning schemes: semver and libtool, they mean the same, but are notated differently. This version does not does not necessary equal the version of the project/package, APIs/ABIs can and do have their own version numbers.

[jchw]

I think this is just solving a different but related problem. Symbol versioning enables you to make observable changes to the behavior of some library like a libc while providing backwards compatibility by providing the old behaviors indefinitely. I've never gotten the impression that it is "prescribed" that shared objects on a Linux system don't have a specific/definite version, and I don't think that symbol versioning is necessarily appropriate for all programs, either-it just happens to be a mechanism used by glibc, and I don't think it is very common outside of glibc (could be wrong.)

On the other hand, the Windows NE/PE VERSIONINFO resource is mostly informational in nature. I think early on in the life of Windows, it was often used to determine whether or not the system library shipped by an installer was newer than the installed version, so that program installers could safely update the system libraries when necessary without accidentally overwriting new versions of things with older versions. That's just simply a problem that ELF shared objects don't try to solve, I reckon, because there aren't usually canonical binaries to install anyways, and the problem of distributing system libraries is usually left up to the OS vendor.

Actually though, there is another mechanism that Linux shared objects often use for versioning and ABI compatibility: the filename/soname. Where you might have a system library, libwacom.so.9.0.0, but then you also have symlinks to it from libwacom.so.9 and libwacom.so, and the soname is set to libwacom.so.9.

There is, of course, no specific reason why ELF binaries don't contain metadata, and you could certainly extend ELF to contain PE-style resources if you want. (I remember some attempts at this existed when I was still getting into Linux.) It may be considered "bad practice" to check the runtime version of a library or program before doing something, in many cases, but it is still relatively common practice; even though there's no standard way to do it, a lot of shared objects do export a library function that returns the runtime version number, like libpng's png_access_version_number. Since distributions patch libraries and maybe even use drop-in replacements, there is obviously no guarantee as to what this version number entails, but often it does entail a promise that the ABI should be compatible with the version that is returned. (There is obviously not necessarily a canonical "ABI" either, but in most cases if you are dealing with two completely incompatible system/compiler ABIs there isn't much that can be done anyways, so it's probably OK to ignore this case.)

So I think the most "standard" way to embed the "version" of a shared object on Linux is to export a symbol that either returns the version number or points to it, but there is no direct equivalent to the PE VERSIONINFO standard, which may be a pain when the version number would be useful for something, but the developers did not think to explicitly add it to the Linux port of their software, since it is often not used by the software itself.

But I personally wouldn't agree with the conclusion that "shared objects on GNU/Linux do not have a finite version" as a matter of course; I think the more conservative thing to say is that there is not necessarily a definite version, and there is not necessarily a "canonical" binary. In practice, though, if your copy of libSDL is an almost-unmodified build of the source tree at a given version 3.0.0, it is entirely reasonable to say that it is "definitely" libSDL 3.0.0, even though it may not refer to an exact byte-for-byte source tree or binary file.

[dmitrygr]

That would require (among many other things) a stable driver API -- one of the things NT gets right and linux is wrong on. Linus has been quite clear that he does not see things this way. So ... not going to happen