[alex7734]

> When we free memory, we should make sure that if the block we return to the free list is next to any other free blocks, we combine them together. This is called "coalescing."

A little offtopic but the default Delphi 7 memory allocator did this, except that it also merged blocks that it obtained from different OS allocation calls.

This worked fine for regular usage, but if that memory was ever used for Bitmaps for UI stuff, it wouldn't work: Since Windows does some of its UI stuff in kernel mode, before doing anything Windows would attempt to lock the entire allocation's VAD entry to prevent you from messing with it in another thread while it was using it. If the Bitmap you were working on happened to belong to two different OS-level allocations, this lock function would fail since it wasn't meant to handle that case.

This would lead to random, extremely cryptic errors such as ERROR_NO_SCROLLBARS "The window does not have scroll bars." since the lock call was deep in the stack and the callers replaced the error with another one as it bubbled up.

In the end we had to replace the allocator to avoid that issue. That was a fun day I spent debugging that.

[meekaaku]

You mean a bug this deep took one day to debug?

[alex7734]

Yes and no, it took me from 8am to 3am once we decided it needed to get fixed but really it sat on the app for years, it only happened on a background process that sent print jobs on a timer, since it used Windows GDI to compose the image we sent to the printer it was affected (our "frontend" should've been affected too but never was, I guess because it had a different memory usage pattern).

We just had it restart itself and try again whenever it got one of those errors when printing but eventually we wanted to add a feature that required the process to not die, and by that time I was already 99% sure that it wasn't something in our code and I had already ruled out threading issues.

I ended up putting it in a VM with a kernel debugger attached and having a script make a snapshot and make it print over and over until it errored, then following along in IDA until I saw what was going on.

Having a way to trigger it (by restoring the snapshot) on demand helped a lot, otherwise it would have taken forever to make sense of it as it could sit without crashing for nearly an hour.

[zeusk]

How do you attach kd to VM?

While I was at MS, it was such a big PITA - we just had a bunch of IT managed machines with KVM console access and KDNET for debugging.

[EvanAnderson]

In Hyper-V it's fairly easy. You make a virtual serial port ("COMPort"), set the bootloader to enable kernel debugging over serial, then connect to the virtual serial port from the host via a named pipe.

https://learn.microsoft.com/en-us/windows-hardware/drivers/d...

I haven't tried it with vSphere but I suspect it'd be similar.

[jasomill]

Kernel debugging over serial should be possible in vSphere, but Ethernet is easier to set up:

1. Make sure at least one virtual NIC on the target VM (with IP connectivity to the debug host machine/VM) is on Microsoft's NIC whitelist[1]. I use e1000e; note that vmxnet3 is not on the list.

2. Follow Microsoft's directions[2] to connect.

I can confirm this works on vSphere[3] and there's no reason it shouldn't also work on VMware Workstation, Player, and Fusion.

[1] https://learn.microsoft.com/en-us/windows-hardware/drivers/d...

[2] https://learn.microsoft.com/en-us/windows-hardware/drivers/d...

[3] Tested last week with a Windows Server 2022 target (e1000e virtual NIC) and Windows 10 debug host (vmxnet3 virtual NIC), both running on ESXi 8.0 Update 1 VM hosts.

[vlovich123]

Delphi 7 was 2002. Was hyper-v around back then? Or is this just a legacy app still running?

[EvanAnderson]

Hyper-V wasn't until Windows Server 2008. I know you could do virtual serial ports w/ VMware GSX and ESX (and later ESXi) forwarded to real hardware serial ports on the host.

[alex7734]

It's a legacy app

[bee_rider]

Owch, I’m guessing that wasn’t -5 hours of debugging.

[cozzyd]

It was actually 250 hours

[ilyt]

I think the far weirder part of this was the kernel-side handling of scrollbars

[alex7734]

If I recall correctly the kernel part of things would return an out of memory error which the user mode DLL translated to that weird error (sometimes, other times it would just say "out of system resources", it depended on what widget the bitmap that overlapped the two memory regions belonged to).

Here's a 2003 forum post from someone else having the same problem: http://www.delphigroups.info/2/1/749064.html

[derefr]

Until Windows 95, Windows was essentially just a DOS application that grabbed the framebuffer and ran an event loop where it drew "controls" (which includes windows, buttons, text views, and yes, scrollbars.) That was the whole point of it. It wasn't an "OS" per se; DOS was the OS. Windows was what a Linux-head would think of as a combination of an X server and window manager. And Windows loaded your "application" as essentially a DLL, with the Windows global event loop calling into your application's event-loop delegate handler (WndProc) whenever it has an interesting event that your application might like to react to.

(Your application wasn't even a "process" per se. Until Windows 95, everything was just happening in one shared address space, in real mode. In fact, it was only in Windows 3.1 where user applications stopped running in ring 0!)

If you think about it, this "the kernel is a game engine and your application is the game" approach isn't necessarily a bad design... for a single-tasking OS's library exokernel, like the Wii's https://wiibrew.org/wiki/IOS.

But, of course, Windows claimed to be a multitasking OS. But it actually wasn't! And I don't mean the obvious thing about it not having pre-emption. Lots of multitasking OSes didn't have pre-emption.

No, what I mean is that the concurrency primitive for the cooperative scheduling wasn't the "task" (i.e. the process or thread. Which, again, there weren't any of.) Instead, the concurrency primitive was the window. Until Windows 95, Windows was a multi-windowing OS.

Each control was owned by a window. Each window had a WndProc. If your Windows executable (i.e. application delegate module) set up two windows, then each window participated separately in the global Windows event loop, up-to-and-including things like having its own set of loaded fonts, its own clipboard state, and its own interned strings table. In OOP terms†, your application was just a dead "class object", running no logic of its own save for one-time load and unload hooks. It was the windows themselves that were the "instances" of your class.

This might make you realize why MDI (or Multiple Document Interface, where there are multiple small per-document "windows" inside one big window) was so popular back then. The MDI "windows" weren't actually windows — they didn't have their own WndProc. They were just controls, like a tab view is a control. Only the big container window was a real window, and so all the resources within that big window were shared between all the virtual windows. MDI was a memory-saving trick!

---

† The actual more interesting analogy is that Windows was essentially a (single-threaded, cooperatively-scheduled) actor system, where windows were the actors. There is a very close parallel between (Windows 3.1 executables, Windows 3.1 windows) and (Erlang modules, Erlang processes).

[Stratoscope]

> This might make you realize why MDI (or Multiple Document Interface, where there are multiple small per-document "windows" inside one big window) was so popular back then. The MDI "windows" weren't actually windows — they didn't have their own WndProc. They were just controls, like a tab view is a control. Only the big container window was a real window, and so all the resources within that big window were shared between all the virtual windows. MDI was a memory-saving trick!

MDI may have saved some memory - I can't say one way or the other on that - but the mechanism you describe is incorrect.

Every MDI child window was a window of its own with its own WndProc. Every control inside those windows was also a window with its own WndProc. Every dialog box was also - yes - a window with its own WndProc.

You wouldn't always be aware of the WndProc in your code, but it was there.

If you ran WinSight or Spy++, you could see the entire window hierarchy including all the child windows, child control windows, and so on.

Later on, a few applications implemented "windowless controls" to save memory, but this was uncommon, especially in the early days. For example, there was an optional windowless version of the Rich Edit control:

https://learn.microsoft.com/en-us/windows/win32/controls/win...

Fun fact: an early informal name for MDI was "Mac in a box", because if you maximized the top level window, you had a somewhat Mac-like environment, with one menu bar at the top that was shared by the child windows.

Source: I was the author of WinSight and the VBX (Visual Basic eXtension) API.

[derefr]

Interesting; through the fog of time, I may have misremembered some "tip" I was given for Windows 3.1 programming efficiency, as being a more definitive statement about the internal structure of Windows than it really was.

That tip, as I recall it, was that the developer should minimize the number of top-level windows they create, because each top-level window in the system gets opted automatically into various things that regular controls don't (including having a bunch of default child controls that probably keep a lot of state and pump a lot of messages.) But MDI child windows don't have the same window-class as top-level windows, and are instead pruned down to be much "lighter", both in doing things like:

- not having a some child controls (e.g. a menu bar) by default (you're supposed to attach the menu bar to the MDI frame instead, and change it to reflect the menu of the active child — as you say, it's "a Mac in a box" behavior);

- implementing behaviors that seem like their own child controls, but which are actually parts of the MDI child's own geometry, to reduce the number of event loops that need to be pumped. (I believe an MDI child's default-styled title bar might be this way, only becoming a full-on child control tree if the MDI child is given non-default styles.)

- "Sharing" child controls, where the control (I think an MDI child window's caption buttons might be this way?)

- routing messages to only the active child, with the others essentially frozen except when they need to redraw

Digging around, evidence to corroborate this is pretty scant, but there is some:

- https://jeffpar.github.io/kbarchive/kb/095/Q95578/ is a bug that is evidence of the MDI child window class having most of its behaviors programmed separately, rather than there being any code reuse between top-level windows and MDI child windows

- https://github.com/TransmissionZero/Win16-MDI-Application/bl... shows that MDI child windows have to be created through a special handler in the MDI frame (WM_MDICREATE); you can't just create regular windows with the MDI frame as their parent

[Stratoscope]

Thanks for the interesting discussion. Yes, the fog of time affects us all.

Funny you mention caption buttons (min/max/close/etc.) and menus and such. I meant to add a "Fun fact #2" in my first comment. So here we go...

In traditional Windows applications, none of those are child windows (or child controls, same thing). They are all part of the "non-client area". You may recall there being a whole series of WM_NC... messages like WM_NCMOUSEMOVE and WM_NCPAINT, with the same names as your usual messages except for the NC prefix. Your WndProc would receive all of these messages, but generally would just pass them along to DefWindowProc which handled this "non-client" activity.

Now here is the fun fact. OS/2 Presentation Manager took a different and cleaner approach. It removed the "non-client area" concept entirely, along with all those messages. Instead, all of those window doo-dads were child windows. The minimize button was a child window. Your "client area" was a child window. And so on. It was all child windows.

On this point:

> implementing behaviors that seem like their own child controls, but which are actually parts of the MDI child's own geometry, to reduce the number of event loops that need to be pumped.

To be clear, a typical application had only one event loop. It was your classic GetMessage/TranslateMessage/DispatchMessage loop. DispatchMessage would pass the message off to whatever WndProc it should be directed to.

[AshamedCaptain]

> Until Windows 95, everything was just happening in one shared address space, in real mode. In fact, it was only in Windows 3.1 where user applications stopped running in ring 0!

Windows 3.0 and predecessors runs in processor which had no concept of "ring 0", so that should not be surprising at all...

> Your application wasn't even a "process" per se

I think this is a bit of a "modernistic", "win32y" view of the definition of a process. Surely there are processes/tasks in 3.x -- you can launch multiple instances of a module, each of them can allocate memory separately, each of them have different resources/handles, and the OS will cleanup for you once each instance terminates (cleanly). Obviously, without any type of virtual address space or memory protection, any such process can write to and destroy the memory of any other process, but they are still processes. The existence of Yield/DirectedYield , which do not take/need a window, is also a hint of that. (Note there are no threads in 3.x).

Many platforms (that either predate VM or decide not to use VM for e.g. power usage concerns) worked like this. MacOS, Windows CE, PalmOS, etc.

[derefr]

> you can launch multiple instances of a module, each of them can allocate memory separately

I don't think this is true, though? You're not getting separate processes; you're just getting separate hInstances. Which don't map cleanly to a process-like abstraction.

Consider: while you can (in theory) spawn multiple copies of a Win16 executable, with each spawn getting its own hInstance and therefore its own locals heap, this isn't an inherent part of Win16's architecture, but rather is something specific to its handling of spawning executables. DLLs weren't included in this handling, and so only get one hInstance+heap each. This means that if you load and call into the same DLL from two separate actively-running Win16 programs, then that DLL must juggle any state it wants to keep for its N active callers on its own single heap. A function call crossing semantically-distinct memory protection domains, without any kind of IPC serialization, is not very "process"-y. It's more of an object system, like the JVM.

(In both Windows and the JVM: each "object" has its own private heap, and a module wrapping access to that heap; and some "objects" additionally have their own concurrent execution thread. But a given execution thread isn't "bottled up in" a particular heap; it just has a "home" heap. If an execution thread calls another object's API, then that execution thread, through that API, manipulates that other object's heap. There's no concept of IPC — of manipulations of other objects' heaps requiring you to ask the other object whose execution thread owns that heap to do the manipulation for you.)

> The existence of Yield/DirectedYield , which do not take/need a window, is also a hint of that.

DirectedYield is an attempt to make Windows tasks seem to act like "processes"... in the Communicating Sequential Processes sense, at least.

But it doesn't really accomplish this. From the Windows 3.1 API Reference Guide:

> If an application uses DirectedYield for a task with no events scheduled, the task will not be executed. Instead, Windows searches the task queue. In some cases, however, you may want the application to force a specific task to be scheduled. The application can do this by calling the PostAppMessage function, specifying a WM_NULL message identifier. Then, when the application calls DirectedYield, the scheduler will run the task regardless of the task's event status.

In other words, what's really acting as CSPs here, are the windows. :)

[mananaysiempre]

> That was the whole point of it. It wasn't an "OS" per se; DOS was the OS. Windows was what a Linux-head would think of as a combination of an X server and window manager.

This seems like a very post-3.0 (i.e. 386-only) view of things—the 8086 and 286 versions of Windows were also fairly advanced memory allocator/overlay manager hybrids.

They parcelled out memory, compacted it to avoid external fragmentation (take that, dlmalloc!), expelled pieces that could be read back from executables, and discarded segments that the application programmer said could be recovered, as necessary. (Yet they couldn’t actually swap mutable memory, as far as I can tell. Why?) For data, they required your cooperation by only revealing addresses between *Lock and *Unlock calls and requiring you to store handle+offset pairs otherwise; for code, the 8086 kernel would reach into your stack and walk the frame pointer chains in order to redirect return addresses to swap-in thunks. (Maintaining LRU lists along the way in either case.) Things became better on the 286 when it could just hand out segment selectors and arrange for accesses to fault as required, but this is the problem statement that Windows 1.0, running on the 8086, set out for itself.

Now, none of this is impossibly difficult (although I shudder at the thought of doing it without good debugging tools), but it feels pretty damn OS-like to me. You might argue there isn’t much virtualization of hardware going on—except for RAM, CPU, display, keyboard, and mouse—but I’d say there is at least as much of it in Win16 as there is in DOS.

One would hope things would get easier on the 386. And then one gets to DPMI and VDMs and still wants to support 16-bit drivers that hooked BIOS calls as though that would help and now the system cannot interact with the user while it’s formatting a floppy[1].

> [T]he concurrency primitive wasn’t the “task” [but instead] the window. Until Windows 95, Windows was a multi-windowing OS. [... E]ach window participated separately in the global Windows event loop, up-to-and-including things like having its own set of loaded fonts, its own clipboard state, and its own interned strings table. In OOP terms†, your application was just a dead "class object", running no logic of its own save for one-time load and unload hooks.

That... doesn’t sound correct on the implementation level. If you’re an instance of a Win16 executable module, you own a copy of your executable image’s mutable data, you own your memory allocations (that are not marked GMEM_SHARE), you own a stack, you get all (“posted”, i.e. asynchronous) messages for all windows you (or your library dependencies) created and you ask the system to munge and route them to the message dispatch routines of the windows you created—or not, if you don’t want to.

Now, the overall effect is very much like what you described, and often it may feel that you could as well throw out all those tedious GetMessage-TranslateMessage-DispatchMessage loops and replace them with a standard implementation of a vat[2]. Then a puny message handler decides to hijack the whole thing and go into a modal dialog loop. And damned if I could describe what that means in terms of an object model.

(What would you say about Symbian? Now there’s a system that throws out processes and only runs objects. Except it doesn’t run inside of a DOS or anything; there’s a kernel and on top of that there are objects. Boom. ... I think?)

> The actual more interesting analogy is that Windows was essentially a (single-threaded, cooperatively-scheduled) actor system, where windows were the actors.

Yeah, that I’ll wholeheartedly agree with. Nevermind the drawing part, it even has separate synchronous sends (SendMessage) and asynchronous ones (PostMessage)! Of course, unlike E’s[3], these have completely insane interactions with the concurrency parts, especially once you get to Win32.

[1] http://bytepointer.com/resources/old_new_thing/20090102_002_...

[2] http://www.erights.org/elib/concurrency/vat.html

[3] http://www.erights.org/elib/concurrency/msg-passing.html