[quotemstr]

> winpty

Cool. There's a third option that you might want to consider for winpty. (I was experimenting with it before I left Microsoft.)

The basic idea is to use a DLL shim to provide your own console API implementation for a process and its children --- you can replace the system console API with your own by shimming DLL import and export tables. You'd use your own type of pseudohandle for your fake console handles; programs written to work with the old console pseudohandles should work correctly with your pseudohandles, provided you've hooked the relevant HANDLE-using APIs.

[ryanprichard]

Yeah, I considered that route, but I was concerned that it'd be unreliable -- maybe with other programs doing clever tricks with import/export tables, or maybe with antivirus. I can't think of a specific reason it wouldn't work, though. I think it'd have to override CreateProcess to propagate the API hook. IIRC, ConEmu's CreateProcess starts a child in a suspended state so it can install its hook before resuming the child. Maybe it'd have to hook GetProcAddress, too.

I was assuming I'd use genuine console handles, but recognize that they're associated with the special console, so divert their API calls.

Maybe the technique would make programs slower. I know that's a complaint people have about ConEmu.

There's a similar (4th?) technique I've considered -- instead of hooking every process attached to the console, hook APIs in the conhost process and reimplement the internal protocol. It should avoid the performance problems and confine the hackiness to one address space. The trouble is that the protocol is undocumented. A small change could break everything without warning, but MS could also redesign the whole thing, making a fix impossible.

[quotemstr]

I also tried using the console accessibility APIs [1] to at least synchronize scraping and forwarding to the pty. The problem with this approach is that the console code issues EVENT_CONSOLE_UPDATE_REGION and such with its internal locks held, so attempting to read from the console from inside the accessibility handler deadlocks.

[1] https://msdn.microsoft.com/en-us/library/ms971319.aspx

[ryanprichard]

To do that, you have to attach the conhost.exe process to its own console with AttachConsole, right? I tried that and also noticed the deadlock. An out-of-context handler avoids the deadlock, but then it's not synchronized.

The other interesting problems I found with that API were: (1) when editing an input line, EVENT_CONSOLE_CARET notifications were delayed, (2) if I hit ESC when editing an input line, the line would clear, but there'd be no update-region notification, and (3) it's impossible to distinguish between a call to ScrollConsoleScreenBuffer and the whole buffer scrolling because we've reached the end. ScrollConsoleScreenBuffer might scroll only part of the buffer.

winpty doesn't use the WinEvents API (yet), but it might be good enough to reduce polling when the console is idle.