Hacker News new | ask | show | jobs
by Lariscus 193 days ago
It also buffers the downloaded data completely into memory last time I checked. So downloading a file bigger than the available RAM just doesn't work and you have to use WebClient instead.

Another fun one is Extract-Archive which is painfully slow while using the System.IO.Compression.ZipFile CLR type directly is reasonably fast. Powershell is really a head scratcher sometimes.

3 comments

The download being cached in RAM kind of makes sense, curl will do the same (up to a point) if the output stream is slower than the download itself. For a scripting language, I think it makes sense. Microsoft deciding to alias wget to Invoke-WebRequest does make for a rather annoying side effect, but perhaps it was to be expected as all of their aliases for GNU tools are poor replacements.

I tried to look into the whole Expand-Archive thing, but as of https://github.com/PowerShell/Microsoft.PowerShell.Archive/c... I can't even find the Expand-Archive cmdlet source code anymore. The archive files themselves seem to have "expand" be unimplemented. Unless they moved the expand command to another repo for some reason, it looks like the entire command will disappear at one point?

Still, it does look like Expand-Archive was using the plain old System.IO.Compression library for its file I/O, though, although there is a bit of pre-processing to validate paths existing and such, that may take a while.

> curl will do the same (up to a point) if the output stream is slower than the download itself

That "up to a point" is crucial. Storing chunks in memory up to some max size as you wait for them to be written to disk makes complete sense. Buffering the entire download in memory before writing to disk at the end doesn't make sense at all.

curl's approach will lead to partial and failed downloads. When a client stops accepting new data, servers tend to close the connection after a while.

There are smoother ways to deal with this (i.e. reduce download rate by faking dropped packets to output speed), but if you just want a simple download command, I think both simple solutions are fine.

If the download doesn't fit in RAM, it'll end up swapped out and effectively cached to disk anyway.

The standard solution to this is to write the download to a temporary hidden file on the same volume and then rename it into place once the download succeeds (or delete it on failure).
That's true when downloading to a file, but Invoke-WebRequest is more curl-like than wget-like. It's designed to return an object/struct rather than simply download a file.

If you want to download many/large files, you're probably better off with Start-BitsTransfer.

Yep. And 'wget' is often alias for WebRequest in PowerShell. The amount of footguns I ran into while trying to get a simple Windows Container CI job running, oh man
"curl" being aliased to "Invoke-WebRequest" is also a massive dick move
yea, curl.exe and curl are two different commands on windows. Fun stuff.
I do still find Invoke-WebRequest useful for testing, because it is magically able to reuse TCP connections whereas curl always opens a new connection per request.
Not "on Windows". In PowerShell 5. PowerShell core removed the curl alias 9 years ago.
But PowerShell 5.1 is still the one that ships with Windows.
It's a completely new shell, new commands for everything, no familiar affordances for common tasks, so they add user-configurable, user-removable aliases from DOS/macOS/Linux so that people could have some on-ramp, something to type that would do something. That's not a dick move at all, that's a helpful move.

Harassing the creator/team for years because a thing you don't use doesn't work the way you want it to work? That is.

They removed it in PowerShell core 9 years ago! 9 years! And you're still fixated on it!

It is still present in powershell on my up to date windows 11 machine today, so it is disingenuous for you to claim the alias was removed 9 years ago. It is 100% still being shipped today.

The alias confuses people that are expecting to run curl when they type "curl" (duh) and also causes headaches for the actual curl developers, especially when curl is legitimately installed!

Why the hostile tone? Pretty rude of you to claim I'm fixated on the issue for years and harassing the powershell development team with zero evidence.

When you open powershell it says something like “Install the latest PowerShell for new features and improvements! https://aka.ms/PSWindows

Isn’t it disingenuous to claim it is “up to date” when you know there’s a new version and aren’t using it?

> “The alias confuses people that are expecting to run curl when they type "curl" (duh)

Yes, once, until you learn what to do about it. Which is … just like any other software annoyance. One you think people would get over decades ago.

> “and also causes headaches for the actual curl developers.

Linux users can’t comprehend that the cURL developer doesn’t own those four letters.

> “It has very little compatibility with the actual curl command.

It’s not supposed to have. As I said in another comment the aliases were added to be an on-ramp to PS.

Why aren’t you also infuriated that “ls” isn’t compatible with “ls”? Because you use the full command name in scripts? Do that with invoke-webrequest. Because you expect command to behave different in PS? Do that with curl.

>Linux users can’t comprehend that the cURL developer doesn’t own those four letters.

probably they can comprehend that MS has a history of making things slightly incompatible so as to achieve lock-in and eradicate competing systems.

Also if any program has been the standard for doing this kind of thing for a long time it's curl, it's pretty much a dick move that someone can't just send a one liner and expect it to work on your system because that is often how you have to tell someone working in another system "yes it works, just use this curl script" and then they can see wow it must be something with my code that is messed up.

Not gonna respond to my point about you getting irrationally rude and upset over my impersonal complaint?
tar.exe, however, beats both of those in terms of archive format support and speed.