[LukeShu]

Whenever someone says "most people don't write large Bash scripts", I have to chime in and say that I would consider most GNU/Linux distros to be giant piles of shell scripts.

A year or two ago, Arch Linux migrated the tests for dbscripts (the server-side of how package releases happen) from shUnit2 to BATS. https://git.archlinux.org/dbscripts.git/

[chubot]

I would consider most GNU/Linux distros to be giant piles of shell scripts.

Yes! That was one of the primary motivations for my Oil project [1]. I was building containers from scratch with shell scripts (in 2012 or so, pre-Docker), and I was horrified when I discovered how Debian actually works.

Why Create a New Unix Shell? http://www.oilshell.org/blog/2018/01/28.html

And of course it's not just Debian. Red Hat, Fedora, Alpine, etc. are all big packages of shell scripts, often mixed with other ad hoc macro processing or Makefiles. Alpine does this funny hack where their metadata is in APKBUILD shell scripts, which is limiting when you want to read metadata without executing shell.

I also point out in that post that Kubernetes is pretty new (2014) and it has 48,000 lines of shell in its repo.

That's true of most cloud infrastructure. If you deal with Heroku, OpenStack, Cloud Foundry, etc. there is a ton of shell all over the place. With buildpacks, Travis CI, etc.

And that's obviously not because the authors of those projects don't know what they're doing. Shell is still the best tool for that job (bringing up Unix systems), despite all its flaws.

The world now runs on clusters of Unix machines, and in turn big piles of shell scripts :)

-----

New release here if anyone wants to help me test: OSH 0.6.pre15 http://www.oilshell.org/blog/2019/02/18.html

Caveat: it's still too slow; I'm mainly looking for people to help test it.

Running bats tests with it would be great. I don't know how bats works with the @test annotation? That doesn't look like valid shell syntax.

OSH also opens a lot of new opportunities for people interested in testing / static analysis and shell.

There was some discussion here, even related to bats, but I'm not sure where it went: https://github.com/oilshell/oil/issues/200

[1] The other being how hard it is to write an autocompletion script!

[tomeon]

> I don't know how bats works with the @test annotation? That doesn't look like valid shell syntax.

Bats runs the test files through a preprocessor[1] that, among other things, converts each `@test` case into a shell function. E.g. `@test "the doohickey should frob" { ... }` becomes `test_the_doohickey_should_frob() { ... }`.

[1] https://github.com/bats-core/bats-core/blob/master/libexec/b...

[jacques_chester]

> If you deal with Heroku, OpenStack, Cloud Foundry, etc. there is a ton of shell all over the place. With buildpacks, Travis CI, etc.

Cloud Foundry Buildpacks were rewritten in Go and the (quite elaborate) automation for manufacturing them[0] is mostly test-driven Ruby these days.

I can't claim credit for the former but I had a hand in the latter. Bash is not a sane production language.

[0] https://github.com/cloudfoundry/buildpacks-ci (though I note someone snuck in some Crystal and I suspect it was Dave).

[chubot]

Also, I looked at the buildpacks-ci repo you linked, and it still has ~2000 lines of shell in ~80 files.

For comparison, there's ~12,500 lines of Ruby.

Another issue I have is that if Kubernetes replaces its 44,000 lines of shell with 100,000 lines of Go (or probably more), that's a mixed result at best. There's just a lot of logic to express and shell does it concisely.

Of course, without a better shell, I don't blame them if they switch, but it's still a suboptimal state of affairs.

[chubot]

I agree bash is not a sane production language, and apparently so does Greg Wooledge, somebody who has maintained a popular bash FAQ for a very long time. I quoted him in my latest release announcment:

OSH 0.6.pre15 Does Not Crave Chaos and Destruction http://www.oilshell.org/blog/2019/02/18.html

I did hear about Cloud Foundry moving from shell to Go for some things, and I also heard that Kubernetes was rewriting a lot of its shell. [1]

(1) If you had to choose between Go and bash, I can understand choosing Go. I saw some blog posts along those lines [2].

Although I don't think it's optimal. I'd be interested in seeing some of those "shell scripts in Go" if you have a link. There should be a better language for sane shell scripting (hence Oil).

(2) I imagine it's not fun to port thousands of lines of shell to Go (or Ruby) by hand. Oil is supposed to help with that via an approximate translation and good errors, but that part isn't done yet.

What IS close to done is a dialect of bash that is sane -- or CAN BE MADE sane with user feedback.

For example, in the latest release I added set -o strict-argv, and there's also set -o strict-control-flow and strict-word-eval. I will add a strict-ALL to opt into all at once, as mentioned in the release notes.

Also, OSH gives you all your parse errors at once, rather than having them blow up at runtime later:

How to Parse Shell Like a Programming Language http://www.oilshell.org/blog/2019/02/07.html

[1] I just pulled the Kubernetes and there's 44K lines in *.sh files, as opposed to 48K lines a couple years ago. If it were proportional to the project's growth, I would have expected it to be 100K lines by now, so it seems they are indeed getting rid of shell!

However this only removes ONE LAYER of shell. Any cloud service that uses a Linux distro (which is all of them) is papering over all the nasty layers of shell underneath! I hope that Linux distros will gradually move to Oil to get rid of this legacy.

[2] https://jvns.ca/blog/2017/07/30/a-couple-useful-ideas-from-g...

[orestes910]

That is actually a really solid point. I still don't know that it's necessarily the right way, but I'd certainly rather those scripts be tested!