[rjmill]

Odd, I don't see any mention of subprocess.run, the workhorse of python scripting.

Quick rundown for the unfamiliar:

Give it a command as a list of strings (e.g., subprocess.run(["echo", "foo"]).)

It takes a bunch of flags, but the most useful (but not immediately obvious) ones are:

  check=True: Raise an error if the command fails
  capture_output=True: Captures stdout/stderr on the CompletedProcess
  text=True: Automatically convert the stdout/stderr bytes to strings

By default, subprocess.run will print the stdout/stderr to the script's output (like bash, basically), so I only bother with capture_output if I need information in the output for a later step.

[bccdee]

Also `asyncio.subprocess`, which lets you manage multiple concurrently running commands. Very handy if you need to orchestrate several commands together.

[theomega]

One thing I can recommend that makes scripting in python with external commands a lot easier is the `sh` module:

https://pypi.org/project/sh/

Basically you can just `from sh import [command]` and then have an installed binary command available as function

  from sh import ifconfig
  print(ifconfig("eth0"))

[theamk]

Please don't use "sh" python library!

By default (1) captures stdout and stderr of all processes and (2) create tty for processs stdout.

Those are really bad defaults. The tty on stdout means many programs run in "interactive" rather then "batch" mode: programs which use pager get output truncated, auto-colors may get enabled and emit ESC controls into output streams (or not, depending on user's distro... fun!). And captured stderr means warnings and progress messages just disappear.

For example, this hangs forever without any output, at least if executed from interactive terminal:

    from sh import man
    print(man("tty"))

Compare to "subprocess" which does the right thing and returns manpage as a string:

    import subprocess
    subprocess.check_output(["man", "tty"], text=True)
   

Can you fix "sh"? sure, you need to bake in option to disable tty. But you've got to do it in _every_ script, or you'll see failure sooner or later. So it's much easier, not to mention safer, to simply use "subprocess". And as a bonus, one less dependency!

(Fun fact: back when "sh" first appeared, everyone was using "git log" as an example of why tty was bad (it was silently truncating data). They fixed it.. by disabling tty only for "git" command. So my example uses "man" :) )

[petters]

> They fixed it.. by disabling tty only for "git" command.

Wow... yes sounds like a library to avoid!

[pxc]

I've used Plumbum for this for some projects at work, and really like it for this.

https://plumbum.readthedocs.io/en/latest/local_commands.html...

It also does argument parsing and validation, so it's generally pretty useful for writing little CLI tools that invoke other CLI tools.

https://plumbum.readthedocs.io/en/latest/cli.html

[Lvl999Noob]

And for the opposite, where you keep your main pipeline in shell but want to use python for some parts of it, there is pypyp.

https://pypi.org/project/pypyp/

It takes cares of the input and output boilerplate so you can focus on the actual code that you wanted python for.

    > seq 1 5 | pyp 'sum(map(int, lines))'
    > ls | pyp 'Path(x).suffix'

[two_handfuls]

So cool! I made pawk [1] to get some of the same features, but yours is better! Congrats!

[1] https://github.com/jean-philippe-martin/pawk

[Lvl999Noob]

I am not the author haha. Just someone who found and really liked that library.

[grumps]

`sh` is nice but it requires a dependency. No dependencies is nicer IMHO. uv makes this way easier but for low dependency systems, or unknown environments stdlib is king.

[btown]

I love how this import trick shows how hackable Python is - and it’s this very hackability that has led to so many of the advances we see in AI. Arguably without operator overloads we’d be 5 or more years behind.

https://github.com/amoffat/sh/blob/2a90b1f87a877e5e09da32fd4...

[code_biologist]

uv for using sh as a dependency in scripts, managed inline, has changed it from “eh, I’ll just use subprocess” to “why not” for me.

https://docs.astral.sh/uv/guides/scripts/#using-different-py...

[grumps]

I love uv, the why not works great if it’s your machine but in places without your machine uv is just another step for a customer.

[zelphirkalt]

Does this give live output (meaning before completion) of processes run?

[c6401]

I'm mostly using this nowadays >>> subprocess.getoutput('ls')

[albertzeyer]

I think the point is that for most things, you don't need to call any external tools. Python's standard library comes already with lots of features, and there are many packages you can install.