As a historical curiosity, in the very first versions of Unix chdir was a normal command rather than a shell builtin.
You see, at that point Unix had no fork system call. There were multiple processes, but they were created statically at startup rather than on-demand. Running a command in the shell would cause the command to replace the shell in the address space of the process, and the process quitting would put the shell back in there.
This worked perfectly with cd being a normal command. Then they implemented fork() and were for a while very confused trying to debug how in the world fork() could have broken the chdir() system call :)
This is a really common way for software to become better.
The pattern of development:
- Make A
- Make B better
- A now broken because it had dependencies on B
- (maybe) Realize a fundamental pattern that unifies A and B.
- (maybe) Unify principles so that A and B become lemmas.
This is a very scientific way of working. The truth emerges as the byproduct of keeping all the plates spinning in your model. As long as you remain aware of all the plates, you'll eventually realize something deep. It's also the strongest argument I can think of in favour of rewriting a codebase plan-9 style. Once you have the truth in mind, bake your knowledge for big wins. Baked knowledge can form the foundation upon which you can climb higher.
But many developers don't keep all the plates spinning. If you don't have everything (or some subset) making sense in some unified theory, don't rewrite. Without a unified grand theory, then you can't make something better. This is why I write unit tests — doing so keeps your plates spinning. It's not about proving your language and libraries are doing what is expected (although that's valuable too in some languages without unified libraries) — it's about showing that everything is working together the way your grand theory intends. Unit tests should be written to reject some part of the null hypothesis.
To elaborate, I really enjoyed reading Dennis Ritchie give some background on the evolution of Unix[1]:
> As a historical curiosity, in the very first versions of Unix chdir was a normal command rather than a shell builtin.
> You see, at that point Unix had no fork system call. There were multiple processes, but they were created statically at startup rather than on-demand. Running a command in the shell would cause the command to replace the shell in the address space of the process, and the process quitting would put the shell back in there.
> This worked perfectly with cd being a normal command. Then they implemented fork() and were for a while very confused trying to debug how in the world fork() could have broken the chdir() system call :)
To clarify, these are very early revisions of Unix we're discussing, prior to Research Unix v1. By the time v1 happened, fork existed, and `chdir` was a shell builtin.
Interesting how the author was looking at "cd" obviously sourced from FreeBSD project (perhaps found on a Mac computer running XNU kernel) and then, to investigate further, she consulted the source code for the Linux kernel to learn about "the" chdir syscall.
Would the title "How does 'cd' work in MacOS?" be better for readers wishing to learn, or worse? I wonder.
The blog posts from her friend at jvns.ca have been similarly Linux-centric but never use titles that inform the reader as such, i.e., "How does X work?" versus "How does X work in Linux?"
It makes me ask, how important is it for students to be aware of the Holy Grail of UNIX: portability.
How long should one remain blissfully ignorant of incompatibilities between UNIX-like OS that work against those "still seeking the Holy Grail"?
As a user, the UNIX programmers I admire the most are the ones who understand the value of portability, set it as a goal and through broad knowledge and utmost care can get very close to achieving it, despite the mind-numbing work or the tradeoffs this might entail.
I find it sad that people that are so curious about how things work end up buying Macs, and therefore unwillingly contributing to less openness and "studyability".
Not one character in the article is actually devoted to how `cd` works...it's just the story of how the author found out where to find the code for it.
An article much more worthy of the title "How does `cd` work?" would maybe, you know, actually go through the code that makes `cd` work.
I don't know if the "Update" by Julia Evans was in the article when you posted this. But I'd point to that as a piece of useful insight about how it works at the syscall level.
>SO!! If you had a /usr/bin/cd program that ran chdir, that would be fine, but when you started it it would change its own working directory and exit which is not very helpful. It wouldn’t change the working directory of you (the parent process)
I am a little confused. What is the parent process and why does that matter here?
The parent is the shell that called the theoretical /user/bin/cd. It means that a cd program would get launched by the shell, change its own working directory and then terminate, which is completely useless.
What you want cd to do is change the caller's (or parent's) working directory, or in this case the shell's working directory.
There are hacks to do it that you can find elsewhere in this thread (process injection and debug APIs to name two) but frankly the shell changing its own working directory via a builtin is a much simpler and more reliable solution that is also cross-platform.
When a process X creates another process Y, what happens is X becomes the parent of Y and Y becomes a child of X. All processes are created this way. They form a tree structure with PID 1 as the root. The very first process is spawned by Linux itself. Processes are isolated by default and thus cannot modify each other's state.
If we have a shell and we use it to start up a separate chdir program, we get two processes: shell (parent) and chdir (child). The kernel isolates them from each other, so they both have completely distinct working directories. The chdir executes the relevant system call, changes its own working directory and exits. The shell is not affected by the system calls performed by its children, so the chdir process effectively does nothing.
By implementing chdir as a built-in function, the shell is able to change the state of its own process.
command is also useful for this sort of thing (with the -v and -V options). It can also be used to ignore shell functions and aliases (without options) so:
$ alias cd=ls
$ cd foo
<lists contents of foo>
$ command cd foo
<changes to directory foo>
> The cd builtin is invoked as part of the Bash shell.
> The Bash shell invokes the chdir function.
Nope! Not quite.
Bash:
~$ ls -ld foo
lrwxrwxrwx 1 kaz kaz 4 Mar 1 6:50 foo -> /etc
~$ cd foo
~/foo$ pwd
/home/kaz/foo
~/foo$ cd ..
~$ pwd
/home/kaz
Raw chdir and getcwd syscalls:
$ pwd
/home/kaz
$ txr
This is the TXR Lisp interactive listener of TXR 190.
Quit with :quit or Ctrl-D on empty line. Ctrl-X ? for cheatsheet.
1> (chdir "foo")
t
2> (pwd)
"/etc"
3> (chdir "..")
t
4> (pwd)
"/"
See the difference? While of course the shell will invoke chdir, it has its own idea of a current working directory, and translates the argument of cd to something else. For instance cd .. doesn't translate to (chdir "..").
Unrelated: it seems like this site has some sort of "debug" script set up that downloads half a dozen copies of some test image ("r20-100KB.png"). Safia, if you're on here, you might want to get it checked out since it's more than doubling the amount of time it takes your site to load.
This looks fiendishly clever, if it really works, but I don't have time to dig through the 'ptrace' man page to figure out what it's doing. Can someone summarize?
Let me give it a shot: ptrace(2) allows processes to control other processes for debugging (for example, GDB and LLDB use it). What it's doing is gaining privileges to debug your shell process, using this privilege to gain control over its memory, and then just copying over the directory string to the right spot so that the shell thinks it has a new working directory.
Actually looks like it copies the path into the address space of the parent, backs up some register state, tickles the i386 syscall interface by setting registers to call chdir(2), then restores the registers to their original state, resuming the program already in progress.
I long ago wrote a cd replacement for Windows. It does a few clever things I like, but I always felt slightly unclean for how it works. Since it's external to cmd, it has to find the cmd parent-process that launched it, write a thread into the process space, then launch that thread that does the call to SetCurrentDirectory and updates the necessary environment variables.
It works, but I'm not entirely sure it should work.
/usr/src/{path with '.' instead of slash}/{command name}
which is by the way, very handy, and in combination with having all your self built packages in /usr/ports/distfiles/<package>.tar.<whatever> a big plus of running a BSD system from source..
Yet another example where the BSD code is simpler and clearer.
I don't think it's particularly controversial to say that when you want to explore how the internals of unix work, you're almost always better off reading the BSD source code first. Even if you don't run a BSD. (musl libc is great, too, but limited to libc.)
I do most of my systems programming for Linux platforms, but when I have a question about semantics my first stop is POSIX to learn how it should behave. My second stop is the BSD source so I can quickly grok the mechanics. Lastly would be the Linux or GNU code, to confirm exact behavior. Setting aside the fact that Linux or GNU code usually feels like it was written inside-out and upside-down, you can't well understand why and how something works without having a more general understanding of the problem and solution space. This applies to everything in life, but in the context of systems software programming I've developed a very concrete process.
Having a copy of POSIX locally (greppable, but also the local HTML frames version is super easy to navigate), as well as easy access to BSD code in /usr/src, can make this a very fast and efficient process. Much faster than Googling, wading through Stack Overflow, and other haphazard habits.
Or try zsh with a good pre-made config like grmlzshrc[1]. It allows changing to a directory by typing only its name (or path). This includes the .. directory. Doesn't get faster.
I believe that you also don't need the space when you specify a path:
cd\foo\x
The cmd parser in Windows is the most bizarre piece of software in common use. I don't believe there is a single person that actually understands how it works completely.
Okay, but now I want to know how it works on something other than (foo)Nix, because that's not an OS space I care about. What about how it works in CP/M or foo-DOS? I've never used a *Nix computer, but I know that I used 'cd' going back to the time of using a TRS-80 CoCo.
I know it's a low-level system call in most on-disk operating systems to change directories. But, for instance, how does it translate physical address to the human-readable name? Do modern (WinXP+) implementations actually take the time to translate folder names from 8.3 to the extended name field? How?
Heck, I'll be dumb enough to ask - why, specifically at the call level, does 'cd\' stick to one level up/down whereas 'cd ' can just pull from just about anywhere? And why, dare I ask, can 'cd' not display like 'tree'?
These are questions about how 'cd' works, to me. Not just, oh, in (foo)Nix it's a system call.
I appreciate the information honestly, but I meant it mostly in the sense that those would be some of the things I'd expect to see in a "how does 'cd' work?" post.
The funny thing here: why does cperciva, of tarsnap fame, show up in the source code of the CD script? I saw it on the article, checked it on my Mac, and it does indeed show up. Intriguing.
Unfortunately, I learned that there's is no way to invoke cd programmatically from a program, but I didn't get an explanation I could understand why this is not possible.
Can someone explain why you can't invoke cd from a program?
I have had this snippet in my `.bashrc` for years. No idea who to give credit to:
# eg. save mc
# cd mc # no '$' is necessary
if [ ! -f ~/.dirs ]; then # if doesn't exist, create it
touch ~/.dirs
fi
alias show='cat ~/.dirs'
save (){
command sed "/!$/d" ~/.dirs > ~/.dirs1; \mv ~/.dirs1 ~/.dirs; echo "$@"=\"`pwd`\" >> ~/.dirs; source ~/.dirs ;
source ~/.dirs # Initialization for the above 'save' facility: source the .sdirs file
}
source ~/.dirs # Initialization for the above 'save' facility: source the .sdirs file
shopt -s cdable_vars # set the bash option so that no '$' is required when using the above facility
What this does is, whenever you're in a directory that you'd like to "bookmark" as you'd call it. Just type `save whatevername`. Then, when you navigate somewhere else you can type `cd whatevername` and it'll change you back there. It's simply adding to this ~/.dirs file and so overwriting is taken care of by just saving the same name in a new (or the same) directory. It just appends a line.
Also, you can just type `show` and any point and it'll tell you what you've saved and where.
The nicest thing is that this persists with new logins (the only drawback is that other shells that are running don't get the update automatically).
For the same reason you can't invoke a method to change the value of a string, or open a file handle, in another process:
You can't change another process's internal state without sending a message that the other process can choose to interpret (or by hacking the memory that the process is using)
As far as I know, you will need to provide the user with an alias or a function or something that can put in their .bashrc or something, and there is no other way to do it. (Anything that could would constitute a violation of the UNIX process model.)
Letting a program arbitrarily modify another program's working directory or its environment variables would be a nightmare for any program that accesses files from relative paths, I think it's pretty obvious why you wouldn't be allowed to do something like that.
cd is part of your shell, so trying to invoke cd from another program would be like opening a new browser tab from another program. It's an internal feature without an outward-facing API.
Not to be negative, but for learning: There are a few "problems" with this snippet of the article:
----
$ which cd
/usr/bin/cd
$ cat /usr/bin/cd
#!/bin/sh
# $FreeBSD: src/usr.bin/alias/generic.sh,v 1.2 2005/10/24 22:32:19 cperciva Exp $
# This file is in the public domain.
builtin `echo ${0##*/} | tr \[:upper:] \[:lower:]` ${1+"$@"}
Oh, bother! Reading shell scripts can be such a hassle sometimes. I know the tr command is used to translate characters. In this particular case, the second half of the command, the part after the pipe symbol, basically converts the command cd dev to CD dev. I have no idea why this is. In any case, this modified command is passed to the builtin command which is handled by the shell (Bourne shell) that we are using.
----
1. The first mistake is typing `which cd`. `which` is a separate program that looks things up in $PATH, which may not actually be what happens when you run the command. You should have used `type cd`:
$ type cd
cd is a shell builtin
As you discover later in the article, `cd` must be a shell builtin. Which makes it a little mysterious (and interesting!) why the file /usr/bin/cd exists; it won't really do anything, try it:
$ pwd
/home/lukeshu
$ /usr/bin/cd /usr
$ pwd
/home/lukeshu
$ # but it will print error messages
$ /usr/bin/cd /bogus
/usr/bin/cd: line 4: cd: /bogus: No such file or directory
So, why does /usr/bin/cd exist? The comment with the CVS ID gives us a hint: It's a common "src/usr.bin/alias/generic.sh" that is copied (hard-linked) in to /usr/bin for several shell builtins ( https://github.com/freebsd/freebsd/blob/0bc1bed704cc7b7292be... ). For other builtins that don't need to be builtins, it makes sense; let other programs call them with exec. For `cd` it doesn't make much sense though, and I'm not sure why it exists. Is it just for consistency with other builtins, or does it serve a real purpose? IDK.
2. The second mistake is about what `tr` is doing. You claimed it's converting lowercase to uppercase; but that's backward, it's converting uppercase to lowercase.
So, why does it convert to lowercase? Recall that we learned that it's the same script being used for all builtins. If it weren't literally the same file (at the cost of a few more bytes disk space), it could have just done a search/replace within a template, having each be `builtin BUILTIN_NAME ${1+"$@"}`. But they wanted to save a few bytes, and instead the script must detect the appropriate builtin name by translating its program path to a builtin name. If you execvp("cd", ...), it will invoke the script with $0 set to "/usr/bin/cd". If /usr/bin is on a case-insensitive filesystem, and you execvp("CD", ...), that will also call the script, with $0 set to "/usr/bin/CD". How is it going to translate from "/usr/bin/CD" to "cd"? The ##*/ bit trims the leading directories, then the tr bit converts the remainder to lower case.
(as an aside: the `${1+"$@"}` is a little interesting too; why not just write `"$@"`? "$@" will expand to the full list of arguments (after argv[0]). The ${1+...} bit says to only do that expansion if the first argument exists (i.e., there are >= 1 arguments). But that should basically be happening anyway; if there are no arguments, "$@" should expand to a zero-length list. IDK, perhaps a weird historical shell?)
One of the most famous shell-portability issues is related to "$@". When there are no positional arguments, Posix says that "$@" is supposed to be equivalent to nothing, but the original Unix version 7 Bourne shell treated it as equivalent to "" instead, and this behavior survives in later implementations like Digital Unix 5.0.
The traditional way to work around this portability problem is to use ${1+"$@"}.
Also some shells will refuse to run a builtin if there is no executable in the path that matches it.
POSIX actually mandates this behavior for any builtin not on a specific list, though most shells (even dash, which is typically obsessive about complying with POSIX) do not implement this behavior.
But you don't have to take my word for it: I've paste-bin'ed the entirety of what POSIX-2001 had to say about shell built-ins (in general, I didn't include man-pages for individual built-ins): https://lukeshu.com/dump/posix-2001-builtins.txt
As for whether that's still true today, looking at POSIX-2008 (Issue
7), 2013 edition (I don't have a copy of the 2016 edition handy), none
of that has changed.
I only discovered this because I implemented a shell specifically by the specification (just for didactic purposes). I was unable to find a modern shell that acted this way, even with passing the "be more POSIXy" options though.
> If /usr/bin is on a case-insensitive filesystem, and you execvp("CD", ...)
That seems like fairly rare edge case, considering that UNIX typically had case-sensitive file-systems, or rather handled filenames as opaque blobs. I wonder what actual system caused the need for case folding? Maybe HFS?
> I started, as I usually do, by searching for the term “chdir” using the GitHub search bar.
That's not bad, but I'd also suggest checking man pages for syscalls. `man 2 chdir` will give the some documentation on both Mac and Linux OSes and call out the specs that are relevant to the call. (Why `man 2`? That searches section 2 of the man pages which is dedicated to syscalls. How on earth would someone know that? `man man` of course. :-) )
On linux, an interesting thing is that you can inspect a process's current working directory (IE the last thing they chdir()ed to) by looking in /proc/<pid>/cwd/. It presents itself as a symlink to the actual current directory as stored in the kernel.
On the one hand, I admire the search for knowledge, taking things apart and seeing how they work. On the other hand, this seems a bit cargo-cultish. It's as if I took apart a record player, trying to see how it plays music, and I told you it works because the motor turns the the record.
In this case, the author got tangled in the shell script and code, and totally missed the whole subtlety and complexity of the Unix architecture. I expected to see something about the processes, file system, and directory entries. Even after the author added key information that was emailed to him by a reader, he didn't really follow up or try to understand more. I understand that not all coders have or need a degree in computer science, but it really surprises me what this author doesn't know (and doesn't know he doesn't know).
Another example from his next blog about 'ls': "I’ll admit, scrolling through all this C-code can be a little tiresome. Oh, how I miss the days when all I had to do was read JavaScript source! Because C gives you so little out of the box, a lot of the code that you end up reading is not that interesting. It’s largely the kind of stuff that higher level languages implement in their standard library." [https://blog.safia.rocks/post/171381157060/looking-into-ls]
Everybody who comes to know about systems calls will have learned the concept for the first time. Some people come to that knowledge by becoming a web developer, gaining some rudimentary understanding of the command line, typing `which cd`, and eventually reading the bash source. The fact that this is even possible is a testament to the author's curiosity and the value of free software.
Others come to that knowledge by reading a 650 page book about System V after graduating from university. Maybe they already learned it before getting the book, in some practical circumstance, like the author. Maybe they have always known it. But for me, an article by someone in the midst of their learning is a great help, an opportunity for others to share what they know, and exemplary of the hacker spirit.
I think gp is not ragging on the blog's author for being a novice at Unix systems internals but lamenting the " I looked into this but then it all seemed so complicated so I stopped" attitude that is present in this as well as the sibling posts on sudo and ls. I would much prefer to read three posts delving deeper into the inner workings of any one of these commands than the existing three muddled surface level treatments.
I had some of the same thoughts. But I learned C and Unix a long time ago, back when you could more easily understand the source code. I think we used a very early version of Minix, just a few thousand lines. It was a simpler time ;-).
Coming to it now, with all the layers that have built up -- the side-track for the script that uses tr, and the idiom with ${1+"$@"}, and all the cruft build up within sh(1) -- it must be pretty hard to separate what is incidental from what is fundamental.
Yes, thanks for putting it more succinctly than I could. I got the impression the author set out to understand 'cd' (which I would love to read about) then didn't, and I don't think we can tell from her article how much she got out of it. I quoted the part about higher-level languages because it's been my experience that a lot of those libraries are ultimately implemented in C somewhere down at the bottom as well.
I will say I did learn about some shell along the way and the bit about processes was interesting and starting to get at the core of 'cd'. It just seems like there are many other pieces of the puzzle, and I encourage the author to keep researching and keep writing about them.
I agree. In the post about ls, I expected to read about how the program builds the list of files and directories, which system calls are involved and so on. The author doesn't seem to like reading C source code and that's fine. Studying the strace of a simple ls invocation and cross-referencing with the Linux man pages would have revealed the inner workings of the program, though.
>It’s largely the kind of stuff that higher level languages implement in their standard library
I think it's extremely interesting. It's in these libraries that the hidden fun stuff happens. For example, memory allocation and related terms like the heap seem like magic but it becomes clearer once one learns about how it works.
HN is not immune to cargo cults. SV and modern "tech culture" applauds people for writing blog posts like this: "Today I decided to find out what makes the sky blue! I just thought I'd write a blog post about it." Then the culture is reinforced, as when you mention how it's stupid to upvote the blog of someone who literally admits they do not know what they are talking about, you get shouted down for not handing out participation trophies. And yes, I realize how rude this comment is, and how stereotypically anti-millennial it is. But it's what has been happening on HN for years.
I will speculate that it's because the author mentions "Julia Evans" [1] who is quite famous in this community for their articles. That or more people than I thought didn't know some details about how basic commands/built-in functions work in Unix. Nevertheless, whatever people find interesting, they upvote. Good or not, this article caught the attention of many, that's all.
You see, at that point Unix had no fork system call. There were multiple processes, but they were created statically at startup rather than on-demand. Running a command in the shell would cause the command to replace the shell in the address space of the process, and the process quitting would put the shell back in there.
This worked perfectly with cd being a normal command. Then they implemented fork() and were for a while very confused trying to debug how in the world fork() could have broken the chdir() system call :)