I miss the Eudyptula Challenge [0], I never got around to complementing it. It was a series of 20 challenges to introduce you to Linux kernel development, and it was done entirely over email. I think it's been shut down for a few years now, I'm not sure when or if it'll come back at this point.
Aw man. That sounds pretty cool. I guess there isn't anything comparable out there? I'm interested in contributing to the Linux kernel, but it's just so hard to get into it.
The Linux Foundation recently released a free course as well titled "A Beginner's Guide to Linux Kernel Development" - haven't taken it yet, but another resource for the community:
I have looked at this one, it is useful if you do know about Linux internals already. It just introduces you to the development process (how to commit your code and etc.)
Started getting into kernel stuff for work. Does anyone have a good idea what kind of market demand there is for kernel devs or similar skill sets? Being new to this I absolutely love it, but have no idea.
The resources shared by everyone here are so helpful! I’ve been muddling through by reading the docs, source code, and various messages on the mailing list and... it’s been a little painful.
As long time kernel/OS dev, right now is not the best time for kernel jobs. Majority of jobs are in user space (particularly Node/JS/React or aws/Kafka/Hadoop type) or at least the interviews are geared towards them. Most companies which used to maintain their own kernel have either outsourced it or use cloud where minimal kernel change is needed. Even if a company has kernel/OS team, majority of their work is to debug, patch, and port. Usually even they don't want to touch kernel much since, it is super critical component of product, debugging is really hard, getting company specific patch upstream is hard, etc.
Obviously, RedHat, SuSe Canonical, Microsoft, Apple, AMD, Intel, firmware and embedded companies will always need kernel people. However, the demand has really gone down and it is no more considered "elite" team within companies. Jobs are still there but obviously teams lean towards hiring experts and veterans of kernel instead of new comers. Moreover, once you get over simple char driver and few lines of patch type stuff, kernel development is extremely complex and hard. You wont get much support either as very likely your problem will be unique to your company's hardware. So, my take - learn and do kernel stuff on side but avoid getting into it full-time unless you are really passionate about it.
Security product companies like mine (Capsule8), Crowdstrike, Sysdig, etc. will always be looking for software developers with a background in Linux (or Windows) development.
We're hiring. :) Email me (address in bio) if you have experience doing kernel development, reverse engineering, debuggers, emulators, compilers, etc. We do cool stuff.
Consistent kernel hackers get hired fast. Focus on a particular subsystem, participate in the community, and contribute regularly - you'll have opportunities in no time.
what would be a typical salary range for 1 year kernel dev + 4-5 years in related experience? let's say BayArea/NY/Austin? I really did like learning about kernel dev before. Might be time to learn again..
If you're doing reliability engineering for any type of service running on Linux servers, being well-versed in the kernel is a great power that separates you from others.
IBM and Redhat (although I guess nowadays they're pretty related) seem to do a lot of kernel work. There are probably even more that I cant named sleep deprived, but you can throw in something like "linux kernel developer" in a job search board to get an idea
The direct demand is low; the Linux kernel is open source so framing yourself as a 'kernel developer' probably is not a very good way to be an ideal candidate for a job opening.
On the other hand, most developers and even some non-developers recognize that kernel development is hard. If you have a name in the kernel development world this will certainly help your reputation. Likewise, if you have a good knowledge of Linux this will certainly help you for certain job openings (kernel development is probably a great background for most low-level job entries).
Someone here probably knows this, so I figured I'd ask: I submitted a small patch to the kernel a while back and I made a typo in it and it broke the tests, and then I couldn't figure out how to get the test bot to pick up my new patch and by that point I assume anyone who would possibly have reviewed my patch stopped bothering. Is there something I should do to get it looked at again?
If your original patch contained a mistake, you just send the new version with the v2 prefix to the mailing list. You can do this with git format-patch -v2 and then send-email will pick it up automatically.
You should note the change in the individual commit or the cover letter.
I had Al Viro CC'd on the emails when I sent it to linux-fsdevel. Are you saying I should send it to him again, directly? (The patch, for reference: https://github.com/saagarjha/linux/commit/4867a403decc364c8b.... I figured I should also take this opportunity to check to make sure there's nothing wrong with it that might cause it to be ignored.)
The idea may be to avoid shifting a 1 into the sign bit, which is undefined behavior.
The values in question are being convertd back to loff_t anyway, but that's an implementation-defined conversion rather than undefined behavior.
So that is to say, doing the shift in the corresponding unsigned type and then converting to the original signed type is better defined at the language level than just doing the shift in the signed type.
I think this is just academic, because shifting a 1 into the sign bit behaves the same way on all platforms targeted by Linux using the only compiler that you can use.
Well, or maybe not. The one problem there is new changes in GCC which assume that all sorts of things that used to work (and were thus de facto implementation-defined) are no longer happening in programs (because they are undefined behavior, so who would do that?) Thus even if the target hardware has stable two's complement behavior on a left shift that alters the sign of a value, and that has worked fine for years, an updated GCC can suddenly decide that it's okay optimize based on the idea that the program doesn't do such a thing.
On a different note, the proposed change is jarringly ugly because u64 is size specific, whereas loff_t isn't. A long long isn't defined as 64 bits; it is at least 64 bits. On a system that, say, makes long long 128 bits, the proposed code change is not correct. To make this change with the proper elegance and precision, what you need is for loff_t to have a sister unsigned type that pairs with it: a uloff_t.
> Well, or maybe not. The one problem there is new changes in GCC which assume that all sorts of things that used to work (and were thus de facto implementation-defined) are no longer happening in programs (because they are undefined behavior, so who would do that?) Thus even if the target hardware has stable two's complement behavior on a left shift that alters the sign of a value, and that has worked fine for years, an updated GCC can suddenly decide that it's okay optimize based on the idea that the program doesn't do such a thing.
Note that Linux compiles with -fwrapv or one of the similar flags that defines overflow to wrap, so this isn’t an issue. I guess this trivializes this patch more, but I figured that they might want the patch anyways in case they ever move off of the flag. (I couldn’t find a uloff_t, so I copied what other code seemed to be doing in this case.)
I’m not at my computer, so I can’t double-check this, but IIRC the issue was that ((long long)0xffffffff << 32) is undefined because it shifts past 2^63-1, which causes an overflow of the long long. (I know the undefined behavior sanitizer flagged some equivalent code in a Linux reimplementation I was working on, which is how I noticed it in the kernel.)
Awesome! I can't wait to dig through this, I've been wanting to contribute to the linux kernel for over 2 years but no one on the linux mailing lists seemed interested in helping a newbie (so much for kernel newbies). Thanks for putting this together.
Or pick a random SBC from Aliexpress or something you have catching dust at home, and try to improve the mainline Linux support for it. That will get you opportunities to learn more deeply the various kernel subystems, too. Especially around driver and arch code. That's what I did anyway.
I pretty much agree with this. For example, the BSD Handbook and documentation is the point of call for a newbie entering into the community and the codebase is much more readable than the Linux kernel.
Last time I checked, the kernel-newbies site doesn't seem to be beginner friendly and aside from following the IRC channel, the mailing list is still the way to do code reviews there, which is quite frankly pre-historic and very unfriendly for beginners sending patches at best.
At least for FreeBSD, they uses Phabricator for code reviews, HaikuOS uses Gerrit and SerenityOS uses GitHub and they seem to be the other OSes that have both good documentation about their kernels and their userland APIs. Perfect places to start learning about operating systems and kernels in general.
Prehistoric? It is actually how the Git workflow was designed to work, and it is a standard method that does not require third-party services nor tools. Just your mail agent.
Sending patches is as easy as sending one in GitHub or any other web-based system. The problem is that you (and many others) have never done it, and so anything different is harder.
You should consider that if learning a handful of CLI commands is such a problem for you, perhaps it is not the system that is being unfriendly to you, but that you are unfriendly to learning anything else that is not your way.
> Sending patches is as easy as sending one in GitHub or any other web-based system.
Its still prehistoric in projects like the Linux Kernel, from a beginners point of view compared to GitHub, Gerrit or GitLab, which most of the active contributors are already working at large companies who's work requires contributing to it, thus already have invested in time and money learning the contribution process. To look at the LKML and do code reviews via back and forth emails looks simple for a veteran Linux developer at a large company but very arcane for a student sending a patch to the kernel.
> The problem is that you (and many others) have never done it, and so anything different is harder.
Your assumption is quite funny here as I have done both and I can tell you that most of the beginners particularly students are introduced into open-source via GitHub or even GitLab these days and its used as a starting point into contributing to a OS project. IIRC, ReactOS and FreeBSD has retained more student contributors from Google Summer of Code than say GNU/Hurd, by both improving the contribution process for beginners. I still wouldn't recommend beginners to learn about OSes in general by contributing to the Linux kernel due to the above reasons. Project making contributions more accessible for beginners isn't a bad thing, its actually how they attract potential long term contributors to stay on, rather than to make things simple only for veteran developers.
> You should consider that if learning a handful of CLI commands is such a problem for you, perhaps it is not the system that is being unfriendly to you, but that you are unfriendly to learning anything else that is not your way.
There's my point, it is still beginner unfriendly. Beginners these days would start with Github or Gitlab with a GUI to send PRs to a different OS project and wouldn't bother learning tons of CLI commands to send a patch in a email + code reviews. It is therefore favours veteran developers at large companies who stay on being long term contributors than the old days of random Linux enthusiasts doing this, which I'm very surprised you don't see the contribution process prehistoric from a beginners eyes. Perhaps something needs to change...
Take GitHub, and boil it down, and it's a mish mash of version control, diff, and patch, all plastered over with a shiny HTTP veneer.
Diff/patch, git, and email alone are the basic tools you need. Even before VCS was a really big thing, that is how software changes were propagated around.
There is nothing wrong with sticking with the basics. In fact, I flat out refuse to tell newbies that they should host everything on GitHub, or GitLab. It's becoming a twisted new form of "social media-esque" hype.
Well, if you want to be more likely to get reaction from any one of 10 or so odd people the typical patch series goes to directly (sometimes simply because they may have contributed some code to some file you're also modifying now) + hundreds more on a bunch of subsys and more general mailing lists, forcing them to register on some random web service of your choice and then monitor it for further communication and replies and patch series revisions is not the easiest proposition.
Everyone knows how to reply to an e-mail. It's very low effort way to find interested people to review your patch/and get some reaction from maintainers.
That's how it is now. That's not to say, current system is not somewhat problematic to some maintainers/subsystems, but that's not really your businesss as a contributor.
If you cannot by arsed to learn a few new development tools or workflows, I am sorry but this field is likely to burn you out very soon, be it Linux kernel development, the next shiny GitHub replacement or the new edition of HTML/CSS/whatever.
And let's not start about learning the myriad of tools you gotta learn to do in professional Linux kernel development (or any kind of specialized development for that matter).
If you want to educate your students on being one more cog in the machine and make them complacent on learning the bare minimum, that's on you. I for sure wasn't taught like that.
Great, thanks for this contribution. So helpful this just came at the moment when I got my hands on The Linux Programming interface book, hopefully this will all help myself to contribute, thanks guys
BPF has some pretty good docs out there if you just poke around the BCC GitHub repo (libbpf-tools/ is a good one) you’ll find a lot of them. I was able to start building programs using libbpf, but a lot of the “understood” info about the kernel is what’s been slowing me down, so I’m psyched to go through all this material and learn.
[0] http://eudyptula-challenge.org/