[sillysaurus3]

== Vim or Emacs ==

Just pick one and force yourself to use it to the exclusion of other editors. Future you will thank you later, because you'll still be using it 20 years from now. "We are typists first, programmers second" comes to mind. You need to be able to move chunks of code around, substitute things with regexes, use marks, use editor macros, etc.

== 6.824: Distributed Systems ==

http://pdos.csail.mit.edu/6.824-2013/ Do each lab. Read the discussion and rtm's course notes.

== Tarsnap ==

https://www.tarsnap.com/download.html How to write C. Study the "meta," that is, the choice of how the codebase is structured and the ruthless attention to detail. Pay attention to how functions are commented, both in the body of the function and in the prototypes. Use doxygen to help you navigate the codebase. Bonus: that'll teach you how to use doxygen to navigate a codebase.

== xv6 ==

http://pdos.csail.mit.edu/6.828/2012/xv6.html

http://pdos.csail.mit.edu/6.828/2012/xv6/xv6-rev7.pdf

http://pdos.csail.mit.edu/6.828/2012/xv6/book-rev7.pdf

Read the book. Force yourself to read it in its entirety. Use the source code PDF to study how to turn theory into practice.

== Arc ==

http://ycombinator.com/arc/arc3.1.tar

You're not studying Arc to learn Arc. You're studying Arc to learn how to implement Arc. You'll learn the power of anaphoric macros. You'll learn the innards of Racket.

Questions to ask yourself: Why did Racket as a platform make it easier to implement Arc than, say, C/Golang/Ruby/Python? Now pick one of those and ask yourself: what would be required in order to implement Arc on that platform? For example, if you say "C," a partial answer would be "I'd have to write my own garbage collector," whereas for Golang or Lua that wouldn't be the case.

The enlightenment experience you want out of this self-study is realizing that it's very difficult to express the ideas embodied in the Arc codebase any more succinctly without sacrificing its power and flexibility.

Now implement the four 6.824 labs in Arc. No, I'm not kidding. I've done it. It won't take you very long at this point. You'll need to read the RPC section of Golang's standard library and understand how it works, then port those ideas to Arc. Don't worry about making it nice; just make it work. Port the lab's unit tests to Arc, then ensure your Arc version passes those tests. The performance is actually not too bad: the Arc version runs only a few times slower than the Golang version if I remember correctly.

== Matasano crypto challenges ==

http://www.matasano.com/articles/crypto-challenges/ Just trust me on this one. They're cool and fun and funny. If you've ever wanted to figure out how to steal encrypted song lyrics from the 70's, look no further.

== Misc ==

(This isn't programming, just useful or interesting.)

Statistics Done Wrong http://www.statisticsdonewrong.com/

A Mathematician's Apology http://www.math.ualberta.ca/mss/misc/A%20Mathematician's%20A...

Surely You're Joking, Mr. Feynman http://web.archive.org/web/20050830091901/http://www.gorgora...

Zen and the Art of Motorcycle Maintenance http://www.arvindguptatoys.com/arvindgupta/zen-motorcycle.pd...

== Above All ==

Don't fall in love with studying theory. Practice. Do what you want; do what interests you. Find new things that interest you. Push yourself. Do not identify yourself as "an X programmer," or as anything else. Don't get caught up in debates about what's better; instead explore what's possible.

[e12e]

> == Tarsnap ==

> How to write C. Study the "meta," that is, the choice of how the

> codebase is structured and the ruthless attention to detail. Pay

> attention to how functions are commented, both in the body of the

> function and in the prototypes.

I just had another look at the tarsnap source code, and while I know Percival is a great guy, and I can't imagine him suing over "mis-use", the bulk of the code is under a pretty restrictive lisence:

"Redistribution and use in source and binary forms, without modification, is permitted for the sole purpose of using the "tarsnap" backup service provided by Colin Percival."

That is, except the code under "libcperciva" which appears to be under a traditional 2-clause BSD license.

So, for example using the bsdtar.c file to teach yourself how to handle command line arguments might be a bit dicey, as it's entirely unclear which part of that file is under the BSD license, and which part you're not allowed to distribute.

It's one of the reasons why I'd which Percival made the entire thing available under a dual license (eg: BSD) and simply required people to use the official client for the tarsnap service.

Then again, I'm not used to audit closed source software, and therefore probably extra scared of what might happen if I accidentially learn something from reading said source... ;-)

On the other hand, Percival does publish quite a lot of stuff that's [ed: entirely free], such as spiped https://www.tarsnap.com/spiped.html .

[supercoder]

Relax

[sillysaurus3]

Actually, it's an important point. One way to learn a given coding technique is to copy-paste it into your own project, modify it until it works, and then delete it and re-implement it yourself. (The "delete and then reimplement it yourself" is the important part. Don't just copy-paste code!) That way you can build up an understanding of the code as you go, without having to understand it in its entirety from scratch. Understanding it in its entirety from scratch is the better way, because it gives you a deeper understanding, but people learn in different ways.

Anyway, illegal is illegal, and copy-pasting from Tarsnap would be illegal. I didn't know Tarsnap was under a restrictive license. It's a shame that people will have to be so careful when learning from Tarsnap, as it's a paragon of modern C best practices, but maybe the license is shrewd.

Though I wish the mentality of "my competitors might steal the code!" will die its deserved death, since evidence thus far suggests it's just paranoia. For example, the codecombat guys open sourced everything and have been fine: https://news.ycombinator.com/item?id=7015126 (But of course that's easy for me to say when I have no competitors to worry about!)

[cperciva]

Though I wish the mentality of "my competitors might steal the code!" will die its deserved death, since evidence thus far suggests it's just paranoia.

Well, Tarsnap is online backup for the truly paranoid... ;-)

Seriously though: I never thought this was likely to be a problem, but given that I was spending a significant chunk of my life on this, I wanted to eliminate obvious risks to my livelihood. I like not starving.

But I've done my best to put the "reusable for a purpose other than building Tarsnap" code into the separate libcperciva tree. And I'm sure that merely reading code to learn from it is not a copyright violation -- there are no laws about copying information into your brain, thankfully. So please, read the code, and email me if you see any bugs; I wouldn't be offering bug bounties if I didn't want people to look at the code!

[Locke1689]

I suspect that your suggested use of the code would fall under the fair use statute in 17 U.S.C. § 107 and the guidelines from Folsom v. Marsh since it seems explicitly for pedagogical or research purposes.

IANAL, but if this were me it seems safe enough that I'd be willing to do it and fight Colin in court if need be. :)

[sillysaurus3]

When I first started programming, a lot of the licensing rhetoric seemed like much ado about nothing. "Look at all these people squabbling about who's allowed to do what with text! It's text! Sheesh!"

But one quickly realizes that the concept of software license restrictions is a fundamental reason for the strength and momentum of open source software. Therefore, it must be true that a central tenant of being a good developer is to respect software licenses. Not merely respect them to the letter of how they're written, but to respect their spirit as well. If an author wishes you don't do something with their code, then you don't do it. There are plenty of reasons why this should be the case, but the most compelling for me is that it'd be lame to ignore the author's wishes while using their work, even if the author won't ever know about it.

If people are still feeling tempted to lift code, well... Remember that you only get to destroy your reputation once.

[laichzeit0]

>Do not identify yourself as "an X programmer," or as anything else.

Yet every single programming job post these days is basically looking for someone with X years of experience in a long enumeration of specific technologies. The usual reason being "They need to be productive. Now."

[golergka]

In my experience, best jobs you get aren't posted — as best candidates don't send out resumes.

[groovy2shoes]

In my experience, networking plays a huge role in landing a good job. Replying to job posts should be a last resort.

[Dewie]

But we can't all be the best.

[8_hours_ago]

The rest of us just say that we are an 'X' programmer when applying for a job that requires the 'X' programming language, and then during the interview mention that we also know a bunch of other languages and know how to use the correct tool for a problem ;)

[wirelessest]

Not at first. But work with people you like and respect, and learn from them. Then ask them when looking for your next job.

Any place worth working isn't looking for the absolute best candidate for X at any cost. A good fit with someone with a good work ethic who wants to learn always works out better than just raw expertise. And a reference from someone you both respect is the fastest most reliable place to find them.

[vog]

> Any place worth working isn't looking for the absolute best candidate for X at any cost

That's so true. If they absolutely need the best candidate, they are about to run a very risky project.

Good management means, among others, to ensure that new people have the opportunity to learn and to "get into" the project. If management requires the "best" developers, it really means that the management is bad and the developers are supposed to make up for that.

(Having said that, there are also people who are simply a sunken cost. That is, managing them requires more resources than the value they produce. However, that's the other end of the scale, and a separate topic.)

[zipfle]

Whenever I hear this, I find I don't recognize the world the poster lives in. Is it a thing to tell coworkers when you're looking for a new job? If not, how many jobs do you need to have and leave before you have a big enough network for this to be a viable strategy? How do you get those jobs?

[Kevguy]

It really depends on the co-worker and the culture of the company. Sometimes I would tell co-workers, sometimes I wouldn't say anything because I was concerned they would leak that to management and I would be asked to leave before I was ready.

I have found user groups meetings a great place to network though. Everyone is helpful when someone says they are looking, even if it is that persons first time at a meeting.

[knodi123]

Take me, for example. ;-)

[wglb]

I think that is also because hiring is generally busted, and recruiters who are not programmers don't know what else to ask.

[cperciva]

== Tarsnap ==

When you're reading this code, be aware that it's a bit of a mongrel -- the code in /libarchive/ and most of /tar/foo (but not /tar/foo/foo) is from the libarchive project and is not mine. So if you're looking to learn "how Colin codes", look at /libcperciva/, /lib/, and /tar/foo/.

s/foo/*/ in the above. Autoitalicization breaks path globs...

[woola]

How long does it usually take to get a reply from Matasano. I have sent a mail couple of days ago, but no reply yet.

[Cederfjard]

Saw tptacek mention the workload yesterday: https://news.ycombinator.com/item?id=7598284

> There are at last count something like twelve thousand people who have reached out to us for our free crypto challenges [...] Every damn one of those people is an email exchange that me, Sean, or Marcin had to have directly, on our own time, with no compensation.

I imagine the response time can vary quite a bit.

[ra88it]

Maybe the first one or two challenges could be available online, and requests should be accompanied by the answers to those. Maybe that would reduce the volume somewhat.

[wrs]

Response time for me on getting sets 1-3 has ranged up to months.

It got better when I remembered we're a Matasano client so I shouldn't feel so guilty about harassing them. But it's an awesome free thing they're doing, so I still do feel a little guilty.

[mmcnickle]

It's a manual process, so can take a while.

[chuchao333]

>>> http://pdos.csail.mit.edu/6.824-2013/ Do each lab. Read the discussion and rtm's course notes.

How to read the discussions? I think it's not publicly available. Btw, thanks a lot for the great post.

[sillysaurus3]

It's a little hidden: http://pdos.csail.mit.edu/6.824-2013/schedule.html

[chuchao333]

I meant specifically the "discussions", as mentioned in the 'course information' page: >>> "Please use Piazza to discuss labs, lectures and papers. We will look at Piazza regularly and answer questions (unless one of you answers first); the entire class can see and benefit from these exchanges."

[jeffreyrogers]

I don't think you'll be able to access these. My university uses Piazza as well and you need a university .edu email address to get access to the Piazza boards.

Fortunately, if the way MIT uses Piazza is anything like the way my school does you aren't missing much. Usually people just use Piazza to clarify something that was glossed over in lecture or to get help on some aspect of the homework.

[sillysaurus3]

Oh, I haven't tried looking for that. If you find a way, let me know! I'd be interested to read it too.

[zura]

I wish video lectures of these were available...

[chubot]

Wow, if you done all that, I want to see your code :) Particularly the distributed systems in Arc. Mind sharing a link to any code you have online (even if it's not that)?

I took SICP a long time ago... all the pure functional stuff was great and elegant. I never really got how you can program "real world" / stateful stuff like distributed systems in Lisp dialects. When I look at Lisp code with hash tables and so forth it just looks horrible to me.

[sillysaurus3]

Well, the point is to self-study, so I'm reluctant to share my code because people will wind up crippling themselves if they fall back on whatever solutions I came up with. If they read my code before attempting their own solution, then they risk overlooking a simpler solution or one better suited for their own purposes. But... I guess I'll share a side-by-side go/arc comparison of lab 1. Lab 1 itself is pretty simple; most of the work was to port some of go's primitives to arc, like channels.

(Note: These html pages are rendered incorrectly on mobile. In particular, the spacing is wrong due to vimscreenshot not inserting   instead of actual spaces. Sorry about that. Try viewing these links from a desktop computer instead.)

https://dl.dropboxusercontent.com/u/231528979/arc/html/index...

Lab 1 in Go:

client.go https://dl.dropboxusercontent.com/u/231528979/arc/html/locks...

server.go https://dl.dropboxusercontent.com/u/231528979/arc/html/locks...

tests https://dl.dropboxusercontent.com/u/231528979/arc/html/locks...

output https://dl.dropboxusercontent.com/u/231528979/arc/html/locks...

Lab 1 in Arc:

client.arc https://dl.dropboxusercontent.com/u/231528979/arc/html/locks...

server.arc https://dl.dropboxusercontent.com/u/231528979/arc/html/locks...

tests https://dl.dropboxusercontent.com/u/231528979/arc/html/locks...

output https://dl.dropboxusercontent.com/u/231528979/arc/html/locks...

Git commit log: https://dl.dropboxusercontent.com/u/231528979/arc/html/log.h...

All diffs between my local codebase and the stock arc 3.1: https://dl.dropboxusercontent.com/u/231528979/arc/html/all-d...

I was interested in seeing how much code can be eliminated by using Arc instead of Go. Turns out: quite a lot. The server is almost 200 lines of Go code, but about 65 of Arc. The client is almost 120 lines of Go, about 35 of Arc. The unit tests are pretty similar in length (~480 vs ~410) but the Arc version is easier for me to read since there's less visual noise.

Anyway it was fun.

[lazyant]

Thanks for the long, detailed quality post.

OT question, what's up with programmers and "Zen and the Art of Motorcycle"? I usually finish all the books I start reading and this was one of the exceptions. After an idea saying something like "we can't define quality ergo everything is quality" or something of the sorts I had to put it down, nothing made any sense and it all seemed to be a grandiose philosophical scheme based on random half-baked ideas.

[danellis]

> Golang

I see this a lot. Do people not realize that the language is called Go, and that 'golang' is just a domain name, like 'ruby-lang', 'scala-lang' etc?

[jotux]

People realize it but also admit it's a terrible name for a language and referring to it as golang is much better in the long run.

[sillysaurus3]

It's a habit I picked up because Google is my primary learning tool. "Golang how to X" is almost always a superior search phrase than "Go how to X".

[S4M]

Did you study everything you are suggesting? That's really impressive. How long did it take you?

Oh well, I should spend less time on HN...

[dorfuss]

Brilliant collection of links, thank you!

[zatkin]

Given that PuTTY has a wealth of features, would you recommend looking at the source code for studying?

[sillysaurus3]

I think you're interested in PuTTY and that's reason enough to study it. :) Copy what you like, discard what you don't. But do whatever is fun. I wish I'd emphasized fun more, since having fun is one of the best ways to maintain motivation and interest.