[isaacimagine]

Chaotic energy haha, I like it. Thanks for the tips re: keeping a journal, I will do this more in the future. I usually keep development notes, though normally in markdown files scattered across the codebase or in comments, never by date in the README. In the future, I might make JOURNAL.md a standard practice in my projects? re:w&b, I used w&b when it first came out and I liked it but I'm sure it's come a lot further in the time since then. I will have to take a look!

Also lol "pretentious perfectionist" I'm glad to finally have some words to describe my design aesthetic. I like crisp fonts, what can I say.

[godelski]

  > Chaotic energy haha, I like it

My boss says I'm eccentric. I say that's just a nice word for crazy lol

> normally in markdown files scattered across the codebase or in comments

I used to do that too but they didn't end up helping because I could never find them. So I moved back to using a physical book. The wandb reports was the first time I really had something where I felt like I got more out of it than a physical book. Even my iPad just results in a lot of lost stuff and more time trying to figure out why I can't just zoom in on the notes app. I mean what is an iPad even for if it isn't really good for writing?

But the most important part of the process I talked about is the logging of all the parameters and options. Those are the details you tend to lose and go hunting for. So even if you never write a word you'll see huge benefits from this.

  > re:w&b

Wandb's best feature is that you can email them requesting a feature and they'll implement it or help you implement it. It's literally their business model. I love it. I swear, they have a support agent assigned to me (thanks Art! And if wandb sees this, give the man a raise. Just look at what crazy people he has to deal with)

  >  lol "pretentious perfectionist" I'm glad to finally have some words to describe my design aesthetic

To be clear, I'm actually not. Too chaotic lol. Besides, perfectionism doesn't even exist. It's more a question about personal tastes and where we draw the line for what is good enough. I wish we'd stop saying "don't let perfectionism get in the way of good" because it assumes like there's universal agreement about what good enough is.

[isaacimagine]

Parameters and options, got it. I try to keep all configuration declarative and make building and running as deterministic as possible. Then I can commit whenever I do something interesting, that I can just checkout to revisit.

[godelski]

I think these are the two main headaches with experimenting. No matter what kind of experiment you're doing (computation, physics, chem, bio, whatever)

  - Why the fuck aren't things working
  - Why the fuck are things working

The second is far more frustrating. The goal is to understand and explain why things are the way they are. To find that causal structure, right? So in experimenting, getting things working means you're not even half way done.

So if you are "organized" and flexible, you can quickly test different hypotheses. Is it the seed? The model depth? The activation layers? What?

Without the flexibility it gets too easy to test multiple things simultaneously and lose track. You want to isolate variables as much as possible. Variable interplay throws a wrench into that so you should make multiple modifications at once to optimally search through configuration space but how can you do any actual analysis if you don't record this stuff. And I guarantee you'll have some hunch and be like "wait, I did something earlier that would be affected by that!" and you can go check to see if you should narrow down on that thing or not.

The reason experimenting is hard is because it is the little shit that matters. That's why I'm a crazy pretentious "perfectionist". Because I'm lazy and don't have the budgets or time to be exhaustive. So free up your ability so you can quickly launch experiments and spend more time working on your hypotheses, because that task is hard enough. You don't want to do that while also having to be debugging and making big changes to code where you're really just going to accidentally introduce more errors. At least that's what happens to my dumb ass, but I haven't yet met a person that avoids this, so I know I'm not alone.