[colvasaur]

3.5 years if you consider my MSc a personal project. There were three hallmark technical failures that are summarized as follows: (a) a computational failure -- at one point I needed to iteratively generate a 3 TB matrix, (b) a mathematical failure -- I made a math mistake, and (c) a software failure -- I tried to use a falsely advertised library to solve the above two problems.

By the end I was knee deep in source code written by mathematicians (read: an absolute mess) trying to fix bugs in code that was not even used in the library, not sleeping, and not eating. I had completely lost my mind.

The taste of failure was bitter on my tongue after the first two failures, but by the last I was truly a shadow of myself. The only lesson I learned from the last failure is that my mind has a breaking point, which is very valuable to know and sense.

My two cents: learn from your failures even if the only lesson to be learned is https://en.wikipedia.org/wiki/Escalation_of_commitment.

[sgillen]

I'm very curious and would like to know more about what you were trying to do, what did you need a 3TB matrix for?

[colvasaur]

PDE constrained optimization. I was optimizing for many hundreds of thousands of parameters. I only needed the diagonal of the Jacobian by the end of the calculation, however it was necessary to build the full dense matrix first without rewriting massive parts of the logic.

[tasseff]

I am also working on PDE-constrained optimization. What was the library that was falsely advertised?