[deltaonefour]

>I realized that even armed with all the theory and context, the perfect system still remains a mythical creature. In other words - it doesn’t exist.

I wouldn't go so far to say this.

The perfect system does exist we just need to define it formally. This will take a long time and a lot of research but we can get there.

Any time you hear the words "designing systems" it refers to some aspect of reality we don't understand and we go through this "design process" where we attempt to guess and check our way to a very sub optimal solution.

Take for example the distance between two points. The answer is a straight line. We have a formal definition of this axiom. We do not need to design the shortest distance between two points. If we complicate the problem and ask the question what is the best way to travel between two points in the United States... well then the answer gets much more complicated. Do you take a car? Do you take a bus? Do you take a plane? Which one is cheaper? Which one is faster? Which one is better for the environment? All kinds of decisions make it too complicated to calculate a solution so we turn to Design. We use "design" to create systems where no closed form solution yet exists. And in the past decade we've used machine learning as one possible way of finding a solution for these types of problems.

While we have to use designs for building planes and such I do not believe that this will always be the case for programming. I truly believe in a world where it is possible to calculate our program designs. If you really squint... I sort of see a path leading to this world within functional programming and category/type theory.

https://www4.di.uminho.pt/~jno/ps/pdbc.pdf

[nyanpasu64]

I don't think we'll ever be able to calculate program designs, like having a computer design the theoretically optimal {sorting algorithm, hashmap, SQL engine} for a given processor, for a particular performance criterion (average-case and worst-case performance), out of the exponentially exploding set of possible type and function definitions. And "the perfect system" is different on a DS, a smartphone, a desktop PC, or a GPU or supercomputer. I think it's orders of magnitude more complex than designing an optimal chess algorithm. But I'll look into the PDF you linked, since I believe "calculating a program design" would be great to have (though I doubt it's possible), and people aiming for it will hopefully create useful tools along the way.

I do think we can and should define a self-consistent internal logic for the program designs we do write (eg. under a given set of valid inputs, the program is is memory-safe, is thread-safe, does not produce UB, has program-specific invariants upheld by each function's preconditions and postconditions, and produces correct results with eg. O(n log n) runtime, making O(n) allocations totaling O(n) bytes ever allocated), then verify that the code we write upholds this logic. Every program has an internal logic, correct programs have self-consistent logic, and any holes in the logic are exposed as bugs in the program. And easily understood code makes this logic self-evident to readers.

[deltaonefour]

>I don't think we'll ever be able to calculate program designs, like having a computer design the theoretically optimal {sorting algorithm, hashmap, SQL engine} for a given processor, for a particular performance criterion (average-case and worst-case performance)

Not talking about efficiency here. The theoretical optimal for time complexity is already done. We know how to calculate a quantitative measure for average case and worst case performance. For many problems we know the solution for how to optimize for both of the above cases.

When I refer to design I refer to the aspect of computer science we do not have theory for. How do you organize your logic? What is optimal organization of code such that it is optimal and future proof? This area I believe can be formally defined.

[Jansen312]

Then it will turn into a consensus to define it. Probably will turn into C or C++ committees with ever changing goalposts. So in the end it will be back to square one but instead of the orginal issue it will be a layer up finding the perfect definitions. I think my supervisor summed it up best....there is nothing constant accept for change itself.

[deltaonefour]

Nah I'm thinking in terms of mathematics. That's what I mean by formal. Do committees exist that define geometry? No. A formal basis can be established on a very simple foundation. This is not a committee, I am talking about a formation of a mathematical formalism on program organization similar to the mathematical theory on time complexity (Note the lack of committees on time complexity).

The theorems and axioms of geometry are consistent and evident. This is what I'm proposing. We must define the axiomatic notion of optimal program organization. There may be several metrics here but like the axioms of geometry we must pick something foundational. For example: the shortest distance between two points is a straight line is a foundational axiom chosen for Euclidean geometry.

Just like how Geometry, group theory or probability follows from a set of rules and axioms I foresee the possibility of such a thing happening for program organization.

> I think my supervisor summed it up best....there is nothing constant accept for change itself.

Your supervisors mind is clouded by the endless circle of redesign happening throughout the industry. He doesn't think above and beyond that. What is design? and what is optimal? are the questions he should ask. Sure everyone has their own opinion on what is "optimal" but at the same time everyone on the face of the earth agrees on some foundational concepts that optimality encompasses (including the trade offs). Therein is where the axioms of program organization lies. Somewhere within this universal agreement that nobody has really sought out to fully crystallize yet exists the formal theory of program organization.

We were able to formalize our notion of "luck" with probability. Probability is humanities universal agreement on the true nature of "chance" or "luck." Prior to probability luck and chance were fuzzy, qualitative and opinionated concepts that were ripe for formalization.

If we can formalize "luck" then we can do the same with how we organize logic.