[tomnipotent]

I'm not aware of a single project, ever, that has gotten their data model right up front and not had to iterate on it countless times as it grew/evolved. Except maybe NASA. Even the best early data models fail after years of updates and evolution.

[Tabular-Iceberg]

This is the rationalisation I get every time when I tell companies that their data model is a mess. Never mind that neither I nor the parent said anything about doing it up front.

Of course they have to iterate, the problem is that there is no deliberate effort anywhere, it’s just piling more crap on top of old crap and deluding themselves that they are some kind of lean, agile visionaries because of it.

[tomnipotent]

No one gets it right, and it's just grandstanding to pretend that "deliberate effort" is the distinguishing difference between good or bad data models. Unless you're dealing with highly specialized technical scenarios, most software is written to solve ambiguous and nebulous business problems that even the business doesn't necessarily understand.

[Tabular-Iceberg]

> most software is written to solve ambiguous and nebulous business problems that even the business doesn't necessarily understand

Obviously if you're not deliberate about what problem you're trying to solve in the first place, no amount of deliberate effort will produce a good data model. Designing business processes must be done with the same deliberate effort, and also need to be constantly refactored.

[tomnipotent]

Have you worked at a start-up? Or in a field new to you, or new altogether? No amount of well-intended deliberate design will make up for domain expertise. Ever. The best you can do is mitigate as much future refactoring as possible, but eve than that's not always the best use of time.

> Designing business processes must be done with the same deliberate effort

That's not always realistic. Businesses processes solve existing problems, sometimes tackle new problems, but just like data models hindsight is 20/20. And just like data models, business processes are constantly evolving and never done.

[Tabular-Iceberg]

> Have you worked at a start-up? Or in a field new to you, or new altogether?

Yes, to both questions. And I still stand by everything I said.

In fact both cases made me form this exact opinion. Being all hand-wavy and haphazard about our business processes is precisely and without a doubt what killed it. The second example was from a big regulatory compliance thing that had taken the particular industry by surprise, again because nobody in that industry can be bothered to understand their own business processes, but cargo cult everything. Managers and domain experts would throw PowerPoint mockups over the fence and tell the devs to "just make it clickable and put it on the web site". I had to constantly fight them to get them to take their own jobs seriously. I nearly burnt out and had to quit shortly after, but we actually released on time and though not perfect, came out miles ahead of the competition.

> And just like data models, business processes are constantly evolving and never done.

That makes deliberate design of business process more important, not less. A culture of hand-waviness is precisely why businesses are still caught by surprise by things like regulations that they are given years in advance to implement. If you don't know where you are, you don't know what you're supposed to pivot to.

[Ericson2314]

Yes absolutely. From day 0 everyone should already know how to do data modeling and perform migrations. If you don't and call yourself and engineer, you have no business starting a company.

[laichzeit0]

I think this is largely a consequence of microservices. What is the "data model" here? You're thinking database, to a microservice that's a repository implementation detail.

[Tabular-Iceberg]

I don't think it's a consequence of microservices, I think the problem is as old as the programmable computer. I'm the old paper and even clay tablet processes had to be constantly refactored too.

And a common argument I hear from microservice fanboys is that their old monoliths were suffering from high coupling and low cohesion and that microservices help with that. But I don't see why they couldn't have just refactored their monolith to address the problems they were having.

[Ericson2314]

I hope this is sarcasm poking fun at why microservices are terrible and small companies should stop LARPing FANG and avoid them?

[laichzeit0]

I’m not saying micro services are terrible at all. But I think the low coupling leads to a lot of “problems” when you try and re-aggregate data for analytics purposes when fundamentally micro services are meant to enable teams to work on them without worrying about how some other micro service is storing its data. It’s a trade off you make.

[Tabular-Iceberg]

> fundamentally micro services are meant to enable teams to work on them without worrying about how some other micro service is storing its data

What exactly is it that microservices bring to the table to achieve that, that doesn't already exist out of the box in every single language in common business use?

[laichzeit0]

It lowers coupling between teams. If my team is responsible for microservice A then I have no dependency on the team working on microservice B. Microservice A writes to its own database. There are no migration dependencies, etc. It's a distributed system. You have to accept that one microservice could be using mongodb, another postgresql, another mssql. You're typically communicating through APIs, one microservice is not reading directly from the database of another microservice. Materializing all this crap when you want to do analytics is where the problem comes in as now you are forced to construct some unified data model.

[SkyPuncher]

My rule is "make it easy for us to fix our mistakes".

Even when we've spent a bunch of time planning out data, but we still got a lot of things wrong in hindsight. The reality is we didn't know enough about our product direction to make any truly informed decisions.

In general, poor decisions seem to stem from working in ambiguity about product, rather than poor technical decisions.

[tomnipotent]

> we didn't know enough about our product direction to make any truly informed decisions.

Bingo. Very few programmers are working on projects that they have many years of domain expertise in, so the data models we come up with are always going to be limited to our experience in the here and now. It's one thing to organize highly technical code around things like graph algorithms are combining multiple b-trees in a single operation to lookup data, it's a whole different thing to tackle line-of-business problems where things are not so well defined (even to the in-house domain experts).

[Ericson2314]

Yes, do migrations well to make it easy to fix mistakes. Don't make vague stupid data models to try to avoid making mistakes by being noncommittal.