[alfiedotwtf]

Sorry. No, I wasn't referring to JIT Manufacturing...

It was merely a tongue-in-cheek to mean just build what's needed right now in front of you, then get it working at scale once there's an actual needs to scale. This is in contrast to starting from day one spending most of your time thinking of scale, then building everything so that it can infinitely scale (along with the complexities that come with it), only to find out that you have 8 users and your data can fit entirely in memory.

[mgkimsal]

there's a vast middle ground between 'astronaut architecture' and 'common sense necessities', and knowing where a good cutoff point is is hard without experience.

I've met more than a few folks who consider web templating systems to be 'overkill', and have 0 understanding of the risk of xss. Same with sql/db escaping - "I just write the SQL and run it, using all those libraries is just a waste of time", etc.

Many of the projects I've come in to over the years were doing with a "JGID" mentality. And they did "get it done", "it" was just a steaming pile of crap when it was done. "Why is this taking weeks to do - the previous guy was so much faster?"

Had this one last year:

"This was never slow before when X worked on it, you've made changes what have you done? We need to call X back in to the project".

X just "got it done". And X was a db admin who was writing code. And X decided it would be good to have views join against other views which joined against other views, and have some queries which used those views run in triggers.

When X was on the project, and there were 40 users, it was fine. They hit 1000 users, and things were 'slow', so they upgraded to a larger EC2 instance. X left, and I came in, and several months later they hit ~30000 users (not active, just ... user account records). The system was dying with more than 5 active users, because of all the views joining other views on 30k+ records.

Unravelling that meant deciphering all the views, all the queries, all the code that touched all of it, and rebuilding a moderate portion, without tests, known 'good' data, or anyone on the project knowing what 'right' was - they just knew when things looked 'wrong' (or slow).

BUT... 18 months earlier, it was "JGID", and it got done. I'm still a bit perplexed why a professional DB admin thought views joining against multiple other views was a good approach.

We've all got horror stories, I'm sure, it's just that the "JGID" mentality is often preached by competent experts as a good approach, but picked up by beginners as the approach used by experts, and has bad consequences.

[alfiedotwtf]

> And they did "get it done", "it" was just a steaming pile of crap when it was done. "Why is this taking weeks to do - the previous guy was so much faster?"

When I mean JGID, I also meant not make it a steaming pile of shit :)

[mgkimsal]

> When I mean JGID, I also meant not make it a steaming pile of shit :)

the tldr of my post is that people often don't know the difference. We also have grown a culture of people promoting "YAGNI", and its easier for some people to dismiss basic ideas with YAGNI.

Yeah, adding 5 language translations to your project on day 1 - YAGNI.

Storing passwords in plaintext? You need something more than that.

[strken]

All the rules of thumb seem to have massive categories of exceptions where they don't work, and that makes me suspicious of applying new ones that I don't fully understand.

DRY up your code... unless the two code paths look similar but aren't the same.

YAGNI... unless "it" is a database backup, a load balancer, or test coverage of your signup flow.

KISS... unless your problem is complicated enough that a simple solution only implements half of what you need.

Microservices are great... if you've got more than one team and can support the operational overhead of SDN and service discovery. Monoliths are great... until you're pushing code to them every minute and they become impossible to refactor.

etc.

[mgkimsal]

and yeah, you should be suspicious of doing anything that you don't understand. this is all knowledge work, and there's some implicit notion that when you make a decision you have some level of understanding. the bigger the potential impact, the more understanding you should have.

[dhuramas]

Out of curiosity - Which database is this? and is it really taboo to have views joining against multiple other views?

If the underlying views were performant- I'd assume the query optimizer would do the right thing(at least 90% of the time).

EDIT: I guess it depends - Just did more research and found this [1]. As long as the views don't do unnecessary heavy lifting or joining unnecessary tables, it should be fine.

[1] https://dba.stackexchange.com/questions/151169/are-views-har...

[mgkimsal]

postgres. I realize this is heresy to badmouth Postgres here, but it was Postgres (9.0 or 9.1 IIRC).

Had a query that selected from table X joining against view X which selected from view A, and view A joins against tables A,B and C, and table B also joins against a view which uses table A and C.

This was just bad. But trying to explain to non-tech people how bad it is, when "it used to work", is difficult. It used to work when you had 50-100 records. No one ever tested was this would be like with 30k records and 50 concurrent users all executing the same nested/circular view mess simultaneously.

But the fact that it was postgres is kind of beside the point. I don't know of any mainstream DB that would handle this well.

The short term fix was to do this large query once at the end of a process and cache the results; the set of queries in question were happening on a 'dashboard' view which everyone hit all the time. It would still cause problems with concurrency, because when 80 people would go through a process and get 'done' (think timed training exercises), the queries would still all be running more or less concurrently, and still cause timeouts, but it wasn't as frequent, because people tended to be staggered a bit more as they finished.

[jacques_chester]

PostgreSQL has progressively reduced the number of optimisation boundaries that are encountered with views-on-views situations, but you can still wind up telling the database to churn and slosh a whole bunch of data from which you only cherry-pick a tiny portion.

I have definitely been guilty of failing to test how my schemata behave with large data sets.

Oh then there's ORMs. I've seen ActiveRecord spit out some frankly batshit insane queries that would stump a room of Einsteins. But somehow PostgreSQL picked it up, chopped it into a plan and got to work plowing through an incredibly wasteful and repetitious query.

[mgkimsal]

> I have definitely been guilty of failing to test how my schemata behave with large data sets.

As have I. It's the dangerous part of the "get it done" approach. And there's no perfect approach - everything is a tradeoff. How much time do you spend dealing with situations that might never happen?

Experience does give you some grounding when making those tradeoff decisions. No, we don't need the architect the application to scale up/down to handle 25k concurrent users in 5 minutes; that's unlikely going to happen. Yes, we do need to spend the extra 2 hours installing and learning a templating system to avoid common XSS pitfalls.

[sonthonax]

I think SQL views are a bit of a code smell. They're really only useful in my opinion as a hack, where creating a rather gruesome abstraction within the database is easier than changing the query layer.

I'm currently working on a service written by a bunch of hipster developers who, wrote raw SQL in Python. And they wrote views, lots of really difficult to understand views that are self-joining (on JSON fields no less!). The performance is really unpredictable given the input sizes. It's the only time I'd call performance of a database chaotic because given an input size (beyond certain safe ranges) I have no idea what the performance is.

The problem is from my perspective a key bit of your application logic get hidden, you're then bound to migrations to change it.

[mgkimsal]

> lots of really difficult to understand views that are self-joining (on JSON fields no less!)

HOLY TAMOLE! You might have my story beat there. That's one level I didn't have to deal with. My sympathies!

[dredmorbius]

That sounds more like JDI / JGID -- just do it / just get it done.

Sometimes "make it so" from ST-TNG.

[OJFord]

Oh, okay, sorry.

(Fwiw searching 'JIT development' does net first page results of people talking about JIT/lean manufacturing in a software development context.)

[rambojazz]

> just build what's needed right now in front of you

What's needed right now, but that doesn't take a complete overhaul when it's time to start growing.

[tyldum]

It requires a very savvy developer in order to succeed. I have seen the massive technical debt this approach can give.