[nnq]

> have any insight to what you can actually deliver

Maybe that's the problem. In lots of AI/ML problems you just CAN'T know ahead of time what can be deliver you need to spend the time and resources to do it and then see how well it works...

The problem imo is on the business side, most businesses don't know how to transform unpredictable progress into profit (even if average on a large timeframe that progress might be HUGE).

So ML/DS people need to overpromise in order to get anything approved, otherwise they'd just have to sit around and do nothing, and overall everyone would be worse too, bc that real but unreliable progress would never happen.

[apohn]

>So ML/DS people need to overpromise in order to get anything approved, otherwise they'd just have to sit around and do nothing, and overall everyone would be worse too, bc that real but unreliable progress would never happen.

IME with 8+ years as a Data Scientist in internal and external consulting roles. What I like to do is propose a phase 1, phase 2, phase 3, etc. Phase 1 is almost always a mix of scoping and "Does it even make sense to do this."

Only a fraction of the teams/customers I work like appreciate and have a tolerance for this longer term strategy. Most of those projects are a success and people get value out of using a model in addressing their business problem.

Most people hear this and respond with "No, Phase 1 needs to solve our poorly defined business problem and show massive value in 3 weeks so we can boast about how innovative we are being. You need to AI this problem. AI it now. Do some of that AI stuff. Give me the AI." Most of these fail and I know they are doing to fail, even if any models turn out be useful.

I will say that there there Data Scientist/AI/ML people who really just throw models at stuff and don't think about the value of what they are trying to deliver, but a lot of them are typically inexperienced and just have new toy syndrome. It's not really any different than somebody who reads a bunch of books (e.g. leadership, business) and then gets overly excited about using what they learned. They'll grow out of it or leave the profession when reality hits.

[hectormalot]

Agree. We do something similar:

* 1 month (or less) to prove that it makes sense to continue. We do start out with a pretty well defined problem definition though. Outcome is typically a bunch of jupyter notebooks proving that we get at least some predictive value that helps solve the problem. If no proof: no-go

* 2 months to upgrade it to something that works well enough to run a real pilot (e.g. shadow run, run with a few agents, in an A/B test, etc.). If no success: no-go

* Then hopefully in production within another 3 months. We're in a pretty regulated industry, most of this time is _not_ engineering.

The phases help set priorities as well. Doesn't always make sense to spend (much) more time on scalability if we don't know it works yet.

[apohn]

>We do start out with a pretty well defined problem definition though.

One of the things that really helps this happen is when the management structure (on both sides) understands a defined problem statement that both sides agree to is a requirement for a project to have any hope of success.

The above isn't specific to Data Science projects. It's just having a management structure who knows the right time to support things being pushed back.

[opportune]

>Maybe that's the problem. In lots of AI/ML problems you just CAN'T know ahead of time what can be deliver you need to spend the time and resources to do it and then see how well it works...

Sure, this is true in some cases, but in most cases I think anybody with a fundamental understanding of ML could tell you yes/no by understanding or estimating whether your input data contains enough signal to explain your desired output. In many cases where ML "fails" it's relatively obvious that there is not enough signal to cover the output; in many other cases you can be quite assured a priori that ML will give you a decent solution without needing to test it first.

There will of course be issues if your practitioners are unable to distinguish these cases or if they are structurally not empowered to say so (e.g. if they are just ordered to make a model using data X to do Y). That is probably what happens in a lot of cases when business leaders blindly decide to "add ML" to their business.

The fact of the matter is that ML is very good at solving problems where an input signal strongly correlates with the output signal, and it's appropriate over other approaches when the mapping is hard to define, such as in recognizing images of birds. If you can apply some transformation to your business problem into something with this structure, you can apply ML; otherwise you probably shouldn't.

[nostrademons]

> The problem imo is on the business side, most businesses don't know how to transform unpredictable progress into profit (even if average on a large timeframe that progress might be HUGE).

There's an interesting whole vs. parts observation in there. Societally, we know exactly how to transform unpredictable progress into profits: markets. And they work very well, over the long term.

It's just that markets work by trying lots of new random things, most of which will be failures, and then creating an incentive to double down on the winners. Every individual market participant has an incentive not to innovate, because most innovations fail. But some of them do succeed, and the ones that succeed end up replacing the companies that never tried to innovate in the first place.

[Jommi]

I would love to learn how unpredictability can be leveraged as a positive. Any tips?

[conjectures]

How does insurance work?

[Jommi]

Is this not exactly the opposite? They try to bet that they can predict the occurrence of something better than what you can.

[nnq]

Not pure unpredictability in general, but unpredictability about how much and when progress will be made in a positive direction, eg. what you'd tend to get in applied AI. A few ideas:

- in a highly competitive market your actions can be unpredictable by competitors: evolution shows this can work even in the most primitive ways - it's so damn hard to swat a fly because it changes direction randomly, being smarter than it doesn't help... you could get supercharged versions of this advantage with random-AI-progress as long as you don't constrain business direction, eg. if your AI research suddenly produces breakthroughs that shows you can have an advantage on smart adult toys by applying you aerospace technology capabilities, then be willing to get in that space and play to win regardless of how much that makes sense (tip: you'll probably want to stay privately owned, markets will likely hate this "random pivoting" as it would look from outside) -- as business you'd be "the fly that no one can swat"

- use what's knows to be good at turning positive unpredictability into overall profits (markets), but internally - have multiple projects & people competing against each other inside a company, cheating antifragility / resilience, and really allow projects to fail (as in make it clear that you could be promoted even if the project/team you lead fails, so encourage the king of risk that would be too high even for a startup) etc. - you'll average out the unpredictibility and be left with the progress + antifragility -- as a business you'll be "the cockroach that survives after radiation, eg. unpredictability, has killed everything else"

Hint: all these are already happening, since before AI... and what you'll see now will be that AI/ML will bring most benefits to large corporations with multiple lines of business and multiple independent departments/subsidiaries competing agains each other, and startups-inside-corporations that until now were "pale fakes"... and probably investors already groked this too, there's a lot of growth in AI/ML applied deep inside the guts of big Cos... when this plays out to real wins, result will probably be pretty dystopic: SMBs will be wiped out, lots independent startups too probably (as a small/lone player you can't turn unpredictable progress into profits), you'll get even more centralization, and more mass-surveilance bc now it will not offer only national security advantages but also massive immediate business advantages with AI feeding on the data...

EDIT+: That's I think the problem we need to solve is how to de-centralize and spread the benefits of applying AI to business... bc it will naturally promote centralization and closed gardens, unpredictable progress is mostly toxic to each individual player alone... we need some business-version-of-democracy++, pure capitalism will result in a marriage of mega-corps + pseudo-totalitarian-govs since these will be the entities naturally thriving in the new landscape.

[lmilcin]

"most businesses don't know how to transform unpredictable progress into profit"

Unpredictable progress is lethal to businesses. As an example, it does not matter how brilliant Steve Jobs was or Elon Musk is, they both need to show regular results that build confidence from shareholders.

About the only time you may not care for steady progress is if you have abundance of resources and clear goal like Bill Gates with his foundation. This guy may believe in something and keep steady stream of funds just on basis of his beliefs that something is right.

But the funds themselves must have been gathered some other way.

It is difficult to get funds for unknown payoff at unknown time in the future.

[nnq]

> Unpredictable progress is lethal to businesses

At peace, or in a quasi steady state, yes. But not in all situations. Eg. in war or other adversarial situations you need to both be making progress and be as unpredictable as possible, in the hope that in some indefinite future you can exterminate and/or enslave the other players.

As the world is becoming more and more war-like, these types of strategies can become more and more advantageous. And you don't need actual military war or even physical violence. Actually it's even better without these, since non-military non-violent war-type adversarial state can be sustained forever!

In an arms race type of business situation, if your progress is predictable it means you're not moving fast enough, and somebody else will outrun you! And moving more from competing on marketshare (fighting for more from the same pie) to competing on progress would be awesome too.

I absolutely love unpredictable progress, and I'm really optimistic about the direction the world is evolving to... endless war and without (most of) the death and violence part will be a great catalyst for high-risk-high-reward technological and scientific progress! And as volatility increases maybe more and more of the businesses that depend on "the world staying stable and predictable" will be wiped out and create some breathing room and opportunity for new more flexible and open players. If predicting the future is no longer on the table, you need to invest in true agility, eg. ability to quickly change direction and speed when you glimpse a new future, maybe replace most specialists with teams of nimble quickly-retrainable AI-augmented-expert-generalists etc.

And back to the AI side... we need to stop playing the "predictions game", you never want to actually predict what will really happen in the real world, you want to predict what could have happened and then immediately interfere and change the direction of things to invalidate that prediction (sabotaging other players that probably also made the same prediction)... hopefully to produce a change in the direction you want... but if that can't work, "using AI to inject volatility and invalidate others' predictions" could be a valid business strategy too...

[lmilcin]

The comment is definitely with regards to businesses and with regards to advances made by the business as seen by stakeholders.

I did not intended this to apply to anything else.

I may have made an error in my post, though. Making unpredictable progress is not lethal in itself if you can show steady progress. It should have probably been worded "lack of steady progress is lethal to businesses". It is ok to have some unpredictable projects as long as you can build trust with steady progress somewhere else in sufficient quantity.