[throwaway1979]

You're absolutely right. That said, I don't buy that big-data is as revolutionary as the Internet. While in theory, every single business can collect data and optimize based on what they see, this is way too complex for most businesses to deal with. While big data has certainly been critical for the business model of ad-based startups, I don't see it being used in other industries. People keep alluding to data-driven medicine and genetic analyses. These are some of the most complex information analyses domains, and yet, I don't see benefit commensurate with the big data hype. I'd love to hear counter examples though!

[tgflynn]

This technology will change the world more than the Internet or any other technology in human history.

You're right that adoption is very slow. I'm convinced that businesses could save trillions of dollars by applying existing weak AI to their problems. Why aren't they doing it ? For one thing there's a huge gulf between the average business person's understanding of what is possible and what actually is. On the other hand the people who understand the technology don't have domain experience in various businesses. You can't develop solutions if you don't know what the problems are and it's very hard to guess at what economically relevant problems exist in fields you've never worked in.

There are probably other barriers as well. Domain experts are unlikely to champion technologies that may, well, replace them. Bayesian networks were developed in academia 20 years ago that outperformed doctors at medical diagnosis. Why aren't they being applied ? There are probably many reasons but I suspect conscious or unconscious resistance on the part of the medical community plays a significant role.

As for a talent shortage, I don't buy it. I'm exactly the sort of person this article talks about, with a strong mathematical background, excellent implementation skills and real world experience in developing and applying machine learning algorithms that have made millions of dollars for my former employers. I have had a website and a LinkedIn profile for over a year that make this fairly clear. How many consulting inquiries have I had ? Exactly zero.

[dbecker]

Whether or not you have these skills: potential employers need to SEE them. There are lots of pretenders out there, and employers are appropriately wary.

Are you showing employers results on your web page that a worse ML practitioner can't match.

Putting results from a kaggle competition on my LinkedIn page landed me my current job (and I am still contacted by potential employers every couple weeks).

The employers of the world aren't stupid, but they aren't omniscient either. So you need to make it easy for them to see that you have the skills you claim.

[tgflynn]

Did you win the competition ? If not what was your rank, if you don't mind sharing ? I'd be really interested to know how much impact this could have.

[dbecker]

I was #4 at the time my current employer contacted me. I've continued in the competition with a small team of their employees, and we are currently #2.

I don't know how sensitive the number of job contacts is to your ranking. My hunch is that a lower ranking would still establish credibility.

[tgflynn]

I couldn't reply to Estragon's comment below, so I'm replying here.

I'm convinced of it primarily because I'm convinced that the majority of activities that human workers perform do not require general intelligence (ie. strong AI). This includes most manual tasks : cleaning, cooking, customer service in restaurants, etc, but also many tasks performed by office or even "knowledge" workers.

A product my previous company developed replaced over 20K workers over a period of years. Few people even know it exists.

Once you are aware of this and follow the news you see it happening in virtually all domains: e-discovery systems replacing attorney hours, automated news story generation, etc. One area that has been lagging is robotics but this will start to develop very quickly, especially with the new DARPA Robotics Challenge.

Now to be clear many of the "Big Data" applications people are talking about may not fall into this human labor replacement category, but the underlying technologies are essentially the same.

[csomar]

I have had a website and a LinkedIn profile for over a year that make this fairly clear

It doesn't work like that. You LinkedIn profile might easily land you any job in Software development, but not consulting.

In my opinion, if you want to do consulting for big corp. you should figure out what it takes to it. An attractive website and presentation, few buzzwords, client testimonials, business cards, and the other blablabla. Yes, it's irrelevant (and shitty) to what you are actually doing, but that's actually the world of consulting.

[BrianLy]

I think you can go further than that. To get people asking for your time as a consultant you have to demonstrate experience and get close to vendors who already support clients you are interested in. For example, targeting a niche "big data" problem with a particular tool, and then developing a relationship with the community supporting that tool. That gives you access to the people who are looking for consulting.

[waterlesscloud]

Big Corp - "We can't find people who can do X!!!!"

Person who can do X - "I've been saying over here I can do X."

Big Corp - "Oh. We don't look over there, it's not how it's done."

I think I'm seeing part of the issue here.

[tgflynn]

That's definitely not my world and one reason I left big-corp in the first place.

EDIT ADDED: It seems like a really broken market if buyer decisions are completely orthogonal to the product being purchased.

[tomjen3]

I hope you are right. If you are, you have identified a potentially very, very lucrative option for a start-up. Broken markets can provide you a lot of money when you fix them.

[lgieron]

This market is already (at least partially) covered by small consulting shops which provide sales front for competent freelancers who don't feel like doing the whole corporate networking&sales ritual.

[ced]

Where do I find these small consulting shops?

[Estragon]

  I'm convinced that businesses could save trillions of
  dollars by applying existing weak AI to their problems.

Why are you convinced of this? Do you have any case studies with obvious broad applications?

  I have had a website and a LinkedIn profile for over a year that make
  this fairly clear. How many consulting inquiries have I had ? Exactly
  zero.

Neither LinkedIn nor your ISP are responsible for marketing you. Have you been presenting your software at restaurant conferences? Do you have any case studies where it saved someone x% of their food costs?

[zheng]

Slightly off-topic, but your website breaks after visiting the RDMS page, as all the other links seem to be relative, so they attempt to go to pages such as /products/people.html

[tgflynn]

Sorry about that, thanks for letting me know. It should be fixed now.