I think the title of the article does not accurately reflect the author's thesis, which is more like "trending tech may not be the best solution to your problem". I agree that riding the hype train instead of using established "mundane" tools is probably a bad idea for most problems, but exploring existing tech that might help you also qualifies as "tech-driven problem-solving" IMHO
I've worked at a large financial services company and a large medical billing company who both thought they could use big data and machine learning to monetize customers(!) data. Niether had expertise with either. One muddled through with local talent, the other brought in a ph.d ML guy. Both spent millions and got nothing in return. Anecedotal, but representative I believe.
EDIT: Completely agree with your comment, and I've had similar anecdotal experiences.
1st rule of Big Data Club
Decide what knowledge you want to get — Then explore how you can generate information that can support this give you this knowledge, and finally you can begin to exploit the data you have at your disposal to generate this information.
The trick is not to generate lots of data, but to know how to extract information from it. Big Data™ Tools can assist in exploiting the data you have, to generate new more meaningful data, but that doesn't make it information, it's just more data.
Note that I distinguish between knowledge, information, and data.
I don't know how big those companies are, but they may qualify for big data if they have millions of users or enormous datasets to analyze. The question is, what's the advantage compared to other methods? Is being "left behind" a valid concern for the company's business or is it just a teenager's version of "left behind"?
The fundamental problem of Big Data is that the low hanging fruit has already been picked. Geezers (in a non gender specific sense) with 30 years in industry X have already either intuited it, or it has been uncovered by traditional KDD techniques. What do Big Data, Machine Learning or Data Science have to offer there? The remaining fruit may be very high indeed, and not economically pickable anyway...
I'm in the same club as you: Financial Services and Insurance.
One of the biggest challenges we had was introducing all kinds of new technology but minimal impact on bottom-line. In fact, it probably caused a great deal of productivity and cultural issues that just slows everything down.
You might be right. I do like my title more than yours though, it's more catchy!
Regarding your point however, I think you might have missed my point by a little bit, so I'll try to clarify.
I'm not advocating for existing tech. I love new technologies, and they're often a better fit than what came before. What I'm advocating for is that instead of spending time and money on investigating new technologies and trying to find problems you could solve with them, you instead spend time on finding and identifying problems that your organization is running into, or developing a vision on where you want your company to be in 5+ years.
I believe that the best and most suitable technology is always already out there, as long as you know what you need to build.
The thing is of course that looking at new technologies is easy, and identifying worthwhile problems to solve is very, very hard. But that shouldn't stop us from considering that way of thinking.
Author provides 2 examples: (1) blockchain and (2) big data Hadoop map reduce
(1) blockchain: most arguments about blockchain always being replaceable with a traditional single node central database conveniently leaves out the constraint for decentralized control. Yes, if you change the ideal goals for the solution, of course you can propose the traditional solution. (E.g. a government central database of property records does not solve the same problem as a decentralized blockchain of property records.)
The problems that blockchains are attempting to solve is real. The more interesting discussion is whether the costs of implementation (whether Proof-of-Work or Proof-of-Stake) will ever deliver on that promise. Two possibilities: (1) subsequent failed attempts of blockchains eventually lead computer scientists to a theorem that states any decentralized scheme always costs more than the economic value returned. (The theorem would be similar to the ironclad conclusions of CAP theorem or Godel's Incompleteness Theorem and also similar to discovering you _can_ synthesize gold but the cost (of a nuclear reactor) to do it costs more than the gold itself.) Or (2) more computer science thinking finally makes blockchains feasible for real world applications. Feasibility includes cpu costs, transactions-per-second, 51% attacks, rogue forks, etc.
(2) Hadoop map reduce: he writes...
>"The entire business plan of a ton of companies [...] set up these huge data clusters based on Hadoop and write complex MapReduce queries for basic operations, and wait for the spice to flow. But there’s no end game. When asked, there’s no vision on what to do with the data that will actually make money"
Author should spell out exactly which data companies are spending millions gathering hundreds of terabytes/petabytes with zero clue what to do with it.
Yes, there are all sorts of examples out there of "solutions looking for a problem" but this particular essay is empty of insightful data points.
His complaint isn't against the Blockchain or Hadoop, but against the people proposing their use for its own sake, without concern about whether it actually provides benefits.
He specifically says that virtual currencies themselves are a valid reason to use the Blockchain.
>His complaint isn't against the Blockchain or Hadoop, but [...] whether it actually provides benefits.
For his blockchain example, I couldn't find his Dutch handicap parking story via Google for more details so I don't know if a traditional central database would solve the same exact problem.
I'm saying that virtually all dismissals of a blockchain in favor of central databases almost always removes the benefits of decentralization. The ironic part is that the skeptics don't realize that the central db "solution" is incomplete when compared to all the decentralized goals (e.g. multiple witnesses guarding against tampering, potentially lower cost ownership verification, etc). Yes, if one changes the rules or moves the goal posts around, one can replace a decentralized blockchain with MySQL. If somebody is the skeptic, does he even notice that he performed that sleight-of-hand in his argument?!?
Besides shandor's point, the Blockchain didn't invent decentralized data sharing. I, along with every developer of accounting systems in my country, was writing hash-chained ledgers before the Bitcoin paper was even published.
The innovation brought by the Blockchain was using a proof-of-work to achieve consensus and avoid double-spending in a network of semi-anonymous peers.
The vast majority of proposed uses of the Blockchain for non-currency reasons that I've seen do not suffer from this problem, and therefore do not need the innovation brought by it.
>The vast majority of proposed uses of the Blockchain for non-currency reasons that I've seen do not suffer from this problem, and therefore do not need the innovation brought by it.
Then we've been exposed to different examples. I've not seen any non-trivial proposed uses for distributed consensus blockchain that can be exactly replaced by a central db. Replacement always involves relaxing a constraint (e.g. trust the centralized organization not to tamper, etc).
shandor's point can be generalized to "solutions looking for a problem" which I already agreed with. Where I differ is based on the actual debates on the serious uses of blockchain I've seen, I don't see that technology abuse as the overwhelming situation.
I think you're still missing the point. I understood that the author was criticizing the mindset of the companies which "want to use blockchain". In those cases, "moving the goal posts" is actually the sensible thing to do, when said companies are looking into the blockchain without understanding if they are actually going to benefit from the core blockchain benefit of decentralization.
There should be nothing wrong in moving the goal posts, when the original goal seems to be the wrong one.
Yup. Most of the goalpost-moving happens to make "decentralization" the goal in the first place, so elaborate solutions can be devised to eliminate the need to trust partners not to falsify their record of the contract, despite the fact that in many real world situations if they can't trust their partners that far, there's no way they should be giving them access to property, resources or cash in the first place.
>For blockchain, I couldn't find his Dutch handicap parking story via Google for more details so I don't know if a traditional central database would solve the same exact problem.
Maybe the "exact same problem" is the wrong thing to try to solve? It might not be a problem in the first place (by itself), just a re-formulation of the actual problem so that it's solvable by blockchain, which seems to be the author's point.
Regarding your point about economic value. Things don't have any in inherent economic value. Price is a function of demand and supply. A thing is only worth something because somebody wants to do something with it.
So, a theorem which says decentralised scheme "costs" more than economic value is not possible to have.
For example, say gold is very expensive to synthesise. But if it is absolutely necessary for some job the cost becomes worth i9t and the price would reflect that.
But do you need decentralization to distribute benefits cards to the disabled?
The point is that many of the proposed uses don't actually benefit from decentralization in any meaningful way, so you might as well save a pile of headaches and use a standard database.
Even if every chinese miner decided to collude and generate invalid blocks the result would be those blocks being rejected by the standard bitcoin nodes in favor of any valid blocks. Having a majority of hashing power does give them power but it definitely doesn't give them control over bitcoin.
Mining is centralized but bitcoin consensus is decentralized.
Big data: they nailed it! Every single application of 'big data' i ever saw was hype-driven, and much easier solved by boring, traditional things that were there in K&R era already: hash maps, Berkeley DB, and memory-mapped files. Sometimes taking 100x less resources, like literally doing on a single computer what took a huge Hadoop cluster.
Maybe on a Google scale of data, that doesn't work as easily. Maybe when you have a billion dollar infrastructure bill, big data works better. But it leaves out 99% of companies who's data isn't that big.
Shut your mouth--I can't keep charging $250/hr. unless people BELIEVE!
But seriously, I just came off a project last year that used Hadoop "because" and for no other discernible reason. I personally did a ton of studying on Data Science and then...crickets. Couldn't find anyone who really wanted that kind of work done. Maybe in time.
I'd be curious if anyone else has had the same experience re: Data Science.
From what I've seen so far it seems like the super big tech companies pay _all the money_ for ML/DS people; outside of that the pickings seem to get slim quick.
I used Hadoop well on an important fraud system. When we first deployed, we only dealt with 59 GB at rest worth of data per day. What I knew would occur is more models would run over the data with each release. I assumed that the models would become more complex over time. They would eventually need to perform calculations overs years worth of data.
Hadoop provided a data-centric approach to parallel computing. Using Cascading, a high level pipe/filter library for Hadoop, we could make complex, locally testable models. Using eventing we could plug those models into a self-managing workflow. Adding a new model meant starting a JVM for that model that hooked into the event system and ran Hadoop jobs as needed. If any one model failed, we could rerun it without affecting the rest of the system.
This scaled to 60 some odd fraud models that looked over up to 5 years worth of data (5 TB). Some were quick since they only looked at a day's worth of data. Some took several hours. In the end, Hadoop made the entire process easier to handle mentally, testable, and scalable.
Every new technology needs firing up, which means hype it and hope some people can do something usefull with it. It's hard to imagine new technology without hype, how else can a large group of people be aware of it existing at all?
The Amish decide 200 years ago 'lets stick with current technology'. It works, they have houses, food, friends, a living.
Why go further? This topic is almost existential :)
The author makes several important points, but I'm not sure he fully understands the mechanics of the market. As a founder, I don't necessarily have to think about solving a problem or making money. I just need to gather enough users. Once I go past a certain threshold, I can sell the company (with the users being the main value), and this is what many of my friends are trying to do. Of course at the heart of the app/service there is some benefit to the user, but few people have the illusion they're solving some world problems - they just want to find a smart way to earn substantial money while doing what they know and like to do.
> As a founder, I don't necessarily have to think about solving a problem or making money. I just need to gather enough users. Once I go past a certain threshold, I can sell the company (with the users being the main value), and this is what many of my friends are trying to do.
I'm not arguing about creating valuation on the market for founders. There are plenty of ways, on a sliding moral scale, to create a high valuation. I don't care about that.
I argue for creating actual value for users, on the long term.
Funnily enough, microcomputers themselves were once the technology solution in search of a problem. The "home computer" fad of the 80s promised lots of applications that didn't quite pan out, including finance, "helping kids with the homework", "the little lady can store her recipes", and home automation. The only ones that really panned out were games (including edutainment), spreadsheets, word processing, and maybe small single-user databases. Anything more required much more sophisticated and expensive equipment, which wouldn't become commonplace till around the 386 era.
> finance, "helping kids with the homework", "the little lady can store her recipes", and home automation. The only ones that really panned out were games (including edutainment), spreadsheets, word processing, and maybe small single-user databases.
finance -> spreadsheets
"helping kids with the homework" -> word processing
"the little lady can store her recipes" -> small single-user databases
Sounds to me like they all panned out except home automation.
"Helping kids with the homework" I meant more in terms of using the machine as a study tool. Aside from edutainment titles, this didn't catch on until online encyclopedias and then the internet took root.
Using a computer to store recipes was wildly impractical. Using it to do basic CRM or point-of-sale tasks with a small relational or navigational database was much more appropriate.
That's a great perspective, and I'm really glad you mentioned this in particular. It's one of those things that was so often repeated, and I always found it to be really bizarre. Recipes? Really?
I'd add music synth to your list of real world uses, in some sense that was a killer app for microprocessors.
> I'd add music synth to your list of real world uses, in some sense that was a killer app for microprocessors.
You're right. The Altair 8080 famously didn't do anything useful at all upon its release. It wasn't until someone in the Homebrew Computer Club discovered that the radio interference its CPU generated could play tones on an AM radio by buzzing the CPU in loops that a use was found for it.
It was probably one of the more obvious examples for a simple non-relational database, neglecting that those early implementations tended to not have any advantages over a card file, and often made things less usable.
Also known as "fad", "buzzword" or "solution looking for a problem". The blockchain and big data are good examples. Some past examples are the internet of things and the semantic web.
This article makes a ton of sense for people in smaller companies that aren't worried about rapidly shifting markets and the long time-to-delivery of new functionality that is made possible by technology. He's effectively arguing "why build android when j2me already exists". The problem is that when the iphone comes out, you want something you can quickly retool into a competitive product - otherwise you might get killed by the movement of the market itself. See blackberry & windows mobile for what happens when you're too late to the game.
Now, you could argue that it makes more sense to try to make the iPhone instead of a me-too system like android originally was. But most companies aren't apple of 2005, and they know it. They know they aren't really leading in anything - they just don't want to be left in the dust. So they ask their tech teams to investigate stuff that might be transformational, like the blockchain, because if it did create a new green field, they want to have access to the new market and at least near-first mover advantage.
I'm afraid you then might have missed my point by a little bit. Let me try and clarify.
I think new technologies are awesome. But I think you shouldn't look at a technology and ask "which problems should I solve with this?", but rather at the problems you have and then consider "which technologies should I use to solve this problem?".
The last one does mean you have to become good at identifying problems in your company or product offering, which is much harder than finding technologies. But I think that is a very worthwhile goal.
Blockchain when requesting wheelchairs
Blockchain is a new technology whose impact is compared to the arrival of the Internet. The Quality Institute of Dutch Municipalities (KING) organizes pilots to investigate what Blockchain technology can mean for municipalities. Stichtse fought with the research question whether Blockchain is useful when requesting a wheelchair. The pilot shows that Blockchain can offer many benefits. For example, the process of requesting issue with Blockchain is faster and more transparent. Residents with Blockchain have more views on the status of their application, as with Track & Trace in a mail package. In addition, the administration around the application becomes a lot easier for all parties.
On the original page there is a link to a dedicated site:
with a detailed process analysis from which it is clear how the "blockchain" has actually no meaning/usefulness in the whole stuff the "problem" (or "non problem") can seemingly be solved by much more traditional technical approaches (which is the article Author's thesis).
I think the author has an interesting direction but don't quite like the examples..
Taking a step back and looking at it more generally I see this as problem solving vs problem finding (in Alan Kay's words), where a perfect example would be the hype with self driving cars and high speed car tunnels. This is problem solving (traffic jams, car accidents, "I need to get there faster"). Whereas problem finding will ask the bigger questions that will lead to better results. In the question of transportation I think you should set up yourself to the problem "How do we build car free cities?"
You might have a point. The article was written to be thought provocative, to make people consider what they are doing.
The actual underlying issue however is that finding and identifying problems to be solved is _hard_. It's much easier to focus on something concrete like a piece of technology.
An example is, well, your example. The problem "How do we build car free cities?" is already much better than "How do we avoid traffic jams". But it's simple to solve that problem, for example by banning all cars; that doesn't solve the actual problem though (and is a bit silly).
The actual problem (I think) is not that there are cars, the problem is that there are too many traffic movements necessary over a physically too limited space to get everyone's needs satisfied.
Rephrasing the problem like that allows you to consider the various aspects of the problem; how can we reduce traffic movements (public transport, carpooling)? How can we more optimally use our limited space (smaller cars, stimulate bike usage and bike lanes)? And what are the needs that are being solved by traffic movements, and could we satisfy those without them (working remotely, grocery deliveries)? And what would it be worth to us to reach these goals, compared to the problem we are solving?
Perhaps I'll write an article on this specific mode of thinking. If I can actually find a way to get it on paper of course.
Yes! I was deliberately phrasing my problem finding as "how to build car free cities", which is too tech, phrasing the problem in relation to tech! But you will quickly get to even bigger questions like "why do we move", "why do we work" which will quickly lead to thinking about deep political issues..
Keep expanding your problem field. How do you create a car free society? I live in Palatka, FL. A small town that's great for tech since housing is low cost and we have reasonably fast Internet to facilitate remote work or cloud based solution development. But to get anything other than food, I have to travel at least 45 minutes to St Augustine unless I purchase online. This means I need a car.
For example, I have 12 5/4 cedar boards in my wife's car for storm window frames. I had to drive her Vibe to Jacksonville, which is an hour drive one way. No one around here sells cedar, let alone in 5/4 width. I can't get that delivered. Even if I wanted to, I need to pick the boards since I have aesthetic requirements.
Not that I'm actually arguing for eliminating cars from small towns:
The solution is public transport; it seems you already have a train that goes to Jacksonville, though it's rather slow; the Brussels-Bruges line is about the same distance and takes only 1h, even with intermediate stops.
As for not having delivery, that's solved by the question itself: if people didn't have cars, they would definitively deliver!
As for choosing the boards, that's hardly a problem, you go to the shop, choose them and they'd package those specific boards and deliver them. That's fairly common for large items, at least here in Europe, where people (even small town dwellers) tend to have small cars.
Hype in tech is terrible. If a technology gets an unusual amount of hype, it's a sign that you should wait a few years before you consider adopting it.
Real consensus about a specific technology stack takes at least a couple of years to build up. If a tech product becomes popular very quickly, it is purely because of its superficial qualities - Often at the expense of finer details.
Yes you're right. But next to making money, a lot of us are also doing our best to do something valuable and worthwhile. It's just an attempt to get people to think in a slightly different direction.