[findjashua]

I laud the effort of making it easier to build apps, but in my experience the backend stuff is mostly building the rest api and cron jobs, which are relatively straightforward. Imho, bulk of the complexity is in user interfaces with lots of moving parts.

[jmspring]

Reading this, I am curious about the systems you've dealt with? If you are talking the back end basically being crud apps, sure, but most services, the bulk of the work is behind the scenese (rest apiece, etc)

[findjashua]

outside the rest api, mostly crawling 3rd party apis, basic map-reduce aggregations, data sanity checks, the occasional s3 dump. No fancy distributed systems or machine learning stuff everyone else seems to be doing. How about yourself?

[onion2k]

It's trivial if your app is just CRUD with domain specific fields, but there can still be some tricky parts;

* Sync'ing data between the frontend and backend if your app has any sort of offline capabilities. Conflict resolution is hard if you want anything better than just a dumb overwrite of what's there.

* Data munging for anything that isn't coming from a well defined API is a horrible and difficult task.

* Scaling and replication on a global scale is still a challenge. It doesn't affect many apps though.

You can knock up a decent REST API in Laravel/Express/Django/whatever in a few hours, but getting the data that you push through it still represents some fun and challenging aspects of app development.

[findjashua]

Agree, those are all challenging problems to solve, but I haven't had the chance to work on them

1. we don't support offline mode in our app. Though, quite a few databases have a version number in their documents (es, couch etc), so the user can be shown conflicts if the server responds with a version mismatch

2. Luckily, I haven't had to work with any overly terrible APIs. The documentation is usually bad, but fiddling around with the req params/body usually gets me what I want

3. We're selling to businesses, so our traffic isn't at that scale.

[mikermcneil]

You're absolutely right. Which is why we should make the backend even easier :)

[lazyseq]

I build whole systems, and have had specialized jobs ranging from 3D game programming to Machine Learning to Rapid Web App Dev. I've observed many the opposite. I would agree that interfaces have lots of moving parts, but part of what you try to do there is to reduce the complexity because it often means you are passing on too much to the user. I suppose there's a bit of Apple philossphy there.

Just cron jobs and REST APIs? Regarding cron jobs, we generally try to avoid them because communicating with them, getting good information, maintaining, and so on sucks. While I've written tons of cron jobs and still do, it's probably 1% of what I've observed happening in the back-end and usually a sign of a duct-taped system. This excludes baked-in unix tools that are actually supposed to work that way of course, rather I am talking things directly involving apps I build. As for REST API, not everything needs to be or should be REST. It introduces all kinds of extra layers of auth, slowdown, caching issues, and so on. For some domains, REST is a bad idea. Of course we use it where it makes sense, but it's not just some magic thing that is and should be used everywhere and anywhere to be representative of the bulk of back-end. Most REST services are built very quickly, it's rather the logic behind them that takes a long time.

As for concrete things I've encountered that are actually complex and happen in traditionally the back-end, I'd say none of these are straight-forward:

1. Caching - As it goes, other than naming, this is one of the most difficult things in computer science and happens everywhere in a good application, front, back, let, right, data, service, whatever.

2. Machine Learning, Sentiment Analysis, Analytics, OLAP, etc. - People forget that a big part of building a system is getting information back out of it.

3. Orchestration - this is hugely complicated. So much so, Microsoft created a very complicated and misunderstood product for it (BizTalk). Doing orchestration is still something no one gets. It happens in millions of forms and related manifestations.

4. Deployment - Versioning, machine provisioning, cluster management, and so on are all very complicated. There are tons of these things that fall under what many people call Dev Ops now too as well. I have yet to see magic here in any app. I challenge you to deploy a running system that is accepting live data transactions, then seamlessly roll it back without data loss, down time, corruption, weird states, unforseen bugs, and so on. No one has done this in full which is why this area is becoming increasingly full of solutions and start-ups. Whether it's Google App Engine, Azure, Heroku, your in-house AWS scripts, Chef, anything, I am telling you no one really does this 100% right. Supporting a real app is just as important as the app itself, and often the two have to communicate and be written in ways to make this work.

5. Streaming - Building streaming related back-ends is anything but simple and plug-and-play. No, I am not talking about streaming Tweets.

6. Data Migrations/Updates/Changes/Flow - Very important part of dev, and usually one of the top 5 time sinks. Complete nightmare and again products and libraries that help, but no magic bullets to be found here other than doing the work and putting in the time. As for simple cases, almost everyone deals with getting data in X form, transforming it to Y, and having it change form again after getting a reply from Z service. A huge part of people's time is simply manipulating data structures, at any place in the system and a reason why I tend to favor Clojure for web apps that don't need to be speed demons. As for migrations, ask someone to do one, then double or quadruple their estimate, sit back, and watch.

7. Authentication/Authorization - The world runs on more than just OAuth and Social Auth. Try dealing with people that have secure systems or many other systems that you need to communicate with, especially across network boundaries.

8. Legacy Integration - Happens all the time. Especially when you build a shiny new app very fast, then have to roll out v2 that actually scales. Beyond that, try any business that has been around 5+ years and actually makes money to see just how horrid this can be.

9. Data modeling - If the scenario of ABC person created XYZ app which now struggles to do Foo new thing without huge pains, you haven't worked on apps long enough. Data modeling mistakes and limitations are one of the single biggest problems and time sinks. More obvious examples include simple facts like having to dupe or dump data into other places to consume it the ways people actually want. While your shiny normalized tables with a REST service might work well for returning a simple list of customers, as that list scales and you realize you now have a new place where you need 2 fields from here, 8 from there, and so on and it needs to happen in 100ms or less in total back to the user, then you are in for a treat.

10. Logging - in big systems, the code here can be non-trivial. Often this means more than writing to a log file, but rather providing visibility into parts of the system in real-time or close to it.

11. Scaling - Everyone thinks they are doing this right until they don't do it right. This can involve anything from servers to message transport format to compression to memory management to clustering.

We could do this all day, but hopefully this makes a point.

[juliob]

Great list. Would love to see that expanded into a blog post :)