[potatolicious]

I've worked with iCloud in conjunction with Core Data. It's as broken as the article describes, and in fact has gotten worse over time.

No one really fully knows WTF is going with iCloud - documentation is basically nonexistent, support unavailable, implementation broken in trivial ways, and error messages inscrutable. I was at an Mac/iOS conference recently where the Core Data iCloud talk morphed from a standard talk into a "here's the mic, anybody have any idea what's going on with this thing" affair. Helpful, but indicative of the state of things.

IMO, from having studied this for many, many hours over many moons, is that what Apple is trying to do is fundamentally hard, if not entirely unfeasible. They're trying to replace a smart server with one that's dumb as a doorstop. More specifically, they're trying to emulate a CRUD web service with a file sync engine.

Conflict resolution is left up to individual clients, since the server doesn't do any "thinking". Likewise, there is no canonical, authoritative state of the store, since the server doesn't "think", only the clients do. Apple was hoping that by shoving a bunch of diffs of your database onto the server, that clients can reliably reconstruct a sane database by playing them back - except that multiple clients are updating the diffs simultaneously and there is no server-side conflict resolution.

Oh, and if stuff fails, there are no regular snapshot states to fall back to, because iCloud is a file store, not a database engine. Your whole store is now corrupt. Enjoy.

Oh, and there's no way to nuke the database and start over - there are metadata files that are undocumented or that we don't even have sandbox access to, that interfere with completely destroying the database.

At this point you might think "Fine! I'll go to my Settings -> iCloud -> Manage Storage menu and delete everything associated with that app and start it off on a clean slate!"

And then you find out that, even today, in iOS 6.1.3 land, there is a bug from iOS 5.0 where if you delete an app's iCloud bucket it can never be created by that app, on that account, ever again. I wish I were exaggerating. In perpetuity. Presumably some state gets fucked and now this app is forever more unable to store anything on iCloud.

To make even the most trivial experiment on the iCloud API involves creating a brand new app.

iCloud (in so far as it relates to Core Data and document storage) isn't just buggy, isn't just poorly implemented - it's non-functional.

[aidos]

Good lord, that sounds like a complete mess. I haven't been anywhere near it as a developer so I can't really speak on how bad it is from that point of view.

As a user though I have absolutely no idea what is going in with Apple syncing. My wife has an iphone / mac syncing explosion that I simply can't fix. I just don't have any idea where the data is or why everything is duplicated in her iphoto collection. Or what's going to happen in when you remove something from one place.

Most of all I cannot for the life of me explain to my wife why it is that when she lost her phone and restored a previous backup to a new one it DELETED ALL HER RECENT PHOTOS from off her mac without warning. Gone, forever. I watched it happen before my very eyes and it's made me want to never buy another apple product again.

[crag]

"Good lord, that sounds like a complete mess. I haven't been anywhere near it as a developer so I can't really speak on how bad it is from that point of view."

Let me tell you it's not exactly happy land for us consumers either. Apple's "cloud" is a mess. And don't even get me going about the "new" and "improved" iTunes. Frankly I pity developers, cause Apple's mess is giving them a bad name.

I can't think of 1 icloud enabled app (on Windows or Mac) that I've used that worked well. The apps themselves are fine, it's the cloud integration that fails. I mean just look at Apple's own forums.

After 10 years as a Mac user, I'm actually considering moving back to Windows. I'm just over it. It's like every new OS (or app) something goes missing. I'm tired of spending hours searching for replacements for features Apple has removed. And I'm tired of being force-fed services and apps (that Apple's PR states is "gonna change my life") still in beta.

Ok ok //end rant

[killerpopiller]

I switched from linux (after 9years) to an iMac and iCloud was one of the first things I couldn't believe how poorly it works. Ubuntu One sets a standards it seems. Right click your desired folder click ubuntu one and it gets synced. with iCloud (and box, dropbox) I have to duplicate the folders, weird.

I spare you the finder, sftp-integration and the keyboard-layout bitching.

[crag]

Well to be fair, Finder's been a mess for years. ;)

[blub]

You mean all iPhoto photos, al photos in the photos folder on the mac or something else?

[kbatten]

Were you using timemachine, and if so did the photographs get removed from there as well? No matter what, that sounds like an incredibly poor user experience.

[kybernetyk]

After seeing the fate of mac.com and me.com I didn't even consider using iCloud for 'cloud storage'/sync for our Mac app. (We went with Dropbox).

Apple has a long story of half-finished features they try to force onto developers and users. And if that doesn't work the technology gets killed off.

The only downside of not using shiny but questionable new features is that your app won't get featured in the App Store.

[potatolicious]

> "The only downside of not using shiny but questionable new features is that your app won't get featured in the App Store."

This is a pretty huge downside. The App Store is a huge mess - there is effectively no web frontend for it, it's tied deeply into a desktop application, and it's obtuse. This makes marketing nigh impossible and results in Apple having an enormous, and disproportionate, ability to pick winners and losers.

Being featured is a Big Deal(tm).

[deveac]

>The App Store is a huge mess - there is effectively no web frontend for it, it's tied deeply into a desktop application, and it's obtuse.

I was a little shocked when the switched to the 'card UI.' On a mobile device where there is very little real estate, they completely nerfed the amount of app information on the discover screens. Added so much physical effort and time it takes for the user to peruse a like number of apps compared to the former UI.

[lake_rogue]

Perhaps rather than killing the technology off this time they tried to get rid of the person in charge of the technology.

Of course, if this isn't fixed soon...

[pifflesnort]

Core Data is barely functional locally, requiring an enormous amount of invasive runtime hooks to create the appearance of a "transparent" (it's anything but) object store on top of Objective-C objects.

The fact that anyone, anywhere (including Apple), actually expected Core Data to work as a distributed object store is totally beyond my comprehension.

[pooriaazimi]

vor_: unfortunately, you're hellbanned.

> I have no experience with iCloud, but I'm pleased with Core Data for managing a persistent object graph. It automatically handles version migrations, undo, and faulting, and some interfaces can be populating with almost no code via bindings. I'm happy with it. Still, I'll be very interested to watch the Core Data presentations at WWDC.

[lobster_johnson]

Really? Not had any problems with Core Data on top of the default SQLite store (not iCloud, mind you). The interface is very reasonable, since it doesn't pretend to be transparent, and in my experience it works as documented. Care to go into detail?

[seivan]

Never had any issues. I guess what I build is not complex enough.

[potatolicious]

I work on a fairly complex Core Data implementation in a heavily multi-threaded app.

It requires a lot of discipline, and we rely on a lot of learning-through-pain since the documentation beyond the most basic implementation is mostly non-existent.

Core Data is pretty simple if you do all of your work on the main thread, but then your'e also blocking your main execution loop for operations that can take a long time. A large save will easily reach into >200ms land on an iOS device, and your app will feel like crap.

Thread safety in NSManagedObjectContexts is somewhat documented, but has a lot of quirkiness to it. There is some general knowledge around what is and isn't thread-safe, but almost nothing surrounding how to build a multithreaded app and not be murdered by Core Data. Much of this is trial and error and observing what open source projects are doing.

Once you settle into a decent threading scheme that won't kill you (probably with multiple NSManagedObjectContexts) you have to worry about communicating changes across multiple contexts - then you delve into parent-child contexts, where the document gets even more bare. Then you get into propagation delays when crossing context boundaries.

Keeping your code clean through all of this is also a challenge. There are also runtime issues - our implementation in Core Data experiences unbounded growth in memory usage on iOS5, but not in iOS6, forcing us to manually mitigate memory use (which Core Data is supposed to take care of), creating even more thread-safety concerns.

All in all, it's a pain, but it's also the only realistic object store/graph on the platform.

[gurkendoktor]

> A large save will easily reach into >200ms land on an iOS device, and your app will feel like crap.

Can you give an order of magnitude at this would happen? I usually only have 50-200 entities in my database in total, saving maybe half of them at the same time. Yet libraries like RestKit insist on a two-context setup to avoid the saving delay you mention. I'm not sure if I should fight the libraries and go back to a single-threaded setup (that I understand) or just go with the parent/child context flow.

[delackner]

Fetch and save operations get pretty slow with even just 10,000 entities (if you touch all those entities). It gets a little better if you pre-fetch all the entities rather than one at a time, but the save is still very slow. Breaking that out into a backround thread is feasible, but as the previous commenter mentions, then you risk momentary divergence between the UI and the backend.

[seivan]

Index properly. Save in a queue.

[jlmendezbonini]

>Much of this is trial and error and observing what open source projects are doing.

Could you name some of those open source projects? Thanks.

[mdellabitta]

I think Core Data did that back when it was EOF: http://en.wikipedia.org/wiki/Enterprise_Objects_Framework

[seivan]

I just hope people can differentiate between Core Data and iCloud from both your comment and the article.Core Data isn't broken. It might be somewhat magical, but it's not broken.

[pifflesnort]

I think there's a good argument for it being broken:

1) Core Data objects are not NSObject objects. You can't treat them remotely like normal objects: https://developer.apple.com/library/mac/documentation/Cocoa/...

2) Core Data requires making your model mutable, and spreading dependencies on Core Data's abnormal objects throughout your entire code base.

3) Since the abnormal objects are spread throughout your code, and they're tied to non-thread-safe NSManagedObjectContexts, concurrency becomes rather difficult. You have to deal with multiple managed object contexts becoming out-of-sync across threads. Most people just give up and perform expensive operations like -save on the main thread.

4) Core Data merges the concepts of your on-disk model with your in-memory model, despite the fact that the two problem domains have very different requirements in terms of API design, data longevity, relational design, etc. This leads to code that is neither a clean in-memory model, nor a clean on-disk model.

5) Since Core Data objects are abnormal objects, one must produce a considerable amount of boilerplate just to define new model objects. What time you save upfront using a GUI to design your original model, you spend 10-fold in maintaining the code to support it, and in being handicapped by the contraints Core Data places on your code.

6) NSPredicate is awful. It's a brain-dead query language that often make it impossible to cleanly normalize your data store. Aggregate operations are expensive and often require O(n) cost, the query language is poorly documented and poorly specified.

Core Data is a very poorly fit abstraction, but it probably works well enough for simple use-cases. Once you move into concurrency, complex/large models, aggregate operations/projections across your data, or have to deal with it failing, it falls over.

After using it myself for a large project and struggling with all of these issues (and more), and watching another team do the same on their own project, I'd never make use of it again. SQL (via sqlite) is a far better abstraction for managing on-disk data, especially when doing so in a highly concurrent application where one can rely on SQLite's transaction support.

[seivan]

1) How is this a problem?

2) I don't get this. Your model is tied to a context manager, even if they are mutable, you don't need to merge context throughout your base unless you want to.

3) They are not thread safe, the same way UIView isn't thread safe. Why would multiple managed context become out of sync unless you allow it? It's not difficult to deal with. Create a context in the queue you are in. Merge it when you're done.

4) You can use Core Data without persistent store and make it fully in memory and everything will work fine. You can even separate your memory context with multiple context managers. Separating your creates and your deletes and etc.

5) Fud. Write a wrapper, use existing wrappers.

6) Agree.

I am actually kinda curious what problems you've gotten into recently. I've seen Core Data be improved ever since I started using it in 4. I wish it had Cocoa bindings on iOS... but ah well. I've never had any concerns here.

I don't use it for complex stuff. Ironically I didn't use it for a graph though I was told it would be a perfect fit, i wasn't really sure how to traverse without pulling everything into memory. So ended up just using NSObjects and keyedarchiver. But I would be curious to see if someone more familiar with Core Data would have solved it with CD.

[pifflesnort]

> 1) How is this a problem?

I touched on this in a peer comment, but it ties your hands in terms of doing rather normal things that would make for a better model API.

> 2) I don't get this. Your model is tied to a context manager, even if they are mutable, you don't need to merge context throughout your base unless you want to.

Using an immutable model makes concurrency far, far easier (among other things).

> 3) They are not thread safe, the same way UIView isn't thread safe. Why would multiple managed context become out of sync unless you allow it? It's not difficult to deal with. Create a context in the queue you are in. Merge it when you're done.

Like I said in reply to your other comment -- The difference is that the 'model' is what binds all of your code together. The constraints of Core Data therefor introduce tightly bound constraints on all your code.

> 4) You can use Core Data without persistent store and make it fully in memory and everything will work fine. You can even separate your memory context with multiple context managers. Separating your creates and your deletes and etc.

I think you're confusing what I mean by in-memory model and on-disk model.

The way one defines and represents on-disk storage is very different than in-memory storage: the data is persistent, it must remain accessible over time, the model must be versioned, and changes to the actual model definition must be applied as migrations between versions.

In-memory models, however, live only for the lifetime of a particular run of that application, may be modified freely, require no versioning, and require no migrations other than code refactoring.

By tightly binding these things together, Core Data creates the worst of both worlds: all the complexity and downsides of on-disk models with almost none of the flexibility and simplicity of in-memory models.

> 5) Fud. Write a wrapper, use existing wrappers.

That involves quite a bit more work than defining a simple class. Tools such as mogenerator are big, complicated, and a hassle. The point of Core Data is to save effort, but it generates more work and complexity.

> 6) Agree.

Well, at least we agree about something :)

[rbritton]

Adding on to your #3, concurrency became MUCH, MUCH easier in iOS 5. Core Data gained the concept of a background/main thread context and the two methods -performBlock: and -performBlockAndWait:. See: https://developer.apple.com/library/mac/ipad/#documentation/... and http://www.cocoanetics.com/2012/07/multi-context-coredata/

[pifflesnort]

This requires performing all mutation on dedicated code paths. If you require non-asynchronous completion, you have to use the 'wait' variant, but this introduces deadlock risk between threads.

This in turn litters your code with Core Data isms, and means that one can not simply make use of their model as a standard in-memory model.

Compare to SQLite, where one can simply open and commit a transaction synchronously on any thread.

[seivan]

Litters the code with Core Data ism? The same your code is littered with UIVism, NSFoundationsm and etc?

And yes, you can still not use a persistent store manage context on multiple queues.

Please stop using the word thread and use the word queue :)

[pifflesnort]

> Litters the code with Core Data ism? The same your code is littered with UIVism, NSFoundationsm and etc?

The difference is that the 'model' is what binds all of your code together. The constraints of Core Data therefor introduce tightly bound constraints on all your code.

> Please stop using the word thread and use the word queue :)

Why? Queues are M:N mapped to threads, so they are still threads. Additionally, not all work is always done on queues, especially when working with legacy APIs.

[phil]

Bingo. Core Data concurrency pretty much works these days.

[phil]

That's a pretty carefully thought out list of grievances.

What do you mean by 1), though? The link shows a list of methods you can't override on NSManagedObject, but they're things like -class and -isEqual:, which seems pretty reasonable for a system that needs to inspect and compare its objects.

Core Data does solve a few tough problems well. Faulting, uniquing, and cross-context change merging are probably the top three. Yes, it can be much simpler to just query SQLite, but I've never seen somebody do that and abstract away updating stale copies of objects.

[pifflesnort]

> What do you mean by 1), though? The link shows a list of methods you can't override on NSManagedObject, but they're things like -class and -isEqual:, which seems pretty reasonable for a system that needs to inspect and compare its objects.

Well, keep scrolling :~)

Especially 'Custom Instance Variables' and 'Custom Accessor Methods' and 'Validation Methods'. I don't think it's controversial that defining derived instance variables, accessor methods, or other derived state on model classes is an unusual or unwarranted desire, yet doing so with Core Data requires considerably more effort and complexity than a simple model object.

[phil]

I've subclassed NSManagedObject and added instance variables and custom accessors literally every time I've written Core Data code. It's usually not a problem -- yes, you do have to do a couple things to preserve the underlying behavior.

What problems have you had?

[pifflesnort]

I've found the 'couple things' to introduce enough overhead, that when coupled with the other downsides, Core Data hasn't saved me time, reduced LoC, provided any performance advantages, or improved code quality.

[lobster_johnson]

> Core Data objects are not NSObject objects.

Well, they are; they inherit NSManagedObject, which descends from NSObject. It's true that model objects need to follow a bunch of conventions, but that's by design.

> Core Data merges the concepts of your on-disk model with your in-memory model

That's a feature. I rather like how it forces you into rethinking your design.

From your other points it sounds (mind you, sounds, I am not judging) like your app was the wrong fit for Core Data. It's not a replacement for a full-fledged SQL RDBMS and was never intended to be. Core Data works swell for its intended scope.

[imissmyjuno]

I don't know anything about Core Data so pardon my ignorance. How has Apple marketed Core Data? I'm asking because the issues you've mentioned are solved well by SQLite, but perhaps that's because Core Data wasn't designed to be used in anything but the simplest use cases? That tends to happen with solutions that are magical.

[alayne]

Core Data is essentially an ORM layer on top of SQLite.

[drewcrawford]

Core Data does look pretty awful if you compare it to an arbitrarily awesome imaginary library.

All you're doing in this post is phrasing problems that are fundamental to data storage in core-data specific verbage, and calling them a grievance against Core Data. But they're not. There's no other magical solution that somebody could write that does it any better.

For example:

> Since the abnormal objects are spread throughout your code, and they're tied to non-thread-safe NSManagedObjectContexts, concurrency becomes rather difficult.

Concurrency is difficult independently of how you go about it!. The fact that concurrency continues to be hard is not Core Data's problem. The question is, is it easier or harder to use Core Data's primitives to achieve concurrency than it would be doing everything from scratch? I think it's easier. But Core Data against magic is not a fair comparison; a fair fight is between [hard problem with off-the shelf tool] against [hard problem without any tools].

Or this one:

> Since Core Data objects are abnormal objects, one must produce a considerable amount of boilerplate just to define new model objects.

Again, the choice is not between [boilerplate] and [no boilerplate]. As you've correctly pointed out, the mapping between on-disk and in-memory is a hard problem, and the solution will involve writing code that bridges the two worlds. The question is, is [boilerplate that's defined over 10 years of Cocoa convention] superior to [boilerplate that I invented from scratch to solve my problem]? I'm going to claim, 90+% of the time, that you want door A. Perhaps you really do want door B occasionally. But there is no door C with a boilerplate-free existence.

> After using it myself for a large project and struggling with all of these issues (and more), and watching another team do the same on their own project, I'd never make use of it again.

And as a person who has worked on probably over two dozen Core Data applications, I would not touch any codebase that had this philosophy with a ten-foot pole. Concurrency, boilerplate, memory/disk impedance mismatch are the same problems wherever you go. It is perfectly fine to say "I don't like using CD syntax to solve these problems" as a personal preference. But it is another thing entirely to say that CoreData is "broken" because one is either unable or unwilling to solve these same problems we solve every other day of the week in a slightly different syntax than one is used to, particularly when that syntax is generally accepted as a platform standard.

To use an analogy, it would be very poor form to try a Ruby project or two and conclude that ActiveRecord is terrible and that it "falls over" once you step outside "simple use-cases" and that you would never do another Ruby project unless you were emitting postgres queries by hand. There are not a lot of Ruby developers who would sign up to work on that project.

[Aloisius]

IMO, from having studied this for many, many hours over many moons, is that what Apple is trying to do is fundamentally hard, if not entirely unfeasible. They're trying to replace a smart server with one that's dumb as a doorstop. More specifically, they're trying to emulate a CRUD web service with a file sync engine.

Having dug through iCloud's underlying structure, I can tell you what they're doing is very close to what Microsoft's Sync Framework does - right down to the naming of some of the columns in the underlying database.

In fact, if I didn't know better, I would think it was a peer implementation of the Sync Framework (which considering the iCloud server is hosted on Microsoft Azure which has native support for Sync Framework built into hosted SQL Server, isn't too far fetched).

The difference is that Microsoft's Sync Framework seems to work.

[joelreymont]

How do you know that the iCloud server is hosted on Microsoft Azure?

[Atanvarno]

It's known from the start: http://www.datacenterknowledge.com/archives/2011/06/16/iclou...

[stcredzero]

> IMO, from having studied this for many, many hours over many moons, is that what Apple is trying to do is fundamentally hard, if not entirely unfeasible.

"Distributed Databases are Hard" has been a mantra for many decades, probably getting close to a half century. We have known, basically forever, that this is hard on a fundamental level. (Google "Two Generals" problem.)

> Apple was hoping that by shoving a bunch of diffs of your database onto the server, that clients can reliably reconstruct a sane database by playing them back

This is called a variety of things. One of the terms is "transactional replication." It can get very very sticky when there is more than one party originating write transactions, though.

[sampsonjs]

That's funny, I had no clue how it's supposed to work as a user either. Or at least not in the 10 minutes I was willing to give it. The Steve Jobs' Dropbox story is even more hilarious in retrospect. Hilarious at the time too, because Jobs had no intention of providing his product(iCloud) to the philistines who use Windows.

[gte910h]

iCloud + CoreData = Pain and Tears and Blood

I just change the topic to a different backing store when the client wants "cloud persistance" and says "iCloud" without understanding the pain they're asking for.

I agree, completely non-functional

[raverbashing]

From what I've heard iCloud is primarily Java (and some C++)

Not that I'm blaming the technology specifically, but my impression is that Apple is being naive.

Of course you can build with Java, but if you knew better and have the resources Apple has, you would be aiming somewhere else. Google can do this because the infrastructure is there.

Not in the case of Apple

And as this shows, the concept itself of how the service works is broken. This is smelling of 'walled garden fundamentalism' + 'technical ineptitude'

[dpratt]

You realize that a plurality of Google's backend is built in Java, right?

I know it's fashionable to hate on Java, for a lot of good reasons, but I think it's naive to dismiss the power of the platform itself.

We may all have (completely valid) complaints about the outdated and verbose syntax of the language, but the actual JVM itself is fantastic. There are more scalable, fault-tolerant, high performance Java systems out there than the rest of the implementations on all other platforms, combined.

[raverbashing]

"You realize that a plurality of Google's backend is built in Java, right?"

Yes, I know. The problem is not Java directly. Reread what I wrote

Google has the infrastructure and resources to make Java work for them at their size. Because they have whole teams dedicated to build their infrastructure (FS, storage, etc)

Apple has to optimize their resources. They're not going to rebuild what Google did and I'm not seeing them using, I don't know: Cassandra, Hadoop, etc

What I'm saying is that Apple is probably trying to reinvent the wheel. With their resources, they should go with Scala, Clojure or use helper technologies.

[melling]

Apple has 72,000 employees while Google only has 54,000 employees. Apple is over 30% bigger than Google. I would guess that Apple has lots of infrastructure people too.

[raverbashing]

Apple has 26k employees in retail: http://www.apple.com/about/job-creation/

Apart from that, in Google most (core) jobs are software related, in Apple not so much (there's hardware, design, logistics, manufacturing specialists - even if the bulk is @ Foxconn and similar contractors).

Even if we consider software jobs only, I believe the bulk @ Apple is working in OSX/iOS, not to mention iWork, Final Cut, etc and not directly in Cloud infrastructure.

[smackfu]

Practically speaking, for the problem in question, none of these employee counts matter much. 200 engineers is a large number to put on any project, but a tenth of a percent of their total employee count.

[vvhn]

>Apple has 26k employees in retail: http://www.apple.com/about/job-creation/

Not that it matters a whole lot, but, as the website says that's just the US number.

The total number of employees in retail according to Apple's last 10-K filing in October 2012

( http://files.shareholder.com/downloads/AAPL/2393147114x0xS11... )

was 42,400.

" Employees As of September 29, 2012, the Company had approximately 72,800 full-time equivalent employees and an additional 3,300 full-time equivalent temporary employees and contractors. Approximately 42,400 of the total full-time equivalent employees worked in the Company’s Retail segment. "

[melling]

That page says Apple has 80,000 employees, leaving them about the same size as Google. Google has people on the payroll buidling Google Glasses, The Google Car, Gmail, App Engine, Go, Macs, Docs, Chrome OS, and a lot more projects than we probably know. Remember this: http://www.google.com/enterprise/search/

It's true that Apple is highly focused, and they probably have half, if not a quarter of the products that Google has.

[mbell]

> They're not going to rebuild what Google did and I'm not seeing them using, I don't know: Cassandra, Hadoop, etc

There is a non-zero amount of irony in your bashing of Java then picking out two projects that apple isn't using, but you apparently think they should but using, which are both written in Java.

[raverbashing]

I am not bashing Java and I know what these are built on, but apparently no one read what I wrote and stopped reading at 'Java'

I am saying is that probably Apple has a NIH mentality with regards to infrastructure and thus are trying to reinvent things like that. And of course, reinventing these in pure Java takes more time than using Go, for example.

[taligent]

There is no evidence that Go delivers projects quicker than Java.

And Go being so early in its life obviously lacks the wide array of third party libraries that Java does.

[taligent]

Apple uses both Cassandra and Hadoop internally.

And this NIH mentality you've invented is simply not true. You are just confused and probably need to lie down for a bit.