[oneearedrabbit]

I am excited to see more solutions that work with Apple Notes database directly. This is my go-to application to capture unstructured memos while researching various topics. Last year I got carried away and built a simple site generator for Notes [0]. As a fun exercise I implemented ORM models to query Sqlite database in Ruby:

  User::Account.server(database: "tmp/test.sqlite").first.
    folders_dataset.offset(1).first
    notes.
    map(&:attachments).flatten.
    find { |el| el.is_a?(User::ObjectTypes::EmbeddedPublicJpeg) &&
      el.height > 700 &&
      el.summary[/animal/i] }.
    media_filepath
 => "Accounts/<uuid>/Media/F669B83A-0C47-408E-9645-015737F3B11F/Pasted Graphic.jpg"

Apple Notes' database structure is a fascinating product story. Notes started as a simple note-taking application that stored its content as raw HTML in a few simple tables. However, as the product grew and needed more features like collaboration, they had to make changes and add new columns and tables. They also implemented two flavors of CRDT-like algorithms that evolved over time: one for plaintext chunks and the other one for embedded objects like tables. On top of that, every change that Apple makes must be backward compatible to support legacy devices and systems; but backward-compatibility was not always possible and users had to go through “Migrate your Notes” steps, but it is again a product story.

I wish that one day, someone from the development team would write an article about it.

[0] https://devlog.notespub.com/2022/08/site-generator-for-apple...

[dunham]

Yeah, I'd like to know the story behind it. topotext is weird and unlike anything I've found in the literature. It is kind of like RGA, but with more links. And I haven't seen anyone else layer attributes on top of text.

It looked like the other stuff came later and was somewhat orthogonal. (Although their ordered set construct was abusing topotext to represent ordered lists of ids.)

I don't know how much has changed since 2018. I think my table decoding may be broken now.

[oneearedrabbit]

Thanks for notesutils, I enjoyed reading your protobuf implementation!

[polyrand]

Hey! I remember reading your blog post a few months ago, and it inspired me to try to create a CMS using Apple Notes + the IMAP syncing features[0]. But everything felt like a hack all the time. Many features are not supported when syncing over IMAP. I wanted to have access to the SQLite database, so I can't have a non-macOS VM syncing and processing notes continuously (or at least I couldn't find a tool that can do that). Parsing the data seems like a lot of work, I was also a bit worried that Apple could introduce a change in the format that would break the current tools. Linking between notes is not well-supported, etc.

At some point, I also had some issues syncing notes between different iCloud accounts when the folder had 100s of notes, and that was probably the tipping point. As I was struggling to make it work, I kept thinking that if I just had a bunch of Markdown files I could move around, the whole process would have been easier.

I eventually realized I didn't need most features of Apple Notes, and moved to Obsidian[1]. It feels quite snappy too, and now I can easily publish some of my notes. It can also use iCloud Drive for syncing between Apple devices. It's missing some collaborative features, but I also didn't need those.

[0]: https://ricardoanderegg.com/posts/apple-notes-blog-cms/ [1]: https://ricardoanderegg.com/posts/apple-notes-to-obsidian/

[oneearedrabbit]

I second your sentiment that it is a hack, and here be dragons. At the same time, I had an honest conversation with myself about the dilemma, and whether I am worried about changes to the database or not. I ended up answering that it is a low-risk factor especially for the project of that kind.

I am sure that Apple won’t be making any substation breaking changes to a database structure unless they have to, there is a price to pay. Such changes will always be tied to a major macOS/iPhone release, and it doesn't happen very often.

That said, I had to put this project on the back burner, it was not immediately clear what is the ROI especially considering there is a long tail of databases to parse. It starts with Mountain Lion and then adds up quickly.

I need to release my source code to Github--ORM approach was worth exploring, it is a concise declaration that is relatively easy to update and manage.

[techdragon]

I’m trying to consolidate my note taking with Obsidian but I’m with you about Notes being a go-to tool… it’s just so much better integrated into all the apple devices, I can scribble quick notes with the pencil, ask Siri and all sorts of shortcuts can easily make or add to notes… but I’m absolutely sick of being held hostage by my software and for me my notes are too important not to protect. I’m glad to see anyone work on this, because I wasn’t relishing the idea of trying to force Apple to open up using the EU digital markets directive. Its my nuclear option if the situation gets worse, the various scripts and fixes stop working and I can’t work around it, because I’m pretty blown away by how complicated the notes format is and I won’t be surprised if they add something new and the format develops another layer of complexity.

[azeirah]

For what it's worth, I am working on a tool that syncs files from a ReMarkable drawing tablet to Obsidian.

I was planning on looking into support Samsung notes and Apple notes next. An open-source project like in the OP will help a lot here!

You can subscribe to a newsletter here if you're interested (in the top right of the page)

https://streamsoft.gumroad.com/l/remarkable-to-obsidian?layo...

[techdragon]

Is this an alternative to needing Remarkable’s Connect service subscription?… Because needing Connect to sync notes off my tablet pretty much killed any desire to buy a remarkable 2 … and I voted with my wallet on it too… I bought a PineNote eink tablet instead (I hope this will eventually be a useful tablet… but I’m not holding much hope given my experience with various Linux distros on the first generation PineTab…)

[e12e]

I think this is a fair perspective on the one hand - on the other hand, as a rm2 owner i appreciate that the software keeps getting significant updates, and i think a subscription model makes a lot of sense.

I do whish the standard software made more concessions to third party software - ssh access is great, but running any kind of gui app feels hacky.

[azeirah]

It is not, it requires the connect service subscription as it uses their cloud API to access data from your Remarkable.

[techdragon]

I thought it might based on a lot of things I saw back when I was still keen on a remarkable tablet. It’s still awesome to see people developing tools to funnel data into Obsidian. It’s great additional evidence I’ve the right long term choice with Obsidian for my notes. And who knows, maybe I’ll end up getting a Remarkable as a gift or change my mind, and I’ll definitely try to remember your tool if that happens… and I’d definitely pay for a tool that automates my Apple Notes to Obsidian… for what that additional data point is worth to your product planning.

[azeirah]

It's possible to do it without their connect API for sure, but that would require users to

1. Look up their ssh password 2. Connect tablet to their computer 3. Run a (my) program on their computer to install custom sync software on the tablet

I'm not focusing on the technically capable ReMarkable users so I opted not to do this. Besides, it would involve a lot more development time which I don't have a lot of as I have a day-job as well.

[some_guy_in_ca]

For Mac - I use Noteplan. Obsidian has some amazing feature but it is soo complicated. I find Noteplan has a more simple user interface and most of the features. They can work together because they both use local markdown files.

[wslh]

I would love to have a tool to retrieve the hyperlink of the current note so you can organize the information as a wiki. I know it uses URIs.

[oneearedrabbit]

I was toying with this idea, and the best (and also the worst) workaround I found is to share a note with yourself; then link a note with ⌘-k. I tried doing this a couple of times--nah, it is a workaround.

[pps]

Hookmark app.

[sillywalk]

Interesting.. Have you discovered any limitations to Notes e.g. how many total or largest size?

[codetrotter]

That is awesome! Thank you for sharing your notes on this :)