Ask HN: How do you store your knowledge?

[nktskr]

Hello, HN community, I'm currently in the ideation phase of a new project focused on improving how individuals manage and organize their knowledge. I'm seeking to understand your current methods and challenges in this area.

I'd greatly appreciate it if you could share your thoughts on the following: 1) List Organization: How do you manage your reading, watching, and listening lists? 2) Content Capture: How frequently do you find yourself copying quotes or highlights from articles/books/podcasts/videos? 3) Information Retrieval: How often do you refer back to your saved highlights? 4) Knowledge Storage: What methods do you use to store professional knowledge? 5) Challenges: What do you find most challenging about managing your knowledge base?

[epc]

Just answering #5 first: on a long enough timeline the only safe way to store information is plain text.

I’ve been caching bits and pieces of information I’ve found online since the late 1980s and anything more sophisticated than plain text has gotten me burned. Anything proprietary is guaranteed to suffer from business development needs pushing for incompatible upgrades to an undocumented format followed by an “our incredible journey” termination of service.

Have learned the hard way that you cannot rely on anything staying online long term, even without organizational change or dissolution. The needs of the business change and one day the entire archive you've been relying on for reference has been removed.

I got into the habit about a decade ago of using Evernote as a stash for items like web pages only to find its web page snapshots frequently looked like they grabbed the content of a page only to lose it some time later. So, back to stashing items in a filesystem.

So…if it’s something I find useful I grab it and stash it in my laptop filesystem (which I back up to my own split geo NAS setup).

I don't know how you build a product to satisfy someone like me because you end up having to fight the distrust that's built up over decades, no fault of your own.

[JohnFen]

I agree entirely about plain text, but I'm also fine with common and long-established graphical file formats and (although I prefer to avoid them because they're terrible for reading on a screen) PDFs.

I also agree that if it's an online-only resource, then it's not preserved at all.

[vunderba]

How frequently do you find yourself copying quotes or highlights from articles/books/podcasts/videos?

Directly copying them? Almost never - at the very least I'll rewrite or rephrase them to assist in committing them to memory.

The lower the bar to "saving" things, the less impact they'll have on you going forward. Think the invention of the digital camera, and the thousands upon thousands of largely forgettable pictures.

What do you find most challenging about managing your knowledge base?

Until I wrote a personal RAG assisted LLM (Mistral based) chat system, it was difficult to retrieve multi-note information in a way that didn't involve searching for each individual note. I have around 10,000 "rich markdown files" (e.g. contains links to media) representing two decades of notetaking for personal projects, work, etc.

Tagging goes a long way but Context-Aware notes aren't really a thing in most note taking applications.

Final Thoughts:

The tools/applications/integrations (GetPocket, Joplin, Logseq) have never made it easier to take notes.

Reading these forums for years and seeing the same reoccurring topic (best note taking tool, how to take notes, etc) I also think that there are three types of people:

1. People who take notes

2. People who don't take notes

3. People who like to discuss the theory of note taking

[jesuslop]

I want all local, future proofish things. A pdf library and a heap of markdown notes. Windows, once prepared, gives me a local search engine of contents and metadata queried via Flow Launcher. I use a custom protocol handler to have file URIs relative to a configurable basis (so URIs remain valid if synched elsewhere). Use files of URL extension that are text files but intepreted by Explorer as symbolic links (being text makes them play well with sync), so the same file can appear in many places of the file tree hierarchy (ground organizational principle).

Now that djvu JBIG2 patents expired I jotted a script from parts to convert djvu to pdfs much better than earlier alternatives (with converted file size swolled a typical mild 66% more than djvu). I read pdfs with sumatra that has customizable external "viewers", that means invoking stuff on the pdf page with alt_f + key. One "viewer" copies a link to the current page in the clipboard. Now sumatra supports pdf annotations, made a couple scripts to convert annotations to pdf clickable hyperlinks (and other to undo that). So basically I have a pdf folder and a markdown folder (managed via Obsidian, if you ask) all interlinkable. I read, and copy and paste into markdown, and paste also a link to return to the source from the note. Pay mathpix suscription, there starts to be free would-be replacements. Use Mozilla reader mode chrome extension and print to pdf to save neat pdf versions of blog posts. Finally I have an e-ink e-reader (2nd hand sony dpt-rp1) and am just writing scripts to sync it with windows via an always-on NAS (the e-reader registers itself via zeroconf, and in the NAS via python one can hook a listener that will call unison two-way sync tool). Add also Omnivore.

[theshrike79]

Have you looked into Calibre + Calibre-web for managing the e-reader? There are dockerized solutions for both if your NAS can run images.

[jesuslop]

This e-reader is quite dumb and only talks to the world via propietary GUI app so I don't think it had a friendship with a selfhosted Calibre. There is a hacked python module to pilot the e-reader but some extra glue would be needed. In my case my data model is laptop first and then have backups on other places including NAS, after, so I prefer to have Calibre in laptop. I can map a NAS path to a laptop folder and Calibre sees it as a device, with a nice send-to-device button. So I just need to end the glue to sync the e-reader with the NAS. Calibre automatically converts EPUBs to pdfs. Having bought a Kindle DX back then, I can even setup a Calibre DRM plugin with the Kindle serial number, buy an e-book in Amazon, click on the AZW file, and have a PDF generated automatically :-) For papers, I prefer to manage them separately.

[utensil4778]

I'm autistic and have serious ADHD. My ideal world is one where everything I browse online, my slack conversations, sloppy notes after a meeting are all fed into a sting of ML models running on a server I own.

Then I get my own knowledge database that I can query with natural language.

I want to be able to ask my second brain "what do I know about x?" And retrieve a list of articles I've read on the topic, conversations where I've talked about it, projects I've used that concept in. My meat brain doesn't have that kind of recall, it takes a lot of mental effort to sort through the enormous pile of things that I know.

The less effort I have to spend on organizing and maintaining my knowledge, the more effort I have available to use it.

It seems like this ideal is just a few years away. Several people are already building systems like this, but it doesn't seem that anyone's actually produced a polished product yet. And absolutely not one that is local-only.

This is not the type of thing that anyone should ever trust to "the cloud". This is something that must be run locally on hardware you own and control.

[nktskr]

Thanks for your thoughts!

When you say "Several people are already building systems like this", could you please name a few? What feature do they lack from your point of view?

About trusting the cloud, what do you think? If the service would offer both local storage and secure synchronization in the cloud, would it be acceptable?

[JohnFen]

I'm not the commenter you're asking, but my take on cloud stuff is pretty simple -- it has no appeal to me and I won't use it. I don't mind if software has the ability to use cloud storage as long as I can keep it disabled and it doesn't phone home anywhere.

[utensil4778]

I haven't been paying close attention, but I've seen a couple of commenters on HN mention their systems. Largely it seems to be a bunch of different services strung together with scripts and it's just too much manual effort. Mostly, local ML models aren't good enoguh yet and everyone farms out at least some of it to OpenAI.

As to cloud, never. This system would ideally contain every detail about my personal life, and frankly I think you'd be insane to trust google or OpenAI with that no matter how heavily you encrypt it. This is a whole new order of magnitude in personal, private data. I wouldn't trust anyone at all other than myself with this.

My server at home gets free solar power and free geothermal cooling. I have no reason at all to ever want to use someone else's computer. I have plenty.

[Leftium]

It's not perfect, but I use https://simplenote.com/

Someday I will build my own clone of Simplenote/Notational Velocity/Taskpaper/Drafts/Bkmks.com...

1. From mobile browser, share to Simplenote adds a new note with title + url.

2. Not often.

3. Not often.

4. Also Simplenote. Also used Notational Velocity: one note per JIRA ticket.

5. I wish Simplenote's tagging system were better (more like TaskPaper)

- I wish it were simple to add (markdown) images like Github. (CTRL-V automatically uploads and inserts markdown).

- Also Simplenote added some features that get in the way (I just want to type `[`, but Simple note opens this inter-note linking interface)

- Notational Velocity supported rich text, but I just wanted everything in plaintext, especially when pasting information into a note.

[romerocarlos]

To answer 1) and 4) In my case I used Obsidian as it stands out as a platform that excels in note taking and knowledge organization through its strong linking and backlinking capabilities. It helps me create a network of interconnected notes, making it easier to navigate between related topics and ideas. On the other hand, I complement it with Notion which, thanks to its versatility and wide range of functionalities, provides me with databases, project management tools and note-taking capabilities, all on the same platform. To answer 3) with great frequency. To answer number 5) it is a bit difficult for me to keep up with the updates due to the type of knowledge or subject of interest, which are many.

[nktskr]

Thanks for your response!

Do you feel comfortable to use two apps (Obsidian and Notion) simultaneously? How do you choose which content goes where?

5) Did you tried tagging? If yes, why it does not work for you?

[romerocarlos]

Hi,

Yes, I feel comfortable, it's a matter of making the habit. Everything specifically related to the knowledge of my profession I use Obsidian. For everything else I use Notion. As for the use of tags, yes I use them.

Best,

[Havoc]

Markdown files pushed into git.

Tried a couple other options and decided flat text is the only thing I’ve got long term confidence in

Briefly ran an obsidian setup on top Of that but leaning more towards simple folder structure.

Expecting that to play well with future LLM tech too. Flat markdown text files should ingest well

[eddd-ddde]

What ideas do you have for LLM tech in this?

Data aggregation? Summarisation? Retrieval? Or maybe some ingestion post-processing?

I'm getting ideas of making a service that works as a git origin that you can just push to update and then access in a web UI to get nice features, maybe a project for a weekend.

[Havoc]

Likely just as a glorified smart search. Maybe load the headings into context and RAG the rest. Not sure - not high on my todo list.

I just want the data to be in right format once o eventually get round to it

[rcarmo]

1) Kindle/Goodreads, streaming service watchlists, iTunes playlists

2) Zero times. I do use ArchiveBox for snapshotting interesting articles and searching for them later, but mostly out of fear of some things falling off the Internet (which is increasingly common)

3) In practice, very very seldom. I stopped using delicious and Instapaper and whatever else because search is a solved problem.

4) Mountains of documents and plaintext notes, with file names prefixed with YYMMDD. Seriously, I can track down what I was doing ten years ago in under 5 seconds.

5) The fact that people keep reinventing note-taking apps and none of them do what I need, so I stick with my wiki (taoofmac.com) and my filesystem + Spotlight.

[eddd-ddde]

1. I don't. I suck at organisation. I have books everywhere in my house and even car that are in progress. I just pick one and read.

2. Hmm probably never?

3. Probably never as well?

4. I've tried obsidian, regular plain text files with vim, notion, nothing works for me. Now I just use bookmarks with webtag.io since it's quick. I don't really write notes, just store references.

5. I can never write store and manage any knowledge-base, my brain seems to just not work that way. I find it too easy to just delete things I don't like, until I realise there's nothing left. I'm too perfectionist, to the point that I end up with nothing time and time again.

[GregDavidson]

My life is in OrgMode structured plain text files which are themselves managed by git along with scripts for managing link integrity. This works fairly well but I dream of replacing all of that with relational databases and knowledge representation systems. So far, every fancier system I've used or built has eventually failed, but I'm working on yet another fancy system built on top of Postgres for stability.

[ebiester]

I use LogSeq as the general tool, and I started with a combination of PARA and GTD and have tweaked from there.

What that looks like is that I use LogSeq's daily journals and extensive use of tags and properties. So maybe that looks like (simplified example):

(for list organization)

LATER #towatch “20 Days in Mariupol” #documentary - Sandra's recommendation category:: film

Then, I might have another page with a query like:

{{query (and (property :category "film") (todo later now))}}

but I can also go to the tag page documentary or towatch and have anything I've tagged there. Now, I don't tend to get too deep into watching lists, and stick mostly to non-fiction.

I have a book template and article template that focuses me on key insights, and I just take general notes on what is surprising to me or useful to go back to. I take liberal copies everywhere and clean up later. I don't go back daily, but I find it helps when I want to quickly brush up on something or if I want to refer to it in a way that I can send someone else something. ___

Now, to your 4) most of my lists are more work-focused. Again, I use PARA as my storage basics and I've adjusted as I've went. I blur projects and areas more than Tiago does, and use intermediate documents a lot less. I also use daily journals as a lot of my organization because my temporal memory and search generally lets me remember where to put things, but it does mean I have to get back to people later. The real problem is managing the knowledge base and I just keep everything in daily journal until I need to organize it.

LogSeq does generally use markdown and lists, but the more features you use the further away it becomes useful outside LogSeq. If I ever moved away from it, my search would largely be "grep".

[gxonatano]

1-5. Org mode, with plugins org-roam, org-roam-bibtex, org-noter, and org-citar. Every book I read has an accompanying bibtex file, and a linked note in org-roam. Usually it's an ebook, in which case there's an accompanying PDF as well. Org-roam keeps the book note linked to my other concept notes, org-roam-bibtex and org-citar help keep it connected to the bibtex data, and org-noter keeps my page-level annotations aligned with the page where I'm taking the notes. 2. Org-capture. When I'm on a web page, or in a code editor, or a PDF, I run org-capture and it stores a link to that URL, code location, or PDF page. 3. Very often. 4. Same: org and org-roam. 5. I guess syncing between devices is the thing I find most challenging, but it's not that big of a deal.

[2OEH8eoCRo0]

To answer your questions:

I never manage my reading. Plex manages my watching/listening. I almost never summarize things that I find online and almost never refer back to those summaries. I use a notebook to store knowledge. The most challenging part of managing the notebook is that it takes up limited desk space.

Speaking candidly:

I've tried a number of systems such a Dendron, Joplin, Obsidian, etc. and nothing feels as good as pen and paper. In theory you'd think these systems are fantastic but I've come to the conclusion that taking notes is more than the raw information taken down. With a notebook I feel like I have a mental map in my head of the geography of the notebook, the places different information is stored, how the notebook feels. It's also low friction. At any time I can touch a pen to paper and nothing can stop me.

[stranded22]

Sure - I have trakt for managing what I have watched. Testing Queue for ios for future watching. I use todoist for general to do lists (moved to ticktick but back because of 2FA). I don't really copy quotes any more but I used pocket previously. I use onenote as my filing cabinet - I take photos with phone and upload for OCR. I used to use Evernote but it's got too costly for not enough product and I already pay for Microsoft 365. For professional storage, I have a combination of one drive (work account) and one note (my own one - folder set up for work).

Over the years I have tried a few different tools but didn't get along with Notion etc - I get bored easily and don't want to spend days setting up a system as I view that as procrastination.

[nktskr]

Thanks for your response! Why did you stop using Pocket?

[Jsebast24]

1) I don't organize. I just collect things. I hoard.

2) Every day. Whenever I find something worth copy/pasting.

3) Only once in a while. But when I read back on my saved stuff, I have a lot of fun.

4) Copy and Paste to a rtf file, or anywhere I can paste text and images and add some formatting. Has to be simple, quick and easy to access. MS Wordpad is the best ever, and I'm not a MS Windows fan. Make sure to put a good title and include tag words that can be searched later.

5) That I can't find anything better than MS Wordpad. It has an I-don't-know-what kind of simplicity that makes it a pleasure to use. And I'm not a MS Windows fan.

[itake]

1/ web articles are stored in pocket. Books are tracked in Libby or Todoist. Most others go into Todoist.

2/ rare. I have a bad memory. I don’t even know that I don’t know.

3/ 1-5% of the time I reference a pocket link

4/ Anki, google docs, my blog.

5/ writing takes a lot of time. Even with ChatGPT help

[nktskr]

Thanks for your response!

1. what is the hardest part using Pocket? 1.1 how often do you open pocket to read something? (just approximately)

1.2 Just to clarify: do you keep your Pocket list clean? Or you are using it like a reference list for future?

2. Have you ever caught yourself on though that you can’t find something you read previously? 2.1 if yes, how often? 4. Just to clarify: Do you use anki cards to learn better?

5. could you tell me please about the last time you wrote something?

[itake]

1/ I read from pocket 3-4 days per week. Biggest issue recently is the offline and text to speach experience.

1A/ if pocket can extract the text into a reader view, it will cache it for offline viewing. If pocket can’t, then I don’t get to read the article I was expecting to read on the airplane.

1B/ 20-40% of the articles I listen to while walking or driving, but they only work if the reader view is available, I must be connected to the internet, and the voice sounds pre-2022.

1C/ the search function isn’t great. I think if I pay for it, it will do full text search. But I think it’s dumb that I have to pay for ctrl+f.

1.2 not sure what you mean by clean. It’s not well organized. I have like 5 tags I use around topics that might be useful later, but I rarely archive articles or properly tag them.

2/ it definitely happens maybe twice per year. I use Anki occasionally, but only for deep topics (learning a new programming language, leet code), and not to learn blog content

5/ I write code and slack messages daily. I probably write internal documents one to four times per month. I’m trying to write one blog article per week, but it’s more like one month.

[karmakaze]

In my head. I've tried many things half-heartedly but never seemed to have a payoff to be worth keeping up with. I try my best to internalize new info so they make sense without effort to remember them. I used to draw lots of pictures while sorting things out but that's become very plain-text based these days. Other recipes/incantations are small and of short term value so a scratch pad (many unnamed Sublime Text tabs that I never save) or Google Keep notes are enough. I use browser bookmarks but rarely ever retrieve info with them. A lot of times I'll just remember where I saw something (e.g. HN) and search there.

[JohnFen]

> 1) List Organization

I don't keep any such lists, aside from bookmarks which I keep on a bookmark server that I run.

> 2) Content Capture

Extremely rarely. If I come across something I want to save, I prefer to rewrite it in my own words because that increases my understanding and retention.

> 4) Knowledge Storage

I keep everything (except bookmarks) on a wiki server I run. This includes specialized knowledge, notes about projects and research, manuals, datasheets, etc.

> 5) Challenges

Honestly, I don't think of anything about it as challenging, but if I had to pick something, it's probably the mechanics of using the wiki. But my process is one I've honed over decades and it serves me extremely well.

[karpatic]

I store ALL my notes on charleskarpati(dot)com using .txt files or by using this tool I made: ipynb2web(dot)com

Ipynb content is mostly static content I'd like to share w people. It requires a git push to update.

.txt notes are stored on a php based shared host and managed via a hidden url. I use it for life notes, and todos. It's been working great so I plan on using this approach to add a blog to my site at some point.

My downloads folder just piles up till I sort what I can and dump the rest in my archive. The archive is real and an llm will sort it all out for me one day :) I hope.

[shivc]

A combination of Notion and notecard system that I first learnt from Ryan Holiday has been my go to method for this.

Tons of other successful people in the history have used the notecard system to create a commonplace book.

What I'm not a fan of is people spending hours on setting up these productivity systems

[auc]

I use some mix of Obsidian and Notion.

Obsidian for storing raw information and connections, and Notion for my to-dos/trips/lists.

[interbased]

1) A combination of Trello and the Apple Reminders app.

2) Any time I share an article with friends and want to emphasize a snippet.

3) I usually just Google whatever source I need at the moment.

4) Brain.

5) Too many ideas come up and I don’t end up writing them all down.

[eimrine]

1. Just feeling an urge to consume something right now.

2. All the times, typically during the conversation

3. Any time I want to dig up with my old crap

4. A lot of HDDs

5. Backups: how to not lost the info because of either tremoring hands or the war which is going on.

[nktskr]

Thanks for you response!

1. Could you tell me the last time you faced this? 1.1 What was the most challenging part?

2. How often do you face the issue when you can’t find something you read/watched/heard? 2.1 Do you feel frustration when you can’t?

3. How often it occurs? just approximately

Stay safe, I know what you’re feeling

[eimrine]

1. Almost every evening, last time I feel I've done an accomplishment is few weeks ago when I've realized I mastered vi/vim command language. I wanted to learn it for at least 5 years and when I had several days off I just did it. 1.1 It was like riding a bicycle, the most challenging part were all parts before the first ride.

2. Always! 2.1 I know I can write down something belonging to one of important projects of mine (writing a thoughtful blogpost, collecting samples for future music track, grabbing something from the Internets for the sake of archiving) and if it is important but I have not done the simple thing of growing the work I feel like very frustrated. Any other cases of forgetting are not important for me, but I can always start a new project if I am sure I want even more information grinding in my life.

3. I have a rule of three slots. If I have remembered about some issue once a week, it is the 1st slot. Next time I move the issue to the 2nd slot. Third time is the time I got to solve the issue. I think I have maybe 4 unsolvalble issues per year which are about mismanaging the data issues - typically ends up in reorganizing something in my life, maybe buying something or writing a long letter to someone.

Thank you. But I do not feel like I understood your intent of asking.

[supportengineer]

Google Docs or Apple Notes are good. With both of them there is the ability to export/backup, which you should be doing anyway, so it forces you to get into a good habit.

[RGamma]

I just dump it into dokuwiki. Not ideal right now but at least it's stored somewhere.

[theshrike79]

1) reading lists in Goodreads and pinboard (moving to self-host that one, because the service isn't really stellar anymore sadly), watching lists in Letterboxd and Imdb along with sonarr/radarr, listening list is just my podcast subscriptions in Overcast with a smart playlist ordered so that the most "important" ones are on top.

2) Obsidian web clipper[0] and a Keyboard Maestro powered hotkey that opens up either my daily note with two taps or creates a new Second Brain note with three

3) When I need it. I'm pretty good at keeping an index in my head of stuff I've seen "sometime somewhere". By having everything in Obsidian I don't need to start searching browser/search engine histories to figure out what was that one site with that bit of knowledge.

4) Obsidian Second Brain. When I spend too much time (by feel) solving something, I spend a few minutes writing the solution down along with links to the sources I used. The solution is generalised so that it's not infringing on any employer copyrights.

5) Consistency and not bikeshedding about knowledge base systems too much.

I've used Evernote (one of the OG Mythbusters recommended it like 15 years ago or something). The OCR system was a life-saver in a few occasions. Then they did some stupid Silicon Valley crap and I moved between OneNote (amazing if you have a laptop with a stylus), Emacs (org-mode bikeshedding took way too much time), Joplin (open source, but the storage format is meh), Notion (pretty and the DB stuff tickles my nerd brain the right way, but too slow for my needs) and maybe a few others.

Now I've been using Obsidian daily for 6 months and even finally bought their Sync service, which works super fast - the free ones aren't bad and iCloud syncing works well too, but the official one is just that bit better. In the end it's just a bunch of files in a directory structure so I don't have to worry about vendor lock-in. Replacements Omnisearch, Templater and QuickAdd are the only essential plugins I need if I want to move to another platform.

[0] https://stephango.com/obsidian-web-clipper

[uslic001]

I have been using Evernote since it came out.

[david38]

Logseq