Hacker News new | ask | show | jobs
by t0astbread 2273 days ago
Is a 400 thousand word text file "very basic"?
3 comments

No, the "very basic test" is not having a 10 second latency in a text editor. Come on, this was a solved problem in the 90s.
If it was a solved problem in the 90s, can't you just use a text editor from the 90s? No need to bash on this project just because it doesn't fulfill your use case.
Because there's contemporary Markdown editors that do work as expected. I didn't bash this one, I asked.
What editor did you end up with?

Modern editors are fairly complex. Splitting a bitbuffer and showing ascii line by line is simple. But modern editors have to deal with unicode, where some characters (surrogates) are two chars long, and can be followed by modifiers such as skin color of emojis. Then some characters are wider then others. So a modern editor must first parse the string encoding, then parse the language (markdown) for further formatting and coloring or building a WYSIWYG, usually on every key stroke. The expensive part is showing the text on the screen, generating the fonts, where old editors didnt have fonts.

I'm using Typora, which works pretty well for this use case. Open source, cross-platform, very nice. I'm in no way affiliated with it, I just like it :)

I get the parsing difficulties. Generally, at least for markdown, pressing a character should affect only the line (paragraph) that contains it, so worst case scenario, you need to parse the entire paragraph. Worst case scenario, you need to re-render whatever fits on the screen. I refuse to believe it's not possible to do this efficiently with 2020 hardware, and I have proof, in the form of editors that work.

If it's 2020, I try to append a character to a 400k file I have open, and this takes multiple seconds, it's because someone implemented Schlemiel the Painter's Algorithm, not because the hardware is slow or the file is too big.

Yes, it is. This is a quite typical size of a book. Typically novels have about 100 thousand words, but four times that size is not at all uncommon.
In the context of a book, yes. But I mean in the context of an online Markdown editor. (Or even a Markdown editor at all. Can you actually write books in Markdown?)
Rust book is written in Md https://github.com/rust-lang/book as well as plenty other books that I can't remember from the back of my head.
> Can you actually write books in Markdown?

Yes, you can. The "Programming Perl" book by Larry Wall was famously written entirely in "POD" format, which is just like markdown. I don't know if it was split into several files.

I wrote the novel I'm referring to, and a Computer Graphics textbook (with images and equations and stuff).
That's untrue. Novels are usually in the 90-110k range, depending on genre. If you're an established author, especially in fantasy or sci fi, insane doorstoppers like that do occur, but 400k is an outlier even for sff.
Here you have a list of very famous books with their word counts. Quite a few of them have more than half a million words. Thus, it is reasonable to expect that a text editor is able to handle that size. After all this is a tiny amount of data compared to what has to deal a program that does image or video processing, so there's 0 reason for a text editor to be "laggy" when dealing with just a few megabytes.

https://blog.fostergrant.co.uk/2017/08/03/word-counts-popula...

Seriously, that’s stupid. You’re collecting outliers. Unpublished writers trying to hawk 400k word novels are for the most part delusional. Every single piece in your list that goes into that territory is something written by an established author.

Though, to be fair, an author organizing a work of that size in a single markdown document has bigger problems than key lag.

That doesn't exactly make your point.
Books should not be written in a single text file, though. That is asking for trouble.
A good editor can handle them without any trouble. I just downloaded Moby Dick from Project Gutenberg and opened it in Emacs. It opened instantly, I could immediately go the end (line 22333), scroll around, count the words (there are around 222617), count the ocurrences of the word 'whether' (there are 91, the last of which is on line 21769), make a change somewhere in the middle. All of this is instant.

You can do all that instantly in Vim too (... and I just checked to make sure). A megabyte of text is just not a lot of text for a good editor.

Why not? Beacuse of bad text editors? Or is there some fundamental reason about that?
Because it is extremely hard to work with it?

Same reason why we do not put programs in a single source file.

Why is it extremely hard to work with the whole novel in a single file? Because most software for writing prose is not up to the task?
break the novel into chapters? Each chapter in a separate md file?