[yorwba]

This project is just funky OCR, i.e. "offline" handwriting recognition that operates on the pixels of the final image only. That means it works on scans, but can't take stroke order information into account.

What you're talking about would be "online" handwriting recognition, where timing information about each stroke is available.

[millimacro]

Yes, that's totally correct! The current version of the plugin supports only so called "offline" HTR, which operates on images. This is ultimately determined by the underlying machine learning model.

I have developed another model however (based on a somewhat recent Google paper by Carbune et al. 2020), that operates on pen dynamics and thereby implements online HTR, see here:

https://github.com/PellelNitram/OnlineHTR

This model is open-source as well and will be part of the HTR system for Xournal++ in the future. Feel free to give it a try yourself locally.

One question that has been bothering me a long time and prevented online HTR so far for me is how to find text on a page in temporal domain (i.e. in online domain and not offline domain). If you have any ideas on that, please do let me know as I would greatly appreciate that! One possible way is a transformer model - but again that feels a bit overkill and introduces a context length.

[Qwertious]

Well, that explains why I could never find a decent stroke-order-aware HWR system that wasn't a service. Sigh. What idiot invented this terminology?

[millimacro]

Yes, you're right, stroke-order-aware HWR are hard to find. One reason for that is the lack of good datasets for machine learning model training!

As such, my stroke-order-aware attempt over at https://github.com/PellelNitram/OnlineHTR/ uses a dataset from 2000 with around 12,000 samples. Contrary, the internal Google dataset is reported to feature around 16,000,000 samples :-D.