| I tried to give this an extensive run through as I think I'm in your future target market for this (content marketing professional in a technical vertical). I used it to evaluate the text of this article [1], which is a pretty typically broad business type article (aka not "Here's how to use XYZ.js"). Feedback - The side by side view feels like a developer thing. Like a markdown editor or HTML/CSS preview editor. I'm not sure it serves a purpose here as you only ever look at the right side after you paste in text. - The "rating" bar for "fit" isn't prominent enough and doesn't convey enough useful information. - Despite this being an article about software bugs, Kerberos, etc. it was rated as only "average" by every single target group. - Some of the substitutions were just wrong. Example, it suggested "attacks" as a replacement for "defenses." - Some of the substitutions were the same? Example, it suggested "security" for "security." - I think the coloring gradations have some meaning in the drop downs, but I'm honestly not sure exactly how it works (pinker is closer to what your target?) - Target categories are odd (I'm presuming that this was more just about where you pulled training data from than anything else), but I'd encourage more verticals. Overall, I think this is an interesting experiment but doesn't seem useful in a professional context. In particular, I think there may be an assumption at work here that it's the specific word choices that define the differences in writing for these various groups, when in fact it's much more the approach, what's considered, or what's left out that really makes the difference. 1 - https://www.varonis.com/blog/zero-day-vulnerability/ |
The reason why technical articles -- or in your case, very domain-specific ones -- get average scores is that the model was intended to be used for recruiting and ad-copy. So at the moment, it works on a common vocabulary for all groups, whereas one reason your article might be intriguing to programmers is because it discusses a lot of domain-specific terms (e.g., discussing 'exploits' and 'network security'). If I modified the model to consider more domain-specific terms, I think the article you provided would rank very highly. There are other reasons as well, like the pithiness and clarity of it, but I do think the vocabulary is an important part of it.
From the little beta-testing I've done, the people who find it most useful are -- for example -- people without a CS/medical/accounting background who have to interact with, recruit, and sell to CS/medical/accounting people. For those folks, I would say there's some value, even at the vocabulary-level, though I agree that there's much more to writing style than just vocabulary.