I've worked with several companies that try to parse things in PDF documents, extracting tables and paragraphs etc. This is actually challenging because a PDF is a large bag of words and fragments of words with x y positions. There is a particularly popular word processor that emits individual characters. Just determining that two fragments are part of the same word is challenging as is detecting bullet points, etc.
The AI approaches are definitely still worse than human-written rules. I can infer - and I've chatted with the devs to confirm - from the quality of the text and table extraction whether the company is using a modern NN approach or someone has sat down and handwritten some simple rules that understand indents and baselines etc.
Yes exactly, table comprehension is a logic driven, non-local inference problem. Critically, its the non-locality that trips up common machine learning techniques. I wrote an approach using mixed integer programming once[1]
I had to check we hadn't worked for the same company! Yeah, text extraction and layout analysis from PDFs is a super interesting challenge and still relatively underdeveloped. I'd put table detection at about the hardest challenge in that field.
Really really interesting, hadn't seen pdfpig before!
In the delicious pics of results I can see the bullets treated as one column and the paragraphs for each bullet point actually run together as single chunk of text?
What do you think about tackling bullets and indents?
Thanks! I think there's definitely room for rules-based enhancements to the underlying algorithms.
My area of work on the project has been the core file-reading and file-creation stuff so I haven't had much of a chance to review the layout algorithm performance across documents.
Having been working on a purely rules-based approach in a private repository for a side project it seems like the algorithms the contributor has implemented get you a lot closer to the correct result than starting from rules alone but it definitely feels like adding some context-aware rules would get all the way there. I'm not sure whether they'd be in scope for the layout analysis project itself or someone could take the open-core and extend it, as I was attempting in my side project.
It depends. We do commercial pdf and scanned information extraction as well as table detection for line items for invoices, receipts and remittance slips. We have been successfully using rule-based system for years but are mixing in deep learning now. I also know a about 5 other companies competing in the same field. So, I wouldn't say it is underdeveloped.
Referring to the above poster's "non-locality", are we talking about denormalization of formatting? Is there a way to "normalize" PDF structure? Calculate margins or common formats beforehand to normalize?
I believe the reference is to logical locality, specifically in the case of PDF that transforms and such are essentially atomic and there's no real boundary layer in which you may say "transform X and transform Y are equivalent within this local finite domain."
There really is no real differentiation between formatting and content in a PDF, so it's not possible to truly separate them.
I'll try my best to answer but I may be misunderstanding the question.
The current layout analysis algorithms don't do much normalization as far as I'm aware, the Recursive-XY Cut algorithm uses page level font-size information [0] to tune parameters but it doesn't infer a common structure or format either as an input or result.
The aim of most layout analysis algorithms is to produce classifications for regions, e.g. paragraphs, titles, lists which I suppose counts as denormalizing the document? Arriving at those classifications generally relies on first splitting the document into sections or regions and then classifying those regions. So far the implemented algorithms mainly focus on the first step, splitting a document into discrete regions. An example of the second step using ML approaches to classify those regions by the same contributor can be found here [1].
With the rule based approaches I've been experimenting with you can use certain information from the common producers to normalize certain features. For example line spacing and font size have a well defined relationship, as do whitespace size and font size (though this is a fuzzier relationship and goes out the window entirely for justified text).
For the values following the subheading "Institutional Investment Manager Filing this Report:" the left hand column are keys for the right hand values.
At the bottom of the document there's a table containing the columns "Form 13F File Number" and "Name".
Now you could use a couple of rules to infer the difference between the key-values and the table:
1) The keys in a key value list end in ':'.
2) The keys in a key value list have a different font/color to the values.
Both of those rules hold true here but not in all or even most documents. For this reason you need to use the whole page to deduce the type of these sections, rather than immediately surrounding features/pixels as an ML algorithm might.
It was really shocking when I learned that the way pdf works is as you describe, literally fragments of text with positions and essentially no semantics.
I think a lot of folks find this out as I did, when they run into a project where they need to extract info from pdf documents. Without knowing anything about pdf, one can easily assume that it will be possible to do things like "can't we just extract some semantic structures like headings, tables, etc"... but nooo, it don't work that way!
Discovering the true nature of pdf is major WTF moment because we're so conditioned to expect documents to have a semantic structure. It's hard to understand how a standard can take the exact opposite approach and be so successful.
It's so successful precisely because it doesn't have semantics. It is a print format with one goal: show the output as desired. Semantics only confuse and limit this goal.
Imagine how bogged down and limited vector graphics would be if every element had to have semantic meaning? "This line connects the <body> of the <car> to the 13th <spoke> on the <wheel>".
ISTM semantics could have been added as a supplement to PDF à la microformats for HTML, which wouldn't have hurt anything. It's easy for processors to just skip some well-defined tokens. Of course, few producers of PDFs for public consumption would have incentives to do that, so it probably never would have taken off...
Yeah, so by eliminating semantic concerns, pdf has achieved transcendence as a document format.
But by its nature text is intrinsically semantic. I am just surprised that a document format utterly free of semantics has lasted so long. Perhaps because we (as people and organizations) can't agree on the structure of documents?
Another way to see this, perhaps, is as the failure of the promise of xml and the ecosystem around it? In the late 90's many of us thought that all documents would be xml for content and styling would be through xslt or even it's big sister, xsl. Well, THAT went nowhere despite all the W3C meetings and papers.
It's interesting you brought up graphics as an analogy. It's true that you can have graphics which are literally just lines and that's adequate for many needs. However, modern CAD drawing systems increasingly use notions of 2D/3D objects and a disciplined series of transformations. They call it "parametric modeling" and it's where where all drawing consist of a series of transformations that can be represented in a timeline. I suspect modern parametric model CAD can very much be semantic.
Books have about the same amount of semantic information as pdf. It's probably just habit.
I think more than lack of agreement, it's just that there aren't really universal document structures. There's relatively useful chunks like paragraphs that are more or less universal (at least for a given language), but those don't need much structure to be clear.
It isn’t in the interests of word processors to round-trip through pdf. If you look at the PDFs the mainstream word processors generate, you see some of them actively trying to stop text extraction. It’s like an obfuscation arms race. They include white-on-white text, and jump all over the page positioning text so no whole words occur in the source etc. Sad but true.
Having worked with OCR products doing table detection for years, simple hand written rules cannot solve the general case. It can work for specific documents, but if you want to be able to handle any document it's just not accurate once you include non-gridded tables.
With collaborators at Adobe Research, my lab published a paper recently showing how to do table reconstruction from infographics (e.g., bar charts) using deep learning [1].
While it isn't the sexiest project, I've had a number of companies reach out about the project. Human written rule-based approaches are pretty bad at the task, and even humans doing it manually aren't great (likely due to sloppiness).
I've found that when PDFs are produced by a single entity for a particular purpose, I can automate this pretty well with a loop and some regex... maybe I've just gotten lucky?
For what it's worth, at my previous place we built a YOLO based model for detecting paragraphs/tables/headlines/page layouts mixed with traditional rule based OCR/layout detection.
This capability has a lot of value for accessibility. Recovering the table structure for logical presentation allows navigation by blind users as well as users who are not using a pointing device.
It is disappointing just how haphazardly most PDFs are structured. Too many of the PDF production tools remove all document structure metadata or fail to include it by default.
This is a very interesting field, and PDFTron has been doing similar work with ML and table extraction as part of our document understanding platform. We've made pretty good progress over the past year -- you can try it on your docs here:
I applaud the Nanonet folks for starting a business around AI API's, and it seems clear there's lots of value to unlock with solutions like these.
But does anyone have insight how hard is it to be in a space where all the big cloud providers seem to be offering very similar products? Can you survive by focusing on a niche segment? Is the market growing so fast that there's room for multiple companies offering (roughly) the same thing?
I think it's pretty hard and even the big cloud providers dont necessarily have a perfect solution. It's not a particularly creative idea to come up with I think. I've thought about making something similar as a product but I'm kinda glad I didn't
I can see the use-case and potential for ML in exfiltrating tables, but I'd be worried about the potential for decision-making mistakes in environments the author identifies, such as finance.
The example of TableNet using deep learning for table extraction on top of tesseract for OCR means two layers of ML, either of which could individually introduce pathologies without human oversight. It reminds me of the photocopier that changed numbers for you - https://www.theregister.co.uk/2013/08/06/xerox_copier_flaw_m...
If an ML engine was trained to be able to do things like look for totals and sub-totals in numerical tables and flag errors in summation, then that would clearly add more value in parsing for moderation (the use-case described at the end). But that doesn't seem to be something that's yet... on the table.
It looks like it's not quite the same thing, in that it identifies Excel values that should be formulae. It could be used in a pipeline with spreadsheets extracted by ML/OCR to reverse-engineer formulae though, which is an interesting prospect.
There is a lot of data locked up in pdfs but even more so in images.
I would like to have a neural net that can give me the data from a chart in an image. I have a hunch image segmentation NN's are able to do this because the size of the surface has a predictive value in the causal relationship. Artificial data could be created at scale with the Google Sheets api.
Also in the extracting-structured-data-from-PDFs solution space, there is Parsr which was recently posted on HN: https://github.com/axa-group/Parsr see https://news.ycombinator.com/item?id=22035258
It's based on a pipeline of various js modules and pluggable backends (e.g. tesseract, GCP cloud vision, Abbyy API, etc.)
For tables with numbers in them, it worked pretty well, but I'm yet to find a tool that can parse/understand documents where the entire page is a table layout with lots of merged cells. I think even for humans it's hard to understand the structure in those cases...
I was very impressed with "Camelot" (https://camelot-py.readthedocs.io/en/master/). My impression was that it extracted maybe 80 or 90% of the text properly, far better than anything else I had tried.
Table extraction has been a feature of better OCR programs for at least a decade. It's easier than the OCR part. Look up "OCR table" for examples, products, code, papers, etc.
You'd think that until you try them with tables that contain empty cells that you still need recognized or tables that span multiple pages. I wouldn't say this has been solved for a decade.
I wish it was, but it isn't. There are various kinds of tables, that may have delimited lines or not, or they may be unaligned cells, each showing a key and a value... If you actually have in mind some solution that works well (either a paper, a github project, a commercial product) I'd be eager to know
The AI approaches are definitely still worse than human-written rules. I can infer - and I've chatted with the devs to confirm - from the quality of the text and table extraction whether the company is using a modern NN approach or someone has sat down and handwritten some simple rules that understand indents and baselines etc.