[jandrewrogers]

The spatial data structure literature is quite incomplete, and there is a lot of confusion about what to use when. Many of the issues with spatial indexing performance at companies I go into is that they are doing it wrong, not that they necessarily have a fundamental problem.

It should be pointed out that R-family data structures should only be used for data sets that are (1) small and (2) relatively static. They scale very poorly in a number of ways. The primary reason they are used in databases is that they can handle interval (i.e. non-point) spatial data types without the possibility of pathological space complexity (bad when talking about databases) and fit B-tree indexing models adequately, which that software understands.

Quad-tree variants can scale well for point data types but are often terrible for indexing non-point data types due to pathological space complexity.

Grid-file variants are the canonical structure for large-scale spatial data sets if the data is static. However, the number of companies implementing these correctly at scale is approximately zero in my experience.

For general, scalable, online/dynamic spatial indexing structures, there is another family of algorithms and data structures (adaptive spatial sieves) that are basically ignored in the literature even though they were first described in 1990, albeit in a not very useful form at the time. If you are doing petabyte-scale real-time indexing of polygons at extremely high rates, this is what you would use but little is published about them because most modern variants did not come out of academia.

And the most advanced algorithm family for indexing point-like data is not in the literature at all.

(Background: my day job involves extreme-scale, high-performance spatial indexing software and I invented a few spatial indexing algorithms back in the day that are still the state-of-the-art in their respective algorithm families.)

[rdtsc]

> And the most advanced algorithm family for indexing point-like data is not in the literature at all.

So what is it? Is it patented, secret(classified)?

And how do you know it is the most advanced without a peer review?

BTW "adaptive spatial sieves" on Google search produces exactly 0 results. Is there another name for it.

Spatial indexing is interesting and complex but not exactly rocket science, there have been a good number of people thinking about. It is hard to believe there are general approaches there that haven't been thought of yet. But again, but I am not an expert, so I could be wrong.

Can you write some more about it, I am very curious.

[jandrewrogers]

What makes spatial indexing different than other areas of computer science is that its fundamental operands are interval types (like hypercubes). Almost all other areas of computer science focus on solutions that only work on point-like data and so many idioms and intuitions computer scientists use are not actually valid when dealing with spatial representations. Data types that require no less than two integers to describe break the assumptions of computer science developed for data types that require one integer to describe.

Oddly, there are almost no people working on the computer science of spatial representations and indexing. That was how I became involved in the first place; I needed solutions to specific problems for which no active research was being done in academia (and I was talking to people like Samet at the time). And to this day, academics still aren't doing any interesting work on spatial structures.

The algorithms are not classified, just not published. There are a couple patents out there on spatial sieves -- I am the inventor on the first practical one -- but more sophisticated and advanced variants exist that will probably never be patented. Widmayer (respected CS academic) was the first to propose this approach to the problem of generalized interval indexing but a useful algorithm was not discovered until my work in 2007.

The point indexing algorithm family I mentioned has never been patented or published anywhere but was based on the development of a novel theoretical CS construct that allows the expression of polymorphic space-filling curves. These are essentially adaptive to the information theoretic properties of the high-dimensionality data set but still embarrassingly parallel and distributable. I don't think these are being used in production anywhere (yet) but they've been around for several years. Very cool but the number of people that grok them can be counted on one hand.

All modern research on computing with spatial representations is being done at one of a few companies, which is why nothing is published. People like me, and I haven't done the pure research in years, don't have time to generate hundreds of pages of fairly deep content. Consequently, learning advanced spatial indexing is fairly prohibitive; you have to figure it out yourself, which is far from easy, or be at one of the handful of places where they are using it.

[kastnerkyle]

I found some related papers for others who are interested:

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.56....

http://bmi.osu.edu/resources/techreports/ahmedBokhariV21.pdf

Also a book:

Space-Filling Curves: An Introduction with Applications in Scientific Computing

by

Michael Bader

[jayvanguard]

Thanks for the insight. I've seen similar patterns in areas such as recommendation systems. There are active research communities but really all of the detail work and pushing the envelope happens in private companies.

The financial incentives drive the pace. In the future I'm sure it will become more commoditized but right now the practical nature of integrating with large shopping, coupon, and marketing systems and scaling for millions of users place the progress mostly in the domain of industry not academy.

[rdtsc]

Fascinating, thank you for taking the time to reply!

[ehsanu1]

Googling for it without forcing a phrase match returns some limited results. Here's a patent for a "Spatial Sieve Tree", which is apparently invented by the parent, jandrewrogers: http://www.google.st/patents/US7734714

[aw3c2]

In that case I am glad it is not covered in literature. Patenting algorithms is bad.

[jandrewrogers]

Yep, that was the first practical spatial sieve (previous variants had pathological spatial skew sensitivity). Most people that use them use more sophisticated variants. Even with that version, which is simple, many implementors do not understand the technical subtleties well enough to produce a non-broken implementation.

[kaib]

> And the most advanced algorithm family for indexing point-like data is not in the literature at all.

Don't leave us comp geom geeks hanging.. Which one are you referring to? :-)

[poiuytrewq88]

Is there a way we can get in contact? I'm really interested in your work.