Hacker News new | ask | show | jobs
by agentgt 3818 days ago
Well with out extensively using your library I can only guess a couples of things must be true for not needing maintenance:

* Your library has a very limited set of capabilities

* Your library is not extensively tested on all supported databases all the time and thus bugs are not being found.

Also I find the claim of streaming support sort of disingenuous or at least the word streaming misleading. Real streaming that is required for reactive like programming (ie reactive-streams) is not supported by the JDBC drivers because operations are bound to thread/connection. If you are just talking about Iterator like streaming than you do realize you are at the mercy of the database driver itself. For example Postgres and many other databases will almost always preload a certain amount of the ResultSet regardless (and in my experience a rather large ammount).

1 comments

Correct, iterator-based-streaming, as java6 was the norm when I wrote this. And yes, you're obviously at the mercy of your JDBC driver. Though I'd argue if your results are fitting into PostgreSQL's default preload size (whatever it may be) you don't really need to worry about it. But I promise you it does stream. (TS is not known for it's small datasets...)
I think you understand this but its more than just handling large datasets memory friendly. True streaming would allow you to keep your connection pool happy. If you stream a large dataset you have to keep the connection open (lets say for a web request which is the common case) and you can't reuse that connection till you retrieve the entire dataset (or I guess read a subset and stop). For batch processing this ok but for web requests this is generally not ok. Asynchronous drivers are push based and are analogous to NIO HTTP like Netty (some non-JDBC async drivers I think even use Netty). But I'm going to gather you understand that and/or either using a very sophisticated pooling technique/drivers.

So if I block too long while reading an iterator like object because the client is taking to long to read... I think you can imagine what happens. This is why so many of the JDBC wrappers (such as Spring JDBC and JDBI ) do not return iterators or at least do not advertise it as an awesome feature.

Our pooling was pretty shitty actually, but it didn't need to be fancy as 95% of our code was some sort of batch processing (as you guessed), after which the JVM terminated. But yes, highly tweaked JDBC drivers over all.

Hibernate has iterator methods, but I recall (in 2010) it still loaded the entire result set into memory, with a //TODO comment. I remember thinking "W...T...F..." I can't tell you how many -Xmx16G (or 32/64) flags I deleted...