Hacker News new | ask | show | jobs
by kipukun 848 days ago
Do you think you'll add local file support in the future? Also, do you have any plans on making the reading of a source parallel? For example, connectorx uses an optional partition column to read chunks of a table concurrently. Cool how it's abstracted.
3 comments

I have just released v0.1.2 which supports CSV destinations with the URI format `csv://path/to/file.csv`, hope that's helpful!
I am working on file support right now as a destination to begin with. I believe I should get local files as well as S3-compatible sources going by tonight.

Reading the sources in parallel is an interesting idea, I'll definitely take a look at it. ingestr supports incremental loads by a partitioned column, but there's no parallelized partition reading at the moment.

Thanks a lot for your comment!

I second this!