Hacker News new | ask | show | jobs
by cstivers1978 3696 days ago
I really like the idea behind Concord. No longer have to stress about Hadoop/YARN platform, and I can use the language of my choice (haven't used JVM-based stack in ages).

Is there documentation on adding more input/output sources?
2 comments
followtherhythm 3696 days ago
If by 'input/output' sources, you mean computations that will pull or push from an external system such as kafka/cassandra, there isn't any documentation currently. However we have written connectors to Kafka and Kinesis. You can check out the Scala Kafka Source here [1]. Internally we are working on a high performance Kafka Source in C++ (based on librdkafka). At the moment this source can push records downstream at a rate of > 350K QPS.

[1]: https://github.com/concord/concord-jvm/tree/master/concord_k...

link
cstivers1978 3696 days ago
I meant connectors. Thanks for the pointer.
link
amn0408 3696 days ago
Concord's FAQ has a section on getting data into the system [1]. An example I've used is processing xml files at scale. A short input function digests the xml stream from our servers and another short output function creates a new document in Elastic.

[1] http://concord.io/docs/faq.html#how-do-i-get-data-into-the-s...

http://concord.io/docs/faq.html

link