Jet looks really cool, I'll but I think we'll stick with Flink for the time being.
I say this as someone who got burned hard with weird bugs using Hazelcast 2.X as distributed lock manager. I'll have a hard think before adopting any part of the Hazelcast ecosystem in the future after that experience. When the analysis of Hazelcast 3.x was posted on jepson.io (https://jepsen.io/analyses/hazelcast-3-8-3) I had a good laugh because a number of issues that were exposed, we had seen in production in older versions. Locks claimed on both sides of a cluster partition, locks never getting released when a node crashed while running, memory leaks, etc. In the end, we had the option of upgrading to 3.X or dumping it entirely in favor of ZooKeeper + Curator. We chose the latter and haven't had issues with our locking system once and nobody has gotten paged in the middle of the night because of a ZooKeeper issue.
After that experience, I'll take every guarantee made by Hazelcast with a giant grain of salt. I've heard good things about later versions so I'm going to assume things have improved but I implore people to look very closely at solutions like these and in particular, the guarantees they make before picking any of them.
The truth is the original Hazelcast replication protocol was not a good fit for some data-structures.
We took the analysis seriously. I know every project and vendor claims that. Here is what we did in recent years:
2. Added a FlakeID generator. This is on the opposite side of the consistency spectrum - it's a k-ordered Available (wrt CAP) ID generator.
It won't generate duplicates even when there is a split-brain. See: https://docs.hazelcast.org/docs/4.0.2/manual/html-single/ind...
I remember a presentation by Kyle Kingsbury (the Jepsen guy), where he talked about the various inconsistencies he found in databases. He mentioned Zookeeper and said "I found no issues, which I consider a personal failure" :-)
spark, storm, flink, beam, hazelcast... and then there are all the vendor locked choices confluent, kinesis, azure probably has something in that space to
The whole cloud computing space got me confused. I don't know what horse to bet on and don't have the time to get familiar with every new framework. Is this the new javascript world? If so I'd like to skip the next couple of years until we found our react equivalent.
edit: Not to be read as an invitation to discuss how react is not the de-facto standard of ui web frameworks
Distributed Systems (the OReilly trout book) has a nice overview of the streaming landscape (the first four you mentioned). The first several chapters being a general tech background of stream processing: events, watermarks, redundancy etc.
Yeah I have that one too and yes it's a good general overview of the broad space and yes there is some overlap. If you're thinking of selecting a streaming solution in particular--and it's definitely not for everyone--Streaming Systems is more in depth into the workings and tradeoffs and might be helpful understanding your problem. I'd check the TOC on the link above to decide.
This is very true. Stream processing is both old and new and I think it takes time for technology like this to really mature. There's currently a standardisation effort around Streaming SQL which may bear some fruit, but probably still many years away. Right even if you want to use some standard language like SQL to describe streaming queries there's differences in each tool both in syntax and semantics.
The paper is part of the standardization effort but is not the final authority on the process. It is a very good reference on how to approach streaming SQL, even though the Jet model will have a few differences to the presented paper.
p.s.: sorry for the late reply (somehow I wrote and didn't publish).
Inferring velocity vectors would be very useful for analyzing traffic flows, impacts of lane widening/reducing, signal timing, ML for adaptive traffic management, etc.
None of those things are nefarious and don't necessarily provide additional knowledge, as long as care is taken to fully deanonymize and fuzz start/stop/end locations of trips or associate trips together.
People agree to provide this information to services like Waze etc for exactly these tasks.
Hmmm... I think there are only a handful of nefarious uses for this technology but a plethora of real-world applications. Almost all of the nefarious uses revolve around GPS and individuals. If you take that out, the set of applications is enormous-1.
It’s strange to me that people read something like this and infer the absolute inverse of the actual situation. That is definitely a “thinking fast” reaction.
Would you be interested in giving them your phone number? how about your contacts as well? maybe your own voice? apps have you've installed? and when you're not using the app, keep them posted on your location while you're at it too.
> Them selling my data to others without telling me is unethical.
I know they do, but a lot of their users don't which is unethical. But this comment was about the Jet database listing a use case about speed vectors. There is absolutely nothing unethical about that use case nor technology. All the other issues you list, absolutely, but those are not things Jet does.
Unfortunately it is under a company known for shutting down products and that spies on you. It is better to use alternatives like Open Street Map, or if there was absolutely no other option, Apple Maps.
The point is that Jet can track several million distinct keys, even on a single machine, and finding velocity vectors boils down to linear regression sliding window against two FP variables.
If your concern is why you would specifically want to track locations, the answer is that there are plenty location-based apps that track locations with user's consent.
Here are some examples where user consent is undisputed: ride hailing, bicycle rental, street navigation, running/biking/sailing contests, location-sensitive searches. These are the kinds of applications for which Hazelcast Jet offers easy scaling into millions of users.
I'm a bit surprised all these systems continue to be built on the JVM. For these sorts of tasks I'd expect something without a VM like Rust to be a better choice
The license is meant to prevent service-wrapping by cloud providers, other than that it doesn't have any implications for standard usage. The core library / server is Apache 2 and the rest of the connectors are community license. You can use and embed both the core module and the connectors for free.
Beam is just an API layer with different backing implementations. But you don't typically use Beam to work with Jet, instead you use its own Pipeline API which is mostly like Java Streams. Jet will also soon get an SQL API.
Very cool! Is it possible to mix apis in a single project with Jet Beam Runner? This would make it easier to port Beam projects to Jet, as the migration could be progressive.
Do you mean for a single job/pipeline? This wouldn't be possible at the moment. Our current focus has shifted from Beam a little bit - as we found out the beam threading model didn't play nicely at all with Jet's green threads (there is no way to distinguish between blocking and non-blocking calls).
While Flink is a fully-featured stream processing framework I think there's some notable differences. Off the top of my mind:
- Flink uses Zookeeper for metadata and coordination, Jet
doesn't require any external systems for resilience.
- Flink uses RocksDB and HDFS for checkpointing/snapshotting, Jet stores it in distributed, replicated in-memory store.
- Flink allocates operators to slots, while Jet uses green threads/cooperative multi-threading. This means you can run many concurrent streaming jobs on the same cluster, with very low overhead.
- Jet is basically a single, self-contained JAR. It's all you need to run a production-grade service (+ some connectors, if you'd like)
- Jet can scale up/down with very little friction. You start a couple of processes and they will form a cluster automatically. Kill a couple of the processes, and the cluster goes on.
That said, Flink have a great set of overall features, especially around persistence and huge states. This is another area we're currently investing in as well as SQL support.
> Flink allocates operators to slots, while Jet uses green threads/cooperative multi-threading. This means you can run many concurrent streaming jobs on the same cluster, with very low overhead.
How does the shift to cooperative multi-threading change the way that the cluster is used? In the "slot" approach, Alice and Bob can run concurrent jobs with relatively little coordination needed to "share" effectively -- e.g. they might use different branches of the same shared repo. In exchange for the lower-overhead, does Jet's approach require that multiple use cases are more carefully planned?
This is indeed a question that we get asked a lot. We have so far not though about adding more advanced scheduling capabilities for the cooperative threads. With the slot system, if you have 48 core available in the cluster, and running 8 jobs, each job will only use 6 cores each. With cooperative threading, each job runs on all the 48 cores. We have tested something like 5,000 concurrent jobs on same cluster, but essentially they may be competing for the same resources, so you'll need to do your capacity planning accordingly. Simple way to work around that would be to create separate Jet nodes (a Jet node is very lightweight) so you could have separate execution pools.
It's not very different than normal SQL. Imagine that instead of a finite result set, you instead have a never ending result set. You can also roughly map operations like windowed aggregation into SQL with some additional syntax. This paper gives a pretty good overview, even though we don't fully agree with the model presented here: https://arxiv.org/abs/1905.12133
I say this as someone who got burned hard with weird bugs using Hazelcast 2.X as distributed lock manager. I'll have a hard think before adopting any part of the Hazelcast ecosystem in the future after that experience. When the analysis of Hazelcast 3.x was posted on jepson.io (https://jepsen.io/analyses/hazelcast-3-8-3) I had a good laugh because a number of issues that were exposed, we had seen in production in older versions. Locks claimed on both sides of a cluster partition, locks never getting released when a node crashed while running, memory leaks, etc. In the end, we had the option of upgrading to 3.X or dumping it entirely in favor of ZooKeeper + Curator. We chose the latter and haven't had issues with our locking system once and nobody has gotten paged in the middle of the night because of a ZooKeeper issue.
After that experience, I'll take every guarantee made by Hazelcast with a giant grain of salt. I've heard good things about later versions so I'm going to assume things have improved but I implore people to look very closely at solutions like these and in particular, the guarantees they make before picking any of them.