|
|
|
|
|
|
by rqtwteye
698 days ago
|
|
|
That seems the way to go unless you have insane amounts of data. We had some could guys write a pipeline in AWS for us with several steps connected by lambdas that get triggered at each step. The damn thing is just not reliable. It skips some changes or sometimes doesn't work at all. I am sure it can be made reliable but I am beginning to be very skeptical of all this complexity. I think we could write the whole pipeline in 300 lines of Python with the same results but somehow this is not "cool" because it doesn't scale. Never mind that the complex stuff doesn't scale either |
|
|