Hacker News new | ask | show | jobs
by wavemode 917 days ago
I remember working at a company where we had a backend web framework that was essentially a DAG. You could use it to create API's where the caller simply specifies what they want, and the server figures out, through graph theory, what series of processing steps and API calls to other services should be performed (some of which can have dependencies on the results of other API calls and/or processing steps), and the optimal way to parallelize them.

It struck me as being both a very intriguing framework, and also exceptionally over-engineered for the kind of problem it was solving (since an async-await architecture gets you all the same benefits).
2 comments
sa46 917 days ago
I worked with a framework like that inside Google in Java. Was okay. You’d write a bunch of methods and the framework would wire up annotated method parameters and return types. The main benefit was improved tail latency since the framework would traverse the graph and run the methods with optimal concurrency.

Same downsides as runtime dependency injection frameworks like Guice—the framework would explode if it couldn’t connect the graph.

Some other upsides: you could decorate methods with all sorts of distributed goodness like retries and hedging. The approach also enable reusable methods that you snap into your server.

link
aleksiy123 916 days ago
Producer graphs?

Maybe just me but I find them completely unreadable and painful.

I don't really think there anything inherently better about having the framework connect things through inputs and outputs or developer explicitly specifying the connections through function calls.

In both cases the executor can decide how to execute the graph.

link
bbsimonbb 916 days ago
Are you talking about this? https://patents.google.com/patent/US7865872B2/en
link
aleksiy123 912 days ago
No, unfortunately the framework itself is not public.
link
bbsimonbb 916 days ago
That sounds very cool indeed.
link
j45 917 days ago
My guess as to why it was over engineered was to be reusable, or at the start maybe an unknown complexity was foreseen requiring unreasonableness flexibility.

Like many, I early in my career once rewrote a partial implementation of windows workflow foundation to work out of MySQL with db functions. It was .. beautiful. :) And the problem learned to behave and not need it again. Still running years later and reasonably refactorable.

link