| This is super insightful thanks a ton for this gold mine. On the python vs yaml part—definitely could've made that way more clear in the demo. Right now are framework lets you call these python objects in your yaml file, but we are working on just a python-centric implementation as well for those that do not want to interact with yamls. On the loom and licensing choice—that's a great point. One of the main issues we ran into is getting adoption as we originally just tried licensing out the framework (mega fail ofc)—found out the hard way that no dev wants to buy something to try it out. We're definitely flexible on our license and will take all this feedback into account. On the barrier of entry—also super insightful. We're working on a local UI offering that will be a 'config' block builder that will be free for all installs. We're implementing a DAG view similar to Airflow on the transform level. We also want to make it super easy to see your code and preview how it changes with this local UI (and have a list of all the params you need for your spark objects without having to go through the docs). We also want to flesh out more features especially on the translate side, as well as host on the cloud. With the complexity issue that's something I ran into Disney as well! As the product grows we definitely want to flesh out our transformers based on the scripts we see. For now, the developer can make one-off transformers—we actually have a catch all "SQL transformer" for cases where you want to just pass in your sql (similar to a dbt model) and run it that way. That way it's a fail safe for if you have one specific transform that you feel is super hard to break down, you can fall back on dbt's way of just modularizing the SQL into a transform, and reference it however many times you want as an input block later on. Thanks so much for the congrats, will definitely reach out and would love to have further discussions in the thread as will. |