Watching the Kickstarter video, it felt very forced. I get that you are trying to revolutionize programming, but try not to keep telling me that and instead show me. I feel that if you had a screen cast of working on a simple demo application (longer than 10-15 seconds), it would've conveyed the message of how powerful NoFlo really is, much better.
I'm not trying to take away from all the work you've done (it looks like a lot and keep at it). It's just a suggestion on how to really show me the ground breaking-ness of NoFlo.
> If you’re a programmer, deep down, you know code is bad, it rots,
> and with each new line your software accrues complexity debt.
> New code means more bugs, more refactoring and more time needed
> to bring someone else up to speed.
Yeah, and moving little interconnected boxes around a project workspace will solve that? It's not like visual programming is something revolutionary new and different - it's just another representation of the same old thing. So the chances that it will solve the current problems with large code bases are the same as a green car being faster than a yellow car only because of the color.
I think their reasoning isn't entirely flawed. I have a hunch that by showing how the code is laying out, tolerating spaghetti mess will be much harder than it is in a common IDE - the amount of complexity that code hides can't be hidden when splayed out as graph.
As for solving problems with current codebases, you're entirely correct. I think this will have a purpose in identifying common functionality for refactoring for small projects, but to wrangle a large project into something usable with this will require good automated code clone detection (which may or may not be good enough as of today) to enable the redundant nodes to be coalesced.
The complexity and spagetti will just end up inside those nodes rather than between them, the same discipline required to refactor and rearchitect as requirements change will apply. This just seems like a visual wrapper on a much more functional programming / actor model style approach.
I think more to the point I can't see anything about visual programming that will make refactoring easier or less necessary.
I don't fully agree. Well, I do agree that it is just as necessary and good software engineering principles are necessary to prevent the code from (quite literally) becoming spaghetti code, but from the work I've done in Max/MSP, I feel that as long as you follow software engineering principles (most Max/MSP people don't because most are eg musicians and such who have no software engineering training) I found it to be a very pleasant experience with many advantages over traditional textual languages. Granted, some things really are best represented textually (certainly complex expressions or formulas) but a lot of code is just routing data around and I found that the visual languages I've used (mainly Max and Synthmaker) really shine at that. in fact, I find that moving data around is the single most important programming task in most programs.
Other areas where visual programming really shines is in experimental programming and I think a big part of that is that 1) you can easily "wire" things up differently and change things just to see what happens and 2) you don't have to name things until you're good and ready. You still should properly name things, of course, but during prototyping and experimentation I find this often gets in my way when using textual languages and I end up giving things temporary names like foo or x. Another area where (some) visual languages shine is debugging because they often let you visually see the flow of data through the code and intercept it and such. I found Max/MSP's debugger to be very nice to use, though I also feel there is a lot of room for it to improve even more still and unit testing tools are needed too.
As for refactoring - you know how some IDE's have a feature where you can select a bunch of code and hit refactor and it puts it into its own function for you? Max does this really well too: click-drag to select all the components you want to factor out and it places it all into a new component for you. Really quick and easy. This isn't a visual vs textual thing but rather a tooling thing, of course.
I'm not quite sure what problem is this solving. What exactly does dataflow programming allow me to do that is not possible with non-data flow programming? It appears that the single components are still text, and data flow is only used to hook up these components. So how is this different than say an IDE with definition and reference lookup?
I've used Pure Data (which they mention) a little bit and here is an example of a somewhat advanced program. http://i.imgur.com/NRtG8Ai.jpg [1].
For Pure Data, dataflow somewhat makes sense (considering the problem domain) but if I had to pick between debugging the schema above and debugging text code, I think that I'd choose the latter.
EDIT: Lol, here's an example of another PD program I found. http://i.imgur.com/Flwviol.jpg Now imagine that you inherit a legacy codebase that looks like that.
Flow chart can be beautiful or ugly just as a textual code can be beautiful or ugly. Also shape of nodes and connectors as important as indentation in text. Look for example of some graph from Nuke - http://www.spaille.be/dev/wp-content/uploads/2010/10/works_n...
> Now imagine that you inherit a legacy codebase that looks like that.
I wager that every codebase beyond a few years of churn looks like that, you just don't look at the CFG to see how much of a mess it is.
There's a key to their approach in that abstraction can be used - the nodes aren't statements as they are in PD, but chunks of code. This means the low-level stuff is hidden from you and you see how higher-level components integrate (if you take the time to tell it what the high level components are, presumably).
Code at Compile time is a different entity than the Code at Runtime. Programmers learns to visualize expanding and collapsing data structures and the web of pointers and call-backs. One simple tiny recursive function can become a massively complex tree at runtime. These two complexities are similar yet very different. Any system that doesn't respect the runtime will fail to understand the mindset of a programmer. I hope they succeed in dealing with both.
Exactly this is the problem with grafting visual programming interfaces onto textual programming semantics. One of the main important reasons to do VP is to be able to inspect the code, to easier see what it does, ideally at a glance. Yet as you point out, simple recursion can make this really difficult to do. My thesis is that we need VPLs that have semantics specifically tailored for the visual interface. It's possible, but takes a bit of thinking to work out
Looks like an interesting way to enforce the pattern of testable, reusable components for developers that tend to end up with spaghetti code. However, I think it's a little dubious to say that designers without coding experience will suddenly become empowered to contribute to the code base (or write their own applications) in any meaningful way.
Looking at one of the sample pictures on the Kick Starter page I see AnchorY out connected to SpringY anchor and Touch start connected to DragX open. I literally have no idea what this is doing. I probably would have connected Touch start to DragX in, though apparently that would be wrong. So as a non-coder I need to understand what every component does (which probably is not explained in layman's terms), what inputs are, what outputs are, what the specific component inputs and outputs are, and what those actually do (again in layman's terms please!).
More importantly, I need to be able to think like a coder and know how to break large problems down into small components and be able to figure out how to then wire those together to get the desired outcome. And as soon as I hit a bug or it doesn't do what I want, I'm guessing I'm just out of luck because everything is a black box to me.
I remember some usability studies that we did at Microsoft that was testing a simplified flow control UI to do basic programming. We brought in people that claimed to have at least scripting experience if not coding experience. We should them a constrained UI that had some operations (ping a server, start a server, stop a server) and basic flow control (if, for, etc). We asked them to reboot a server as a warm up question - since no 'reboot' operation was provided they just need to combine a stop server with a start server. We lost 75% of the participants at that step. It was pretty eye opening to see first hand that not everyone can think like a developer, it has nothing to do with providing a nicer UI or pretty visuals. Problem decomposition and recomposition are learned skills.
I really like the approach and think it can be very effective at improving the overall experience for developers. But to say this is somehow suddenly going to allow somebody with no coding or developer experience to "skip the tens of thousands of hours becoming fluent in syntax and start visually hacking on your application now" just reeks of marketing bs.
What would have happened if the question had been rephrased to "Ok, you have stop car, start car and check car. What do you do to restart a car"? Perhaps they would have done a lot better because they have a better understanding of the car domain.
The purpose of a visual language is not to replace domain experience. If you put me in front of a drawing program with a pallet of red, blue and green and ask me to create the color cyan... Then we're gonna have a bad time.
Visual programming does not claim that critical thinking is not required: aka programming in this context.
We've chatted with people who've said "I can not get the coding thing. That has always been hard for me. Designing and taking bigger concepts and parts and putting them together: assembly. I get that." These people can solve problems, they understand critical thinking. They have expertise in their domain.
What they don't get is a mapping process of automating a real world system (what programming is) in a computing device by coding out the solution.
Don't even get me started on education where students are expected to learn how to think critically (do that mapping thing) and fight with, to them, totally nonsensical compiler errors as they try to learn coding and syntax.
In my opinion, a lot of brilliant future programmers are lost because they don't enjoy fighting the compiler/interpreter.
The participants in the study were IT professionals that managed servers for a living. Servers and actions regarding servers were definitely within their domain experience. So we did exactly what you were suggesting we did.
I still think that connecting up a hundred little boxes is just as complicated as actually looking at code syntax, but even more so since everything is a black box. How do you suggest people fix errors in the output when the only action they can take is to connect boxes with lines?
Where these projects tend to fail is in being designed with the assumption of needing to be as decomposed and abstracted as textual code. Suddenly the components become "syntax" again and you're just doing regular old coding with more clicking. On the other hand, the most popular implementations in the historical record - projects like Pure Data - have relatively weak coding abstractions and instead gain strength from staying tightly focused on their domain and never deviating - they are big libraries with a very tiny language, not the other way around.
NoFlo is building on the existing legacy of what has been shown to work, and is most progressive in the sense that it aims to have libraries for many different domains, rather than just one.
NoFlo's black boxes can always be opened and modified, whether there is another graph or JavaScript inside.
The tools for interactive prototyping for designers will be built on the foundation that we're building now. So the draggable/spring example could be inside of a black box that exposes the important variables. Designers can use that box as-is, but there is very little friction to seeing and experimenting with how it works inside.
As much as I want to love this thing, I can't help but feel like they are trying to solve an unsolvable problem.
We don't consider Michaelangelo a genius because he thought up the David statue. We consider him a genius because he went through the process to make the statue exist.
Great software exists not because someone thought it up. Great software exists because someone went through the process of making that software a reality. There is no shortcut, in the same way that there is no shortcut between thinking up a really nicely painted portrait, and making that portrait a reality. The only way to make a painting a reality is to learn how to paint. The only way to make a program a reality is to learn how to code. Any tool that claims to make coding easier is nullified by the fact that you have to still have to learn that tool too!
What is your point? Regardless of how easy or hard the tools we make are obviously someone has to learn them to use them, and when they use them well it will be praised. If this allows people to make better and more robust software with less skill, the bar gets raised and that's a great thing. You could say the same thing about any new tool or language but, well, that's how things develop.
Is it a magic bullet? No Yes
| |
Ok. |
|
Find some larger things to shoot. Repeat.
More than making coding easier, I think we're trying to give strict structure to code and pair it with a visual metaphor to help manage complexity. There is a learning curve, but we're designing it to stick close to JavaScript.
If we succeed, I imagine that we'll just increase the complexity of the coding that we take on. That's an interesting possibility. I'd like to imagine a new generation of startups with more interesting tech than CRUD.
The strict structure kinda bugs me. I expect situations where a clean solution using higher order functions outside this framework is possible, but if I implement it, I no longer have this clear picture of the system. Or I submit to it, perhaps duplicating code and spreading gnarly business rules on top of what should be "pure" data flow logic.
EDIT: Still, I'm all for experimentation. If they can pull this off, great!
>Any tool that claims to make coding easier is nullified by the fact that you have to still have to learn that tool too!
This statement is proven false by the number of IDE's that claim to make coding easier, and in fact do.
Noflo makes code easier to maintain and collaborate on by providing a visual map of how code is designed. The benefits of which should be plainly apparent if you have ever been tasked with maintaining >10,000 of lines of code.
> NoFlo graphs instead are only the coordination layer
that manages the control flow of your software.
reminds me a lot the whole SOA/BPEL/BPMN mess that was hot few years ago in enterpriseland. The surge in visual programming just resulted in developing the same logic with less powerful tools in languages that were not designed for programming (XML and in this case JSON).
These kind of tools focus on the wrong side of the problem. It is not about how to connect the nicely designed services, but how to actually get services to be nicely designed. Once you have service that can be easily connected in the visual tool, it can be trivially used from the conventional programming language as well.
Reasons why services can not be just trivially composed include:
- every service that is comprehensively mapping real problem is inherently complex
(even for seemingly simple things like address geocoding)
- any two services will have slightly different APIs
(consider various signatures of substring method in various languages)
and oftentimes the backend system details
- limitations are leaking over the service API
(datatypes, formats, paging, ...).
For these reason there exists the large branch of programming that deals with integrating services (sometimes called Enterprise System Integration) that basically wraps, formats and preprocesses data before shoving them to the target service.
For restricted domains - it could probably work. But for general service integration within larger company, the best approach so far has been a lightweight approach with mixture of Spring Integration and plain old Java.
This reminds me of Quartz Composer[1] in a lot of ways: you wire the inputs and outputs of a bunch of different nodes together (called patches in QC) and each patch gets you a step closer to the solution. This seems to work especially well for graphics programming type problems where there are many different variables to tweak and you're applying many transformations to the image at each stage.
The problem is each component in the process still needs to be written in code. In QC I would often be frustrated when a patch didn't quite do what I needed and I had to keep trying to make my idea fit the limitations of the available patches.
So while this might help non-programmers understand on a big picture level what's happening, to go beyond that they're still going to have to learn how the individual components work which comes back to learning to code textually.
Gridspy ( http://gridspy.com ) is built on a lot of discrete components. We've got a variety of servers all talking via several APIs and pretty complex data flows in twisted and also in javascript in the browser.
I'd love a tool that simplified this - even if it just helped ease the pain of API integration or the flow of data.
I need to be led to more concrete information around what happens on deploy. Do you compile applications from the flow diagram?
I want to see a nice migration to and from the noflo platform. The last thing I want to see is lockin either way.
For me I need Python and Javascript support. We're built in Python.
Very nice video, very artistic. You certainly had fun pulling focus and playing with exposure.
The network is assembled at run time by instantiating the various components and connecting their ports. There is no compilation step.
As for Python, we'd love to support the language! It is one of our stretch goals. In case we don't hit it, the Python community can of course do it themselves :-)
We at Interface Vision ( http://www.interfacevision.com ) are working on a Visual Programming Interface as well and we wanted to say congratulations on a successful first day on Kickstarter. It is great to see someone else share the space. Great job guys and again congratulations!! If you ever want to chat please don't hesitate to reach out - great job guys and good luck to us all!!
One big difference between actors and FBP is that actors can give feedback whereas flow-based programs can only send data from one port to another (so, providing feedback to the process feeding it information requires the creation of an input port on the feeder and a feedback output port on the processor...)
As with anything else, FBP isn't a hammer that turns everything else into a nail, but it's an interesting architectural pattern. I'll be curious to see their noflo-based Jekyll replacement.
I think of this as the "Rick Allen Solution." Rick Allen is the drummer for Def Leppard, who lost his arm in a car accident. He says that overcoming his disability has turned him into a better drummer.
Fundamentally, forcing yourself to do visual programming like this, is like tying one arm behind your back. It's so impossibly difficult, that you'll demand that the tools you use are the best quality, and intelligently solve as many of the problems at your layer as possible.
The truth is, if developers insisted on good tools with good abstractions, they'd achieve all of the supposed benefits of visual programming.
Most developers forget they're responsible for making their own living space well-organized. They just code, leaving a mess everywhere.
From a teacher's perspective this looks excellent. I start teaching my students programming using Scratch. Continuing on to a more advanced visual programming interface would be great. It reminds me a little of the visual scripting interface inside Cinema4dD- Xpresso.
Lets solve spaghetti code with more spaghetti code hidden behind some nice UI. How about actually thinking about problems and finding fundamental flaws in workflow? I know it's harder, I know you can't then film fancy promo video in some expensive hotel. But it actually might work.
I'm not saying that ideas of visual programming are bad. What I'm saying it that hiding complexity behind some more complexity and saying that we invented simplicity is bad. NoFlo doesn't look like something new but more like just wrapper around existing methodologies. Nothing new, just hiding details from programmer. OOP way of thinking.
I concluded that they are trying to open source what https://www.mashape.com does behind the scenes, visually. Connecting the different parts of the SOAd web, in the sense of docker and containers. It's true- why can't newly released services just work like legos? Say you want to support a new payment system, or a new analytics, it's just drag and drop with this, hopefully; right?
Throwing this out there, I have shelved a similar web-based project that did this with Verilog designs, which lends itself more naturally to this struture imo. I stopped because I don't believe there's a big enough market for it. That, and I wasn't sure how to tackle the problem of thousands of I/Os in a visual format (would make things real hard to see).
I think this is a great idea, but experience is everything, as others have stated, the diagrams presented still have quite the learning curve, knowing what names connect where. I think if NoFlo allowed actual coding through a visual interface like Scratch, with the ability to group components into functions and classes, this could really go somewhere.
Interesting. This would seem a natural fit for stream / event based processing. Digesting server-logs, social signals, etc. Something like Twitter Storm but with components on a more fine-grained level. Wonder how concepts like iterative map/reduce map to this.
noflo looks like a novel yet familiar way of designing software, which really glues well with TDD.
I don't sense a very strong emphasis on runtime profiling from the video or the description, but if this were a core feature then it would certainly add value to the product.
I'm not trying to take away from all the work you've done (it looks like a lot and keep at it). It's just a suggestion on how to really show me the ground breaking-ness of NoFlo.