[jcranmer]

What I like least about this article is that it's completely soured the entire context of asynchronous programming. Invariably, any time someone discusses design of an async functionality, function coloring is brought up, with almost no analysis as to how it applies and why it's a good or bad thing. (Ironically, I probably see more in-depth analysis these days as to why this article isn't apropos than why it is when this happens.) It's just reduced to "anything that makes a separation between async and sync is function coloring and that's automatically bad." The existence of any sort of trade-off, or really, the entire meat of the article, is just completely ignored.

One thing that can be better called out is that this issue of function coloring isn't just an async problem. Exceptions cause function coloring--and not just Java's controversial checked exceptions. An infallible/fallible domain split is function coloring. Javascript's async handling is called out not because it's doing the function coloring but because--in 2015--the tools that existed for dealing with async code in JS libraries were really, really bad, largely reliant on callback hell. Promises and the async/await keyword fix most of the issues, and the ones that aren't fixed boil down to the fundamental issue that an asynchronous event-loop model and a synchronous batch model are just different programming paradigms to begin with.

[gpderetta]

The problem with function color exists when you can't abstract over it[1]

Some statically typed languages (I believe both Haskell, ocaml) have powerful type system that allow abstracting over function types and function colors. Color is not an issue here.

Some other statically typed languages (C#, rust, and C++ (at least with the built-in stackless coroutines)) can abstract over types but not over colors. This is a problem.

Some statically typed languages (Go) do not encode async-ness statically, so it is not an issue[2].

Some dynamically typed languages (scheme, Lua, lisp) also do not encode async-ness statically. Everything is fine.

Finally there are some dynamically typed languages (python, js) that, eskew static types but for some reason still decide to encode async-ness statically. For me this is the most bizarre decision, especially as some of the justifications for static async-ness (performance, memory usage) are less relevant.

[1] for example, a litmus test is being able to implement an higher order function that inherits its color from one of its parameters. Essentially this is the problem of generically turning an internal iterator to an external one.

[2] fundamentally in these languages continuations are first class values that can be passed around, so the asyncness is naturally not bound to the function that created a continuation.

Edit: you can in principle abstract away color in any async language by simply assuming that any function call is async and await it. Then sync functions can trivially be made async. But at this point async annotations no longer convey any useful property: the language might as well implicitly await any function and require call-site annotations for diverging control flow or reentrancy requirements. More practically as languages with async evolve and grow asyncness becomes pervasive and pushes away any sync component.

[akst]

> The problem with function color exists when you can't abstract over it

Hopefully it's safe read this as there's no common static type between function and async function meaning APIs (that take functions as arguments) have to provide seperate methods (or overloading) for these different colours.

Like in typescript you can write `<T>(f: () => T) => T` because an async function statically is just the return type wrapped in a Promise, not something like `async () => T` you can still pass in an async function as an argument.

I think that's a reasonable thing to take issue with, and its _possibly_ an avoidable design problem. That said I can see it being less avoidable if the async function requires some special kind of invocation (like being associated with some kind of async runtime and its a compiled language).

When I see people bring the issue of function colouring, the focus tends to be on the fact that a function is no longer interchangeable with a sync function and now you have to handle a promise, which I personally find unconvincing if the return type really should be a promise then it shouldn't be interchangeable with a sync function.

[codebje]

Your first paragraph links having the colour in the type system as allowing you to write functions that take arguments of parametric colour; your last paragraph says you're unconvinced that you might also like to write functions that return results of parametric colour.

An example: a vector of things to a thing of a vector, for "thing" in (promise, option, result<E>, ...). Such a function should only really return a promise if it's given a vector of promises, and, with an interface that "thing" supports, can be written generically for all those things.

(In Rust, there are separate implementations of that for Option and for Future.)

Higher-kinded types are the (a?) design solution, but they _do_ come at a cost, and for some that cost is higher than the cost of colours.

[akst]

I think you're confused, I was talking to two different points, while I'm sure I could have communicated with more precision, either missed it, it was unclear or you don't understand, either way I don't really get the gotcha tone when you could ask for a clarification:

Anyways, the two points:

- The first point was, "not having a common way to generalise over both sync, async or blue, green, brown functions, seems avoidable and bad". This is when the type system struggling to common up with a common classification for function invocation independently of colour.

- The second point was that, was "so what if there are different return / wrapping / container / monad types", which focuses on a more common interpretation of this article but a different one.

In Haskell a type in a result, State, Config, Parsec, Maybe is in it for a reason, and thankfully we can generalise over that. Higher kind types (abstracting over abstractions) is a whole other basket, as an ex haskeller I would love to see them more mainstream but admittedly I don't think language authors are convinced and there isn't much we can do about it, so we should learn to make do with what we have outside of haskell.

[codebje]

I think I was simply not very good at expressing what I was trying to convey, sorry, and it is a fault of mine to come across as gotcha-y even when trying not to. Thank you for responding with patience despite that.

The first point I interpret as "colourful arguments are avoidable and bad", with which I agree.

The second point I interpret as "colourful returns are unavoidable but good", with which I disagree - even if that interpretation is too strong and is more "... are unavoidable".

A function's type is its full signature, including inputs and outputs. When you have first-class functions, you have values with function types, and those values are inputs to other functions. Necessarily, then, if you colour outputs you have also applied colour to inputs.

Transposing a vector of things to a thing of vectors is an example of where colourful output forces colourful input. If you cannot abstract over abstractions, you must write and re-write the sequence function for each abstraction.

I'm in agreement with your closing paragraph's sentiment. That HKTs aren't a broadly adopted solution is something I accept, but I reserve the right to low-key begrudge it.

(And the more I write about this, the more I wish the original article had used "flavour" rather than "colour" as I try and probably fail to find phrasing that doesn't simply sound like portions of a racist rant.)

[akst]

> I think I was simply not very good at expressing what I was trying to convey, sorry

No its, and I appreciate you taking the time to read my reply and consider my perspective here.

> The second point I interpret as "colourful returns are unavoidable but good", with which I disagree - even if that interpretation is too strong and is more "... are unavoidable"

Thats fair, but yeah I wouldn't go so far to say its good or imply we should celebrate it in anyways, more so it as a unavoidable constraint that warrants engaging with.

For sure HKT would generalise many stray ends, and there are definately more complicated usecases where you can write much nicer types with HKT, although my experience has been theres been more pain in writing them without HKT than using them without HKT so the pain is a fixed cost of building the library and not an on going problem of using it. Although I am sure there are cases where it's also the case that usage of the library is more painful without HKT.

IDK, I haven't written a ton of Haskell in a while maybe I've forgotten some of its magic and internalised some of the suboptimal aspects of the absense of working without HKT.

But I do from time to time find problems that would be nicer to solve with HKT, I think generally quite a few of them are DSLs or some form of meta programming. I guess in typescript conditional types you can get away with a lot cooked things.

> Transposing a vector of things to a thing of vectors is an example of where colourful output forces colourful input. If you cannot abstract over abstractions, you must write and re-write the sequence function for each abstraction

I've unfortunately also had similar issues from the lack of HKTs with some linear algebra APIs so I don't find this too surprising.

[squirrellous]

This is the kind of comment I come to HN for. Thanks for teaching me something.

[dnnddidiej]

I am so happy I have never heard anyone IRL say colored functions. It would annoy me. The concept is interesting but like all engineering it is a trade off. In Node amd Go you don't get a choice anyway. In C# you might choose based on performance thinking of thread pools etc IIRC.

When programming in Node I find in practice async and "colored functions" no issue especially with async await. Except for performance issues they come with sometimes but not at a programming level.

[rtpg]

> When programming in Node I find in practice async and "colored functions" no issue especially with async await. Except for performance issues they come with sometimes but not at a programming level.

JS solves this problem in two ways in the ecosystem:

- basically saying "all functions must be async" in practice

- allowing you to await a non-awaitable ("await 3" is valid)

so library authors can "force" async/await, but users don't actually have to interact with it when they don't need to. But "everything" being async/await means it's all 'basically fine' anyways

There's also the fact that JS libraries tend to be "pass in a bunch of callbacks" vs, say, Python's "override this class". It makes it much easier for libraries to have everything be async and have it really not get in the way.

Python libs tend to have much larger API surfaces due to how OOP works. So async-y internals works are harder to isolate cleanly without breaking the public API. But if you make your API "async-first" then the debugging experience in Python is miserable (try pdb'ing your way through awaitables....)

Even here though there are problems. For example, I've tried in the past to replace some lib with a more performant WASM-y thing. But it couldn't be a drop in replacement because the original library was a sync-only API, and the replacement was async!

Something very silly: you write "function add(x, y) { return x+y }". A bunch of people do things like "add(add(x, y), z)" everywhere. You find out you could make "add" "better" with async/await. You now have to get all callers to rewrite.

So what everyone does is just throw _everything_ in to the async/await pile. Which... I guess is fine but I personally dislike writing "await add(await add(x,y)), z)".

(aside: Rust's postfix await at least makes this kinda refactor less annoying)

[leonidasv]

The Node world was built with asynchronicity in mind. First via callbacks, then Promises, then async/await (Promise-based), so it feels natural now.

But if you take Python (for example), it's a shitshow. You usually have two versions of the same API, split by function name, client, package, or namespace: `foo` and `afoo`, where the a-prefixed one is async and meant to be used inside async function call chains, and the other one is the blocking version for non-async chains (which are still very much in use). It's a pain to develop for, to maintain, to scale, everything.

[EGreg]

Actually, it's not async programming. It's only async programming the way JS does it... which is unwinding the entire stack, and then starting another stack on the next tick.

Instead, many languages have fibers / coroutines / etc. which simply start new stacks elsewhere, and capture the context.

[dnautics]

what I like least about this article is how people seem to just substitute some other, usually theory, notion of what function coloring is to elide the argument (usually to excuse their favorite PL) without actually RTFA. The article is about ergonomics, not PL theory.

> Exceptions cause function coloring

do they? Do they?

1) Every function has a color

2) The way you call a function depends on its color

3) You can only call a red function from within another red function

4) Red functions are more painful to call

5) Some core library functions are red

[nemothekid]

Java's checked exceptions fit the 5 criteria:

1. It either `throws` or it doesn't

2. If the function `throws` you have to wrap it in try/catch, or make your function `throws`

3. Your function is `red` if it `throws` the same exception.

4. see (2)

5. See the FileReader class in core.

Now, C++ exceptions might not satisfy all of these, but the problems CheckedExceptions were meant to solve still exist in C++ and as a result some style guides forbid them entirely. Like async, the biggest problem with exceptions were the ergonomics.

[Terr_]

> Like async, the biggest problem with exceptions were the ergonomics.

I know it's not a popular take, but I prefer the idea of Checked Exceptions over unchecked ones [0], and suspect current opinions would be vastly different if Java had shipped with some sweet syntactic sugar for: "If an exception that is of kind A or B or C occurs, automatically throw another checked exception X with the original exception as a cause."

> Ex: If I'm writing a tool to try to analyze and recommend music that has to handle multiple different file types, I might catch an MP3 library's Mp3TagCorruptException and wrap it into my own FileFormatException.

This would reduce the temptation for developers to ruin the type-safety characteristics by wrapping everything in a RuntimeException just to get the ticket out the door.

[0] https://news.ycombinator.com/item?id=42946597

[ameliaquining]

The problem with checked exceptions is that they don't compose with the rest of the type system. Hence the infamous problems with things like Streams. Result types have basically all the virtues of checked exceptions without the problems.

[vips7L]

Result types do have one problem that checked exceptions don’t. Checked exceptions automatically combine into union types in a throws or catch clause. I haven’t seen a language that lets you be generic like that.

    T fn() throws E, F, G

vs

    Result<T, E | F | G> // not even Rust lets you do this.

[ameliaquining]

That's more a consequence of Rust needing its tagged unions declared up front so it can lay them out consistently in memory without runtime type information. Python and TypeScript have untagged unions (that are discriminated at runtime by the RTTI attached to all objects in the underlying dynamic language); they don't happen to have an equivalent of Rust's ? operator, but if they did it'd work like you're describing.

[steveklabnik]

The E | F | G could be two different features, "anonymous sum types" or "union types".

TypeScript is an example of a language with union types: https://www.typescriptlang.org/docs/handbook/unions-and-inte...

[skybrian]

The problem with Java's checked exceptions is that it has too many kinds of exceptions to choose from and they're overly specific. Compare with Go, which has a single error interface and had it from the beginning, so it's used everywhere. Returning a new kind of error is always a local change, unless it's a function that didn't previously report errors at all.

Type systems permit either standardization or fragmentation and that's an ecosystem issue. Another example is that a language without a strong consensus on which string type to use will result in a fragmented ecosystem when each library goes its own way.

[Terr_]

> too many kinds of exceptions to choose from

I don't understand, why would you need to pick a checked exception? It's the dual or mirror of feeling paralyzed over a return-type because there are "too many kinds of Object to choose from."

If you're writing a CrystalBall class with a gaze_deeply() method, you'll probably return your own VisionResult (extends Object) unless it throws your TooCloudedException (extends Exception).

When someone else writes a wrapper or higher-level layer that uses your code, then it'll be up to them to convert or wrap those results and exceptions into something suitable for their level of abstraction.

> Returning a new kind of error is always a local change

One of my axioms here is that return-values and checked-exceptions are two sides of the same architectural type-system coin. While I'm not familiar with Go, that sounds like something that would be a symptom of bad architecture if it occurred for return values.

In other words, suppose all Java methods always returned Object [0]. That would also ensure that a new return type is "always a local change" to the compiler, but I think most developers would be rightly horrified if they came across code that worked that way.

[0] Let's ignore Java primitives for now.

[lmm]

> you'll probably return your own VisionResult (extends Object) unless it throws your TooCloudedException (extends Exception).

> When someone else writes a wrapper or higher-level layer that uses your code, then it'll be up to them to convert or wrap those results and exceptions into something suitable for their level of abstraction.

Why though? What do you gain other than longer stacktraces with all those wrappers? People always trot out some theoretical notion that a caller is going to catch that framework's different exceptions and handle them differently, but have you ever seen calling code that actually did that?

> In other words, suppose all Java methods always returned Object [0]. That would also ensure that a new return type is "always a local change" to the compiler, but I think most developers would be rightly horrified if they came across code that worked that way.

There are many different kinds of values. There really aren't that many different kinds of error - there's "transient error that you might want to retry", "programmer called the API wrong", and that's about it, most other cases (like bad user input) probably shouldn't be exceptions.

[skybrian]

This isn't like returning Object. It's more like returning a String. After using a language with a common String type, who wants to go back to writing code to convert between between different kinds of strings? Having to choose among different string implementations because there's no standard usually leads to boilerplate code doing conversions at the borders.

Usually you just want to propagate or log errors, so having a generic error interface is sufficient. It's true that in Java, you can wrap exceptions, but that's extra boilerplate.

(And yes, Go does notoriously have error propagation boilerplate that they should fix, but that isn't a type system problem.)

[gpderetta]

Exceptions are a very good comparison because they also perform non-local control flow.

Checked exceptions are a form of coloring, while unchecked aren't. But Java (which has checked exceptions) has an escape into unchecked land in the form of RuntimeError, most async languages do not, short of spawning a background thread (for sync->async) or force blocking (async->sync).

Interestingly, Result<T,E> based error models are semantically (and even syntactically, mostly, except for the call-site annotation) equivalent to checked exceptions. Usually these languages have enough abstraction capabilities (HKT for example) to make coloring not an issue, or, again, an escape into unchecked land (for example panic in rust or Go, although the latter hardly counts as having Result-like error handling).

[sheept]

#3 is not satisfied, as you noted in #2. You can call `throws` methods from non-`throws` methods by wrapping the call in a try catch, and `throws` methods can call non-`throws`. There isn't an exclusivity asymmetry like there is for JavaScript async.

[nemothekid]

That only applies to Javascript, which, is mostly only red functions anyways (there are no blocking apis in javascript). Javascript doesn't have the coloring problem in the way Python or Rust has it.

In Python, you can wrap the call with asyncio.to_thread, in rust with tokio::spawn_blocking.

[sheept]

I think you got it backwards: JavaScript has the coloring problem while other languages don't.

"Red functions are more painful to call" alludes to async functions. Every await yields back to the event loop, which adds overhead. Making every function red/async adds a performance cost (and makes it harder to reason about race conditions), which is why JavaScript has a mix of blue and red functions.

Other languages can escape the "red functions can only be called by red functions" trap, like Python asyncio.run or Tokio block_on. JavaScript has no such alternative, not even in Node. Therefore, Python and Rust don't have function coloring, but JavaScript does.

[vips7L]

I think the big difference is that in an application that cares about throughput and being non-blocking simply blocking isn’t really possible as it affects the whole performance of the application. Wrapping a checked exception in an unchecked one doesn’t do that.

[dnautics]

Ok, sorry it's been about 20 years since I last javad IIRC you didn't have to declare exceptions in your function signatures. However, wrapping in try/catch seems to violate #3. Try catch is not a heavy lift of a seam between red and blue

To be fair, #3 seems to have shades of grey. In some pls, you can call an async function from a sync one by wrapping it in a whole damn event loop system. Should that count?

[Terr_]

> Ok, sorry it's been about 20 years since I last javad and you didn't have to declare exceptions in your function signatures.

You're probably remembering RuntimeExceptions, which are a subgroup [0] that are exempt from "checking" by the compiler, which means it does not require method signatures to declare "I might emit this."

[0] https://docs.oracle.com/en/java/javase/26/docs/api/java.base...

[vips7L]

You can declare your runtime exceptions too. The compiler won’t enforce you to catch them though.

    void fn() throws IllegalStateException

[nemothekid]

Your question just kicks the can down the road. The problem with the article, to me, is the author doesn't want to accept a certain amount of complexity and has erected arbitrary road blocks.

To you, a "whole damn event loop system" is too high a price to pay, but try/catch is not. The complexity of exceptions is invisible to you. However there are certain environments (e.g. FFI) where I dont want "the whole damn exception runtime".

[melagonster]

Maybe author do not want to hide the complexity in functions?

[dnautics]

i mean no? the coloring problem appears to be solvable in zig. i maintain an FFI binding library for the BEAM vm, where once zig finishes its stackless coroutine support, the same zig function written once should be fully interchangeable between non-async, threaded-async, or async-with-yieldpoints-wrapped-in-the-BEAM's-scheduler.

[lmm]

> Try catch is not a heavy lift of a seam between red and blue

> To be fair, #3 seems to have shades of grey. In some pls, you can call an async function from a sync one by wrapping it in a whole damn event loop system. Should that count?

I think you have to count any extra overhead where you can't just write f(), including try/catch. It's always possible to call whatever kind of function from whatever other kind of function if you put enough effort and hackery in, so if we can't use functions of kind x in functions of kind y as normal "f()" function calls then that has to be what we mean by colouring.

[jcranmer]

Let's go through the list:

> 1) Every function has a color

Every function either throws an exception to indicate failure or doesn't. There's actually several different function colors available here, based on how failure is indicated: throwing exception, aborting the process, composite return value, error code return value, global errno-like variable, error code as a parameter, ....

> 2) The way you call a function depends on its color

See above.

> 3) You can only call a red function from within another red function

Some of the failure methods, like aborting on failure, cannot be converted to another mode at all (or only with very great difficulty). Others, like exceptions and errno-based routines, come with environmental constraints that could be contained by an error conversion routine in theory but may be precluded due to how the system as a whole works (e.g., a global variable errno doesn't play well with threads). Which isn't quite the same thing, but then again, "red function" here is async function, and the call-async-from-sync variant is the easier one to pull off (you spin the event loop), and has roughly the same issues as trying to box an exception routine: it only works if the system as a whole has mechanisms to make it work.

> 4) Red functions are more painful to call

Okay, you've got me here... the exception routines are the easier ones to call, syntactically than non-exception-based ones. Internally in the optimizer, however, exceptions are definitely the worst form (even errno somehow ends up working out better, and that's also deeply problematic).

> 5) Some core library functions are red

Oh yes, standard libraries love using a mix of all of these error-handling routines. Look up C++ <filesystem> for example.

[tardedmeme]

Failing is a color, but throwing an exception isn't. An exception-throwing computation can easily embedded in a computation that doesn't throw - you can catch and return null, etc. But very rarely can a computation that may fail be part of a computation that may not fail.

[smcg]

this article seems like a nice view into what things were like in 2015 but we've come along way since then.

[tardedmeme]

What I like most about the article is that it drove the conversation to realising that async is just a poor reimplementation of threads, and put the focus back on how to do threads faster.

[dnnddidiej]

Probably you are right except for some pathological scenarios. Threads and green threads and models where you have have 10000s of threads and not even hit the cardio fat burning zone.