[ssmoot]

This seems like it could've been called "Why Actor Systems Matter".

If you're on the JVM, I'm not sure what Erlang buys you in practice. It's slower and more obscure. It has a much smaller ecosystem. While process-safety is frequently touted, in the real world this is a non-issue among non-issues. It's just not an actual thing. It's not like the JVM goes around Segfaulting all the time.

I'm totally sold on Actor Systems and think it's something more programmers should expose themselves to. I'm just not sure there's much of an argument for Erlang vs the JVM unless you're completely sold on the notion of process isolation for some reason.

[Xixi]

It's not at all about segfaults. Assuming the Erlang VM is as likely/unlikely to crash as the JVM, then it's a complete wash: Erlang processes are green-threads, not OS processes, so you don't gain anything there.

It's about the share-nothing architecture of Erlang: it means your processes are isolated and can be stopped/restarted independently, crash/hang without writing into the memory of another process, have their own garbage collection, and be moved to a different physical computer (or even data center) with no impact at the logical level. Erlang forces you to architecture your program as a collection of nano-services. You can of course emulate some of it on the JVM, but you can't go lazy and cut corners with Erlang.

[ssmoot]

Akka checks most of those boxes AFAIK. And there's really not much you can do in the way of cheating unless I misunderstand you. Or at least idiomatically you wouldn't cheat in Scala anyways.

Speaking of which, I just realized the AtomicLong I'm using in my IdGenerationActor (performs an atomic increment of a processId counter in the database, then uses that with the AtomicLong as the input to Hashids; fast, in-process, cluster-safe short-Id generation that will leave popular Redis based solutions in the dust) is completely unnecessary. Copied and pasted from non-Actor code without consideration.

I guess Actors still can't keep you from doing dumb things yet. ;-)

[querulous]

this is just not true. every other week i have production issues because akka managed to exhaust the thread pool with long running/nonresponsive actors

[ssmoot]

You've done something wrong then? Or maybe using experimental features? I might play with them a bit, and it can be frustrating to wait, but I've never launched anything on -experimental before.

I was referring to "cheating", with the idea that you might pollute your Actors with... I dunno. Programmatic connection pooling for your database driver? Passing mutable messages around?

Both of those things would be very unusual in Actors written in Scala since pretty much everything is immutable by default. Seeing that sort of thing should at least raise some eyebrows.

As far as non-responsive, I've never seen that. But I do take care to use Futures where appropriate, and pipeTo. Maybe it's good habits, or maybe I'm just very lucky.

One of my first Akka projects is still running, still processing content it's notified of by Postgres through LISTEN/NOTIFY, still posting that content into Cloudant (basically a managed Lucene deployment in this case).

And it's been running since September 2014 without a restart AFAIK. Which would have never happened with a previous non-Actor solution we might have used if for no other reason that a network blip might detach the listener, whereas here the Actor just restarts.

My experience has been overwhelmingly positive. Despite doing the wrong thing occasionally.

If you have Actors that are hung, I guess the first thing to try is figuring out which one(s). After that, you could just schedule the supervisor to routinely PoisonPill them, and forcefully stop them after a grace period.

I'm not sure that OTP is going to help with this sort of issue either. You have an apparently blocking process that makes an Actor non-responsive. The fact that it's single threaded within the Actor and stalls it's mailbox is kind of the point of Actor systems AFAIK.

I guess one thing I feel like helps me is to keep my Actors small and doing one thing. It's hard to do too much damage when you only have a dozen lines of actual message handling. "One thing" is sometimes coordination/aggregation BTW. Which means I might have a GetRequestActor, PutRequestActor, DeleteRequestActor, etc. Which do the one thing. But then I also have a DatabaseActor that basically just coordinates/forwards for all those so you don't actually have to deal with them yourself.

And from there if I decided that no GetRequestActor should take longer than 10 seconds to do it's thing, I can easily schedule the DatabaseActor to forcefully stop any task exceeding it (without the need for an actual watch if you choose). Which you can then log an ERROR for and work out why. And maybe some requests just take longer and that's OK. So maybe you then write a LongRunningRequestPathExtractor and put that pattern before the normal one. And now you can have multiple timeouts for different paths.

Or maybe you just record the epoch for different requests, and if you've had 1,000 updates since the last View request, you know the next one is going to trigger a reindex. So you give it extra time. Or you set it to allow stale results and reschedule the same call to occur again in 1 minute to minimize the disruption of indexing the changes. Just some thoughts.

[coldtea]

>You've done something wrong then?

Well, the idea is with Erlang you can't.

[ssmoot]

I don't believe Erlang goes around magically imposing it's own timeouts on message handling.

[vvanders]

My Stop-The-World GCs would like to have a word with you.

[ssmoot]

And maybe in a project with truly massive memory requirements that's an issue. I've never encountered it (as an issue that is) But I only got into the JVM on 1.6 with Jruby, made the switch to Scala around 1.7, and have been on 1.8 pretty much since release.

A 3GB heap is pretty typical, but that's mostly just room for cache.

If you're writing web-apps, there's a ridiculous amount of opportunity for using Actors to good effect.

On the other hand I try to keep an eye on resources. I try not to throw millions of messages at an Actor. If I'm batch processing, I'm probably work-pulling, mapping and prefetching my Sources to attempt to minimize latency. Even if I could tune mailbox sizes to avoid all that I prefer not to code in upper bounds on input.

I would be surprised to hear this is a widespread problem in people's work. You don't often hear the same complaints from Rubyists for example and they're much worse off in that regard. So I feel like this case is massively overstated.

But if you have that problem I can certainly see how that might figure into your equation.

[vvanders]

Not everything is web-apps either. If you've got soft realtime requirements(<30ms) GC will hurt you. We spend a lot of time doing twists-and-turns in Java to not allocate anything to avoid that Gen-0 GC.

Erlang's GC model is just incredibly elegant. Almost like pooled allocators that we used to use in GameDev but in a completely transparent and intuitive way.

[ssmoot]

I'll concede that. But you have to admit this is an incredibly niche problem space.

OTOH the JVM will have much higher throughput on the other 99% of apps.

Don't get me wrong. I like Erlang. It's definitely in my top.. Uhm... 3 languages of interest. Being the only other language I know with Pattern Matching is a huge huge part of that.

For most developers I'd just be careful in making that jump. Because most of what this article is about isn't Erlang specific. It's more like a pretty small club of platforms/languages. IMO.

[vvanders]

Yeah, but real-time is about latency not throughput :).

FWIW a lot of game engines use message passing between components/actors due to the low coupling and architectural benefits so it's not quite as small of a domain as you think.

[wcummings]

Erlang has pretty decent JVM integration. I assume this is why there's a nice Erlang IntelliJ plugin.

http://erlang.org/doc/apps/jinterface/jinterface_users_guide...

[strictfp]

I would love process isolation on the JVM (several smaller heaps), and I'm not even into the actor model.

Plus I can imagine accidental mutability really being a problem. In my Akka course much effort was spent on explaining how to avoid side effect of it (no pun intended).

[ssmoot]

I guess the accidental mutability is a good point. Working in Scala, I've never encountered that.

You do learn not to close over local scopes in futures pretty quickly though.