[s_Hogg]

Not knowing this myself, could someone explain the attraction in using a web scripting framework for stuff like this? I get that this particular example makes the whole thing seem fairly straightforward, but have never really heard why people like adapting Node.js and similar to non-web environments beyond "you can". Is there anything more to it than that?

[spion]

Main strengths:

* The ecosystem is pretty strong and constantly evolving. Lots of companies and individuals are putting a serious amount of work into making high-quality tooling and libraries. (Although with that also comes a lot of lower quality stuff and its sometimes hard to tell)

* The language isn't owned by any single company and is well specified

* TypeScript provides a powerful and flexible type system with advanced dependant-types like features. Lets you gradually evolve from a messy JS prototype to well-organized code with fairly strict type checking

* Fastest dynamic language. The performance is also quite comparable to statically typed GCed languages. (For example, you can probably get to 50%-90% of Java performance on most single-threaded workloads)

* wasm support in the runtime a promising escape hatch to the integration with other languages should that become needed. (Similarly not owned by a single company and well-specified)

* sharing code and types between client and server, including interfaces, validation and data models.

* particularly well suited to handling heterogeneous structured data due to how cheap it is to define new object types (even in typescript)

* Async-IO-first (in fact, async-only IO for the most part)

Main weaknesses:

* Poor standard library with an anemic set of classes, anemic set of implementable protocols/interfaces (more stuff like Symbol.asyncIterable needed) and lacking convenience functions. E.g. see https://api.dart.dev/stable/2.7.1/dart-async/Stream-class.ht... and compare with... well nothing in the language! The official node streams have a terrible API.

* Combining lack of both protocols and standard library leads to pretty bad userspace library fragmentation. (What is the go-to stream library?)

* Restricted data sharing between threads (only SharedArrayBuffer) makes it quite limited for multi-threaded problems.

* (mainly TypeScript) - Insufficient reflection capabilities

* Rigid (non-configurable) node_modules resolution algorithm accepted as standard limits flexibility when organizing projects

Mythical weaknesses that don't matter that much:

* Implicit conversions - largely irrelevant since TypeScript

* DOM/Browser related weirdness - often attributed to the language but actually problems with browser APIs

[gargarplex]

> The performance is also quite comparable to statically typed GCed languages. (For example, you can probably get to 50%-90% of Java performance on most single-threaded workloads)

Citation needed, please. I disagree with this. I recently was able to achieve a massive speedup in a Node application through writing a native C++ module... and implementing Garbage Collection is required for N-API.

[spion]

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

[gargarplex]

Excellent source; thanks!

It’s probably worth clarifying that Node is ⅓ the speed of Java in some of these cases and that the Node implementations use both fixed size typed arrays as well as worker threads, features that aren’t common practice in most Node programs.

[spion]

I wanted to illustrate the fairly "wide dynamic range" available. A lot of typed, GCed languages require significant effort to write highly optimized code anyway, and the optimized code rarely looks like the idiomatic one.

But even with idiomatic code the performance is quite good - the JITs are very high quality.

[gargarplex]

I didn't downvote you – in fact, I upvoted you. But I should also make the point that worker threads do not support importing native Node modules that were built directly on the V8 API. For example, most popular sqlite libraries (better-sqlite3, sqlite3) are not usable with worker threads.

I'm mulling contributing N-API support to the libraries... but I haven't even done any research or planning work yet.

[spion]

An interesting blog post on how that might be done: https://mrale.ph/blog/2018/02/03/maybe-you-dont-need-rust-to...

[ignoramous]

That was amazing to re-read, thanks for sharing. news.yc discussion back on tfa back in the day is a pretty good read, too: https://news.ycombinator.com/item?id=16413917

[spion]

For those situations where the GC becomes a performance (or more likely memory usage) problem, it seems reasonable that the built-in WASM support of most engines will be able to provide an adequate performance (its already getting quite close): https://www.usenix.org/conference/atc19/presentation/jangda

[golergka]

Anything CPU-bound is a very bad fit for node. But most web services are IO-bound, and Node is excellent for them.

[staticassertion]

> But most web services are IO-bound

Node's still relatively slow for those workloads.

https://www.techempower.com/benchmarks/#section=data-r18&hw=...

See how far down in each section you have to scroll to find node, even for workloads that are purely "accept a request and respond with a static string". You'll see lots of Java and Go on your way down.

And most services will have far more compute than just shoving bytes in between services. There's request parsing, response encoding, usually at least a tiny bit of data manipulation.

[spion]

This benchmark mainly measures the performance of the HTTP parsing and database libraries, often in a suboptimal default configuration. In node land, they're admittedly not amazing, but nothing about the language prevents them from being better.

For example, for a long time, the only reason that node was slow on these benchmarks was the built-in URL parser. Replacing it with this carefully written module https://www.npmjs.com/package/fast-url-parser resulted in 2x improvements on the benchmark. I haven't looked closely at the situation nowadays but I imagine its still quite similar with lots of low hanging fruit lying around and stalled due to backward-compatibility concerns.

For proof find "es4x" in the benchmark list, which basically replaces the entire stack of HTTP parsing and database libraries with the ones from vert.x and runs JS on Graal, even though Graal is currently at least 2.5x slower than V8 in terms of JS performance: https://github.com/graalvm/graaljs/issues/74

Node core (and the libraries around it) has unfortunately stalled in the "good enough" zone for quite a while. The good bit is that they stay in the good-enough zone after adding your own code.

[fahrradflucht]

I agree, although for server side rendering of web frontends, it is still the best choice even though producing vdom and rendering it to string is usually cpu bound.

[bogdanu]

> (mainly TypeScript) - Insufficient reflection capabilities

You can always use the compiler api to extract type information. Sure, a bit tricky but doable.

The compiler api exposes a lot of awesome things, I'm suprised there aren't that many tools that use it.

[0x445442]

With respect to AWS Lambda specifically, a real consideration is how fast the context loads. In the absence of warming schemes, any performance gains you get using Java for example, might be out weighed by the the cost of loading a much larger (in size) context. In general, the zip for the Lambda implemented in Nodejs is going to be considerably smaller than the zip implmented in Java.

[ignoramous]

NodeJs, in particular, out performs all language run-times in terms of AWS Lambda cold-start time (<200ms), and is second-best for warm-starts (<11ms) behind Python (<9ms).

https://levelup.gitconnected.com/aws-lambda-cold-start-langu...

[christophilus]

It’s handy having a single language and toolset for programming. There’s less friction when switching between codebases. I’ve been paid to program in a whole lot of languages over my career, and personally preferred the environments where I didn’t have to switch languages all the time when working in different layers of the stack.

These days, I’m mostly paid to write web applications. So, I default to JavaScript. That said, there are plenty of problems where JavaScript is such a poor fit that I’ll reach for something else.

[k__]

Deno comes with TypeScript out of the box.

[chrisseaton]

It's a Turing-complete, general-purpose programming language. What should stop you using it for anything?

[tnolet]

Node is especially good at handling async stuff. It’s in the JavaScript DNA. Whether through callbacks, Promises or async/await. As many backend apps are basically tying together 3rd party and 1st APIs it’s actually a very logical choice.

[ZitchDog]

Kind of silly in a lambda though, as lambdas do not run concurrently. A bash script would suffice.

[ignoramous]

Lambda for NodeJs doesn't / won't run multiple requests in the same process but different V8 contexts at the same time?

That sounds wasteful, and imo makes Cloudflare's Serverless tech superior for strictly network-io bound workloads. Lambda, to be fair, supports way many event triggers and all sorts of runtime and user-space constructs, but still manages warm start times <10ms which is really impressive.

[manigandham]

Cloudflare uses V8 Isolates feature and they build a lot of the API backends to follow the WebWorker spec. It's very efficient and well-suited for logic running in network calls at the CDN edge but limited to Node/JS code. [1]

AWS Lambda uses their Firecracker micro-vm tech which supports more runtimes, environments and triggers than just Node and also runs container workloads. [2]

1. https://www.infoq.com/presentations/cloudflare-v8/

2. https://firecracker-microvm.github.io/

[fahrradflucht]

But you can still have concurrency while handling a single invocation.

[virmundi]

I’m new at this, and would love feedback if I’m wrong, but I think that a lambda instance could be reused if the system had need. As a result the best practice is to closure global-like data in the handler function itself. This would then get passed down through the layers much like Golang’s context.

[Normal_gaussian]

I have many that call a handful of backend services, handle errors, process the data, and send it to the client.

Very async.

[manigandham]

I find that to be a mess compared to other languages like C# that have much more solid and clean async/await implementations.

[tracker1]

A big part of it is "because you can". I will say, that in general, and acknowledging I have a solid JS understanding, doing projects with node tends to be considerably less friction, because it's so easy to work around friction.

With npm, there are lots of options... generally if one package doesn't provide an interface I like, or performs poorly, or just has odd dependencies it shouldn't need, there's another that's probably closer to what I want. Worst case, if there's a smaller change, I fork the project on github, update to a scoped name, and publish my fork.

It's not always pretty. That said, I tend to be considerably more productive with it.

I've also worked a lot with C#, and am recently learning Rust... I feel the first version of most things should be done in a scripted language, and JS/Node is just as valid as any other option in the space.

[warpech]

IMO, Node's superpower is not really in the language but in the NPM ecosystem. If that is not attractive to you (maybe you develop the whole system by yourself, or need fine-tuned performance), Node only has to offer an ubiquous language, and that's about it.

[hombre_fatal]

People overlook the fact that JS is a pretty nice modern language in its own right. "Because you can" is pointlessly dismissive and indicates to me that one still thinks JS is nothing more than a language to toggle a CSS class on <h1> because they haven't used it since people used the word DHTML.

- async-everything, not a blocking language where anything async relegates you to a subecosystem like you have it in basically every other language from Java to Rust to Python.

- simple stdlib Promise that everything uses. no fragmentation between competing byob futures and byob abstractions.

- async/await.

- single-threaded making it ideal for i/o workloads, a crawler being the perfect example.

For these reasons I think it's one of the best languages. Certainly didn't used to be this way.

[jtdev]

> “ JS is a pretty nice modern language“

Are we talking about the same JS here? Because the JS I know is a dumpster fire of a language.

[Drakim]

Also just the fact that JS runtimes have had a lot of hard work put into them to make them extremely fast (motivated by the fact that JS is so widely used on the web).

[geewee]

I think it's partly "you can", but surprisingly enough I also feel that node's single-threaded architecture makes code surprisingly easy to reason about.

[sk0g]

Golang is about as far from single threaded as you can get, and I find it much easier to understand than Node. I actually switched from Node to Go on a Greenfield project, and I find myself much more productive now, and when things need to be done in parallel, Go is much easier to read, write, and maintain.

Edit: to respond to the GP, I think sharing a language between front end and back end can enable people on either side go full stack with more ease. That's a benefit I guess, but not one I considered of particular value.

[virmundi]

I’ve seen people think they’re reasoning about it when I fact they are not. I was on a project where the developers we Node pros for years. They were flummoxed when they couldn’t introduce transactions because they didn’t close over the database connection. When I said they had to hold the same connection through all the method calls, I was told I was wrong and that a global pool would be able to handle that without some external tracking mechanism.

[rcarmo]

Until you have to deal with aaync/callback/promises and mixing those to deal with various library dependencies.

I would love to see JavaScript turn into a less verbose, more consistent and less golf-oriented language.

[hombre_fatal]

I don't really see how JS could be less verbose in its concurrency-management side.

    result = somePromise() // run while the next promises are resolving
    pages = await Promise.map(url, url => crawlUrl(url), {
      concurrency: 4
    })
    allResults = await Promise.all([result, pages])

is some of the simplest concurrency code there is. You can keep chucking in more async logic and it doesn't get much more difficult to understand and doesn't introduce much more code.

If you want verbosity, look at Go's equivalent (wait groups) or, god forbid, any concurrency management in Swift.

The only problem you run into with callbacks imo are event-emitters like streams which you need to actually understand. Though I don't think this is any more trivial in other languages with evented/callback APIs like Java and Rust, I think reasoning about event callbacks is always harder for humans but a useful construct and necessary evil in evented code.

[franzwong]

Friction is less when I write node.js application. This language and runtime keep evolving.

Sometimes I can remove dependencies and code because they are replaced by the new feature of the runtime, thus the maintenance becomes easier.

[d_t_w]

I use node.js in AWS Lambda because I run Clojurescript on top of it.

My application (https://operatr.io/) is Clojure (JVM / Back-end), Clojurescript (Browser / Front-end), Clojurescript (AWS Lambda) - there's enormous leverage is using one language in all cases.

I have at least one function that exists in all three environments, but more than that I have one common language for delivery.

[makstaks]

I'm not sure if I understand your question. What do you mean by "web scripting framework?". NodeJS is a server-side runtime. It just so happens to use Javascript as the programming language. Its strength is handling async event-driven requests, so given that AWS Lambda's strength is existing within an event-driven architecture, it's a natural pairing.

[dragonwriter]

> Not knowing this myself, could someone explain the attraction in using a web scripting framework for stuff like this?

Node is not a web scripting framework, though the fact that JS is used as the main language for web front end is a big part of the attraction; as it lets frontend and backend share code and be served by the same language competency.

[yyyk]

It's mainly economic. There are plenty of JavaScript web developers. Now management can take them off the web silo and put them to work anywhere.