[benjreinhart]

Hey Chris , can you further qualify performance?

Before I share some thoughts on this, let me just say that our primary motivators for Axilla have much more to do with bringing better AI tooling to an otherwise flourishing ecosystem rather than shaving milliseconds off an arbitrary task or request. Given that, I'm not sure how fruitful a performance discussion will be.

If by performance you meant maturity of third party packages for AI-related functionality, then yes JS/TS is lacking. This is what is motivating us :). We want better tooling for AI applications in TS.

If you're referring to performance for CPU-bound tasks, then yes JS would not be as good as lower-level languages like Rust or Go. If you're referring to JS compared to Python, then I don't know how true that is. Python doesn't have a great concurrency story either (at least not today). JS may be single threaded for the most part, but with web workers and WASM (+ WebGPU!), we now have tools at our disposal for dramatically speeding up CPU-bound tasks while not blocking the main thread. Assuming we get the interfaces right, we can swap out a subset of the implementation with a WASM-based implementation later if justified.

There is nothing about Python the language that makes it especially well-suited for AI/ML-related functionality. It is just the language whose ecosystem has the most maturity when it comes to that functionality. We hope to chip away at that over time.

[fullstackchris]

I'm no expert in actual ML implementations but I was under the impression that Python (i.e Tensorflow) is actually C/C++ based under the hood. I just meant I can't imagine the V8 engine can be as performant for all that matrix math in those models.

But now that I'm looking at the actual code samples, I'm not even sure JavaScript is doing any of the actual heavy lifting? (I see you use OpenAI's embedding) so this tool is more of the glue connecting all the parts? Again, I'm out of my wheelhouse here.

[benjreinhart]

Ahh yes, right now we're operating at a higher-level of the stack.

That said, we are investigating serving from Node and possibly on edge devices with WebGPU. For serving from Node, it would be similar to what you describe with Tensorflow compiling down to C/C++. There are various backends for frameworks like Tensorflow, Pytorch, etc. and those backends are often C/C++. We would bridge this lower-level code to Node through e.g. Node API (https://nodejs.org/api/n-api.html) or use frameworks like ONNX / ONNX Runtime.