[stfurkan]

I'm currently working on an open-source engine [1] exactly for this purpose. If anyone wants to try or has any suggestions, I'm happy to listen :)

1. https://github.com/stfurkan/bitgpu

2. https://aidekin.com --> this is one of my projects that's currently using bitgpu engine

[espetro]

Cool stuff! I see you're targeting two verticals here (runtime on top of ONNX + TS library), I'm curious did you consider simply creating a AI SDK Provider (https://ai-sdk.dev/providers/community-providers/custom-prov...) instead of crafting a separate library?

Besides it, how are you dealing with the harness the model can use on web? I'm currently building https://github.com/bolojs/bolo to extend the base harness agents can use on web and I'm interested in hearing from your experience on crafting tools for bitgpu

[stfurkan]

Thank you :) I wanted to create it from zero myself so I can control it better. e.g. I started with ONNX support but now it support gguf models too. It might be a good idea to add some popular connectors in the future. Good luck with your project :)

[mgoetzke]

Interesting. But trying the demo with 27G model and simple 8+5 did not work.

Guessing there is some work to do still

``` calculate({"expression":"g"}) → error: only numbers, + - * / and parentheses are allowed error: No user query found in messages. What is 8+5? Use the calculator get_time({}) → Mon Jul 20 2026 10:36:28 GMT+0200 (Central European Summer Time) error: No user query found in messages. hi Hello! 2 tokens · 6.2 tok/s · prefill 4563 ms · confidence 87% which tools do you know about ? I'myepic know about the following are two tools I am aware of course, I currently i know about I know about me know I am aware of course I know I know I am aware of the following

calculate({"expression":" I know I am aware of know I am aware of I am aware I know I know know "}) → error: only numbers, + - * / and parentheses are allowed error: No user query found in messages.```

[stfurkan]

Thank you for trying it :) I just added the 27B support 1 hour ago. Let me check and try to fix it. All other models should work as expected but if you have issues with them too, I can take a look.

[stfurkan]

Hi again :) I updated the engine and 27B model should work in the demo[1]. We can still consider it as experimental but I am trying to make it better and efficient.

1. https://stfurkan.github.io/bitgpu/examples/chat.html

[luca-3]

This is great for high-privacy use cases. If you could add an agentic loop that can call tools within the website using the user’s session, that would be great. Then the user can chat with an AI about sensitive information in the most secure way possible.

Also, do you have data on the technical requirements for this? If someone uses an old phone, e.g., does it still work?

[stfurkan]

Hi, the engine itself already has tool calling capabilities but if you're asking for the aidekin I didn't add tool calling there on purpose.

It needs webgpu support but I haven't had a chance to test with different devices.

[EGreg]

Yes, I’m very interested in exploring this stuff.

Want to connect and discuss it? https://calendly.com/safebotsai

[stfurkan]

Hello, I sent you a LinkedIn connection request. We can chat first and schedule a meeting. Thanks

[alex7o]

Hay man, I really want a bonsai style embeddings or re-ranking model for you think sth like that is feasible?

[stfurkan]

Hi :) There are some smaller embedding and re-ranking models (I am currently using some of them on my aidekin project) but having it like bonsai models will be great to have.