Hacker News new | ask | show | jobs
by NortySpock 33 days ago
If I can ask a separate question: what scalability problems did you run into with Victoria{Metrics|Logs|Traces}, and at what scale did you hit them?

VictoriaMetrics and Logs have worked fine in my quiet homelab, and VictoriaMetrics appeared to work great for the infrastructure team of an open source online video game I contribute to (say about 10 physical nodes and 20 applications/services ) ... I was going to suggest VictoriaLogs to them next but wanted to ask what roadblocks could come up.

1 comments

I honestly think you are a bot. When ever I see Victoria mentioned it is always the same, always asking about hitting a scaling problem + promoting it, never responding to any comments. Hope I'm wrong, but it's been one too many. I refuse to use a product that is this dishonest.
I work at VictoriaMetrics.

Just to clarify: VictoriaMetrics doesn't use bots for HN or for any other media for promotion.

I don't know the person who you responded to. Most of the activity you see is coming from community members who genuinely use the project or from the core engineering team trying to answer user's questions or address misunderstandings.

> never responding to any comments

Could you please share examples like this? I can't say for community members, but our internal policy for engineers is very much focused on great support. You can check our slack/github to see that every question is answered and well explained.

Hi, first of all, thank you for your response, I really appreciate actual Victoria team commenting.

This is probably the 5th comment (almost identical) I have seen about VictoriaMetrics, mostly on Reddit. I engaged with a few trying to learn more about your product and eventually just gave up. If you really want you can comb through my reddit comments, but be warned, I have commented on a lot of things... a lot...

You should be proud of what you have built, I've looked a bit more and your product looks incredible. I personally think that a sales person might have been testing an automation tool, but if it was actual customers that just shows how good the product is!

Traceway is not working on addressing the problems y'all are solving, it is more focused on having an out of the box experience with preconfigured dashboards, SLOs, integrations, automatic endpoint ranking, frontend session replays/RUM, symbolication etc.

Again, thank you for your comment.

Hi, I am not a bot. Also I do not work for VictoriaMetrics.

Please feel free to go through my post history and observe I comment on things I am interested in, like databases, servers, and video games.

Got it, sorry, your comment just looked like a bunch of others and felt extremely out of place as nobody mentioned hitting any limits, especially with the Victoria stack (that I could see).

The comment read out of place/generic and given my previous experience I incorrectly assumed it was another generic bot - my bad.

Hopefully no hard feelings and Victoria looks great.

What do you like the most about it and how was your experience scaling it?

No worries, no hard feelings, I was just surprised that what I thought was a specific response was assumed to be a generic-ish bot response. (then again, I didn't spell out that the game I contribute to is Beyond All Reason.) I do totally sympathize with the feeling of being overwhelmed by AI slop.

After digging into Traceway documentation, it looks like you were looking to primarily use OTEL for ingestion? Or would you say that's a misreading of the documentation and you actually support metrics, logging etc easily? It looks easy to setup via docker, I might try the SQLite version just to get a taste for how it works and how easily data can be ingested.

For myself, I was initially interested in the Loki/Prometheus/Grafana stack but it wasn't going to fit in the 4GB of RAM I had available on a Raspberry Pi that was already hosting two services that consumed a GB of RAM each. So when I found VictoriaMetrics (a) happily ran in 200MB of RAM (b) was used by CERN (c) had excellent, comprehensive documentation with plenty of examples (d) supported so many different ingestion and export/reporting APIs that I would be able to set up everything I wanted for my homelab without any shim scripts or one-off API converters and (e) offered a basic reporting UI with sane defaults (auto-detecting rate vs sum for a graph) even without having to set up Grafana, I was blown away and grateful that such a useful thing existed. Same for VictoriaLogs, it was just easy to set up once I put my mind to it, because the documentation for everything was very clear, and they clearly had "sane defaults + configurable options" once you needed something slightly different. Having sane support for backfills and tolerating duplicates was also nice. "Throw us your data in one of these shapes , we'll sort it out" was just nice to finally see rather than digging through pages of Prometheus documentation for what the edge cases could be if I sent duplicates or the data was from a month ago.

I just have a homelab of random docker container across a few nodes thrown together with underpowered hardware, but VictoriaMetrics met me where I was and made it trivial to experiment using the nodes I had rather than have to migrate to bigger nodes, and it was very well behaved at idle, steady-state, and "I want to trickle-feed a million data points via http calls" loads. I don't yet need OTEL, I don't have cattle, I have homelab pets and very little time to play with them. I just want to either scrape metrics or fire metrics at some sort of endpoint that can figure out what I meant if I get close enough.

But VictoriaMetrics was so easy to get working because the documentation was laid out as "here's the starter command line options, here's how you ingest data in a variety of input URLs, here's how you retrieve your data via a variety of output URLs, if you want specialty stuff that's described farther down the page..." it was about as hard as falling off a log. It just became the obvious place to base anything else around because it had so many connectors and sane defaults.

So when the Beyond All Reason infrastructure team asked "is there a infrastructure and application metrics solution for a handful of nodes that is self-hosted, easy to set up and won't break the bank or require babysitting?" I had one recommendation: VictoriaMetrics (+ Grafana)

Admittedly I do sort of wish for unified metrics and logs and traces, but that's merely a platonic ideal dream state for me. In reality I can see that both I and an organization generally sets up metrics, or logs, or traces, in a piecemeal fashion. An organization (in my limited experience) generally doesn't think about all three at once, and so the "do one thing and do it well" becomes a nice simplification of scope rather than a mark against VictoriaMetrics or VictoriaLogs not having the whole enchilada under one common roof.

I have not personally worked on scaling it horizontally yet, and I didn't set it up myself, but (a) I observe the Beyond All Reason VictoriaMetrics server has 8 GB of RAM, 3 vCPU and appears to serve 75k active time series (14.5 billion data points, ingest about 5 thousand data points per second) without complaint. The resource usage graphs are flat, humming quietly and (b) I did appreciate that the vmagent and vlagent do send to multiple targets easily (tested this with vlagent) , making "active -> standby fail-over" easy to setup -- all ingestion agents would multiplex to all sinks and you were done, any sink "should" have the same data.