[ynniv]

Are the logs being read from disk? In my experience, python is highly optimized for reading (possibly compressed) files from disk. If your infrastructure keeps logs in memory, python will lose this advantage and compete on computational performance where Haskell has the advantage. This is important for those of us who grind logs on disk and might be considering a language switch.

[enneff]

What do you mean by optimized? Python makes the same read and write syscalls everyone else does.

What you're probably observing is Python's slow code generation being masked by the inherent slowness of I/O.

[ynniv]

Python makes the same read and write syscalls everyone else does

Except, when python's pants are on, it makes gold records.

I haven't looked to see if there are any explicit optimizations, but your statement is ridiculous; an effective IO strategy can have an enormous effect on performance.

[enneff]

I'm sorry, what? Just being disagreeable is not an answer. What does "IO strategy" mean? You are being incredibly vague and unhelpful.

Reading data from a file handle into a buffer is a trivial operation. It's what you do with that data afterwards that is important. In C (or Go) you have complete control over what happens next. As for Python, I don't know what happens, but I don't see how it could possibly be more efficient than any other sane language.

[ynniv]

Reading data from a file handle into a buffer is a trivial operation.

If that is all you're doing, then yes; there isn't a much more efficient way of doing that.

In C (or Go) you have complete control over what happens next.

It is up to the programmer to know what to do next. Does haskell strike you as a language of micromanagement? Python can sometimes be multiples faster than command line grep. I haven't looked into why, but I have some ideas.

You are being incredibly vague and unhelpful.

If you don't understand what I'm talking about, it doesn't make /me/ wrong. And just saying so is also not an appropriate response. I had a specific question that was answered by the OP, and was useful to me.

Hacker News comment threads are rarely a place of education, but I will reinterpret what you said as a question.

An IO strategy is how and when you make those system calls. Reading from disk takes a vast amount of time, during which you can be doing computation. To be fast, you should be asking for the appropriate amount of lookahead, at the correct offset. Is that 4k? 1Meg? 100Megs? 1GB? Do you use threads for this? Can you skip any of the input stream? Do you let the operating system, programming language, library, or program code decide how big the read is? Where that data is stored after being read from disk is also important. Especially fast strategies use mmap to avoid copying from kernel space into user space. And of course everything is always chunked at specific intervals, so knowing where those are can sometimes reduce the number of calls. The ability to optimize for these is one thing that makes dedicated database software so successful.

It is a dark art, and it is not expected that the average person know these things. If you happen to be the kind of person working on a programming language, it could be useful to be aware of them. Here are some quick links, but there is a vast amount written on the subject.

[http://lists.freebsd.org/pipermail/freebsd-current/2010-Augu...] [http://tangentsoft.net/wskfaq/articles/io-strategies.html]

[jamwt]

Nope, this is a process that BLPOP's logs from some Redis queue, does some processing on them, then writes them to disk.