Hacker News new | ask | show | jobs
by 10165 3306 days ago
"grep ^2017-05-29 /var/log/somefile | grep -v 'INFO|WARN' | tail -n5 | cut -f1 -f3"

Did you mean cut -d' ' -f1,3?

Assuming delimiter is a space the above example can be reduced to:

    sed '
    /^2017-05-29/!d;
    /INFO/d;
    /WARN/d;
    s/ /INFO/;
    s/ /INFO/;
    s/ .*//;
    s/INFO.*INFO/ /;
    ' /var/log/somefile \
    |tail -n5
"... compile a set of commands into a single program and run that."

Linux: see busybox ("multicall binary") BSD: see binaries on install media ("crunched binary")

sed and tail are both compiled into busybox re: bsd compiling in tail with crunchgen is easy

As for "objects", the k language can mmap the output of UNIX commands and run parallel computations on those results as "data" not "text". It can be faster than I/O.
1 comments
eriknstr 3306 days ago
I intended for the delimiter to be tab in this example.

Using sed or awk is an option, yes, but I am so used to the standard utilities that I would rather keep using them.

Also I'm not sure if your sed script does what I intended for it to do.

1. Take ever line that starts with 2017-05-29.

2. Out of the lines we have, remove any that contain INFO or WARN.

3. Take the five last of all of those lines.

4. Take the first and third fields of all of those lines.

Let's create a sample file

    cat > /tmp/somefile <<EOF
    2017-05-28T08:30+0200	nobody	ERR: Foobar failed to xyzzy.
    2017-05-29T13:01+0200	nobody	INFO: Garply initiated by grault scheduler.
    2017-05-29T13:37+0200	nobody	DEBUG: Garply exited with 0.
    2017-05-29T14:12+0200	nobody	WARN: Plugh quux corge.
    2017-05-29T14:55+0200	nobody	ERR: PLUGH QUUX CORGE!
    2017-05-30T00:17+0200	nobody	ERR: Failed to retrieve baz needed for thud.
    EOF
(Note that due to how HN formatting works you won't be able to copy-paste this because HN requires leading space to format as code but then it includes the spaces in the output also and we don't want leading space in the file.)

    grep ^2017-05-29 /tmp/somefile | egrep -v 'INFO|WARN' | tail -n5 | cut -f1 -f3
Note: I had a typo in my original example where I said "grep -v" where I meant "egrep -v".

Result:

    2017-05-29T13:37+0200	DEBUG: Garply exited with 0.
    2017-05-29T14:55+0200	ERR: PLUGH QUUX CORGE!
Running it with your pipeline

    sed '
    /^2017-05-29/!d;
    /INFO/d;
    /WARN/d;
    s/ /INFO/;
    s/ /INFO/;
    s/ .*//;
    s/INFO.*INFO/ /;
    ' /tmp/somefile \
    |tail -n5
results instead in:

    2017-05-29T13:37+0200	nobody	DEBUG: exited
    2017-05-29T14:55+0200	nobody	ERR: QUUX
But that's just because you assumed that space was the delimiter when it was not.

Your use of the word "INFO" as a placeholder for the first two occurrences of the space character threw me off quite a bit when reading your script, since the word "INFO" occurs in the file we are working with itself, but it makes sense to use a word that we know is no longer possible to be present since we've already removed all lines that contain it. However, while a neat trick, those kinds of strange hacks are the kinds of things that has brought me to believe that having objects (like they have in Microsoft PowerShell) instead of pure text would be beneficial in Unix also.

---

As for your comments on compiling into a single program and run that, I don't think you understood what I meant, or I don't think that busybox and those chrunched binaries you mentioned perform the optimization I am talking about, do they.

---

Using the k language you mention in that fashion seems more like a hack and will require a lot of work each time. I would rather rewrite all core commands of my system so that they produced true objects, or actually, rather than objects just structured binary data. I don't need the output to have methods you can call.

One of the main things I want from structured binary data is to be able to select the columns of data by name instead of by index and without having a mess of some commands using tab for delimiter and others space and so on.

So instead of

    zfs list -H | cut -f1,3,4
I would like to

    zfs list | cut name used avail
for example, or something like that.

Also all commands that output tabular data must have a "header" command that will show the column headers. So to see the headers that the 'list' subcommand of the zfs command will output, I would say

    zfs header list
And it would tell me

    NAME	USED	AVAIL	REFER	MOUNTPOINT
and furthermore since I am authoring the shell, when you tab to complete a command it will call the binary with the header subcommand and output the headers so that you can have them easily accessible while you're working on a pipeline. And furthermore this shall work with pipes already present.

So if I type

    zfs list | cut
and then tab to complete, it will call

    zfs header list
and pipe that to

    cut header
which will return the input it saw

Naturally, "header" will be a reserved word.

All commands will understand how to work with tabular data.

When you use grep you will either specify which column to use, or you can tell it to look across all columns with *

The shell will only expand * to file names when the * is positionally last in the argument list of a command since all commands will take the list of files as the last ones in their list of arguments.

---

All of this being said, I appreciate all replies, including yours.

link
10165 3306 days ago
This is the output I got, tab separated:

   2017-05-29T13:37+0200 DEBUG: Garply exited with 0.
   2017-05-29T14:55+0200 ERR: PLUGH QUUX CORGE!
                        ^tab
All you have to do is substitute tab for space.

Ctrl-V then tab. Or if using GNU sed just type \t.

We can reject this simplicity as a "trick" and demand something more complex.

But that suggests that the goal is not to solve the problem, it is to satisfy someone's desire for having some underlying complexity that moves the solution out of the realm of "trivial".

link