Hacker News new | ask | show | jobs
by nine_k 808 days ago
Could LogQL do.something like

  select * from stdout, stderr
  where session_id = 123456
? If not, why?
3 comments

yes it can, if you tag your log stream correctly - either by having the stream externally tagged via attributes, or internally by following certain conventions in the log line.

You can also do something like

select client_ip from requests where elapsed_ms > 10000

which is incredibly powerful

yep, with the caveat that you probably don't want to have the backend of whatever log system you use (not exactly sure how Loki does it) to have an index on something as high-cardinality as session id so that query could get slow.

But these log query systems can also optimize these queries for instance by by sampling, using distributed trace ids to ensure you get shown corresponding, allowing you to get only logs where at least one step in the trace errored, etc.

strace and gdb can trace and close and reopen process file handles 0,1,2.

ldpreloadhook has an example of hooking write() with LD_PRELOAD=, which e.g. golang programs built without libc don't support.

When systemd is /sbin/init, it owns all subprocess' file handles already, so there's no need to close(0), time, open(0) with gdb.

Without having to logship (copy buffers that are flushed and/or have newline characters in the stream) to a network or local Arrow database files and or SQLite vtables,

journalctl (journald) supports pattern matching with: -t syslogidentifier, -u unit; and -g grepexpr of the MESSAGE= field:

  journalctl -u <TAB>
  journalctl -u init.scope --reverse
  journalctl -u unit.scope -g "Reached target"  # and then "/sleep" to search and highlight with less
  
  journalctl -u auditd.service

  # this is slow because it's a full table scan, because
  # journald does not index the logfiles;
  # and -g/--grep is case insensitive if the query is all lowercase:
  journalctl -g avc --reverse
  journalctl -g AVC --reverse

  # this is faster:
  journalctl -t audit -g AVC -r

  # this is still faster,
  # because it only searches the current boot:
  journalctl -b 0 -t audit -g AVC

  # these are equivalent:
  journalctl -b 0 --dmesg -t kernel
  journalctl -k

  # 
  journalctl -b 0 --user | grep -i -C "xyz123"
There is a GNOME Logs viewer that has 'All' and a few mutually exclusive filter/reports in a side pane, and a search expression field to narrow a filter/report like All or Important.

There is a Grafana Loki Docker Driver that logships from all containers visible on that DOCKER_HOST docker socket to Grafana for querying with Loki: https://grafana.com/docs/loki/latest/send-data/docker-driver...

Podman with Systemd doesn't need the Grafana Docker Driver (or other logshippers like logstash, loggly, or fluentd) because systemd spawns containers and optionally pipes their stdout/stderr logs to journald.

Influx has Telegraf, InfluxDB, Chronograf, and Kapacitor. Chronograf is their WebUI which provides a query interface for configurable chart dashboards and InfluxQL.

Grafana supports SQL, PromQL, InfluxQL, and LogQL.

Graylog2 also indexes logfiles.

But you can't query stdout and stderr you or /sbin/init haven't logged to a file.