Summarizing /var/log/syslog Errors into JSON with jq for Grafana Dashboards

 Posted on October 6, 2026  |  4 minutes  |  755 words  |  Robert Nilsson

Why JSON‑friendly syslog matters

Syslog is the default source for almost every Linux service. When an error happens, the kernel or a daemon writes a line that looks like this:

Oct  6 12:34:56 myhost kernel: [12345.678901] EXT4-fs error (device sda1): ext4_find_entry:1073: inode #123456: comm: myapp: file name too long

Grafana loves structured data. A plain text file is hard to query, filter, or aggregate. By converting the error lines to JSON you can:

  • Filter by severity (error, warning, info) in a single query.
  • Index timestamps for time‑series charts.
  • Join with other metrics (e.g. CPU load) without complex parsing.
  • Export to other tools (Prometheus, Loki, Elastic) with minimal friction.

The real trick is keeping the pipeline lightweight so you don’t add latency to log ingestion or chew up too much disk space.

Typical syslog layout

The format is defined by the rsyslog or systemd-journald configuration. A minimal line looks like:

TIMESTAMP HOSTNAME TAG[PID]: MESSAGE
  • TIMESTAMP – usually Mmm dd hh:mm:ss or ISO‑8601 if systemd-journald is used.
  • HOSTNAME – the machine name.
  • TAG[PID] – the program name and optional PID.
  • MESSAGE – the actual log text.

If you’re on a recent Debian/Ubuntu system, systemd-journald writes to /var/log/syslog in the same format. For reference, see the systemd documentation.

Extracting error lines

The first step is to isolate the lines that contain an error. grep is fast, but it can be noisy if the message contains the word “error” in a different context. A safer approach is to use awk to match the priority field that rsyslog injects when configured with the RSYSLOG_SyslogProtocol23Format template:

$ sudo tail -n 1000 /var/log/syslog | awk '$0 ~ /error|err|critical|fail/i'

If you’re using systemd-journald, the priority is in the PRIORITY= field. You can extract it with journalctl:

$ sudo journalctl -p err -n 1000

For the rest of this article we’ll assume the plain text file and use grep with a case‑insensitive pattern.

Building a JSON stream

jq is the de‑facto JSON processor for the shell. It can read a stream of text, split it into fields, and output JSON objects. The following one‑liner turns a syslog line into a JSON object:

grep -iE 'error|critical|fail' /var/log/syslog |
awk '{
    # split the line into timestamp, host, tag, and message
    match($0, /^([A-Z][a-z]{2} +[0-9]{1,2} +[0-9]{2}:[0-9]{2}:[0-9]{2}) ([^ ]+) ([^:]+): (.*)$/, m)
    if (m[1]) {
        # convert to ISO‑8601 for Grafana
        cmd = "date -d \"" m[1] " " strftime(\"%Y\") "\" +%Y-%m-%dT%H:%M:%S"
        cmd | getline iso
        close(cmd)
        printf "{\"timestamp\":\"%s\",\"host\":\"%s\",\"tag\":\"%s\",\"msg\":\"%s\"}\n",
               iso, m[2], m[3], m[4]
    }
}' | jq -c .
  • The awk part parses the line with a regular expression.
  • date converts the month/day/time to ISO‑8601; we append the current year because syslog omits it.
  • jq -c . prints each object on its own line, ready for ingestion.

You can pipe the output to a file:

./syslog_to_json.sh > /var/log/syslog_errors.json

or stream it directly to Grafana’s JSON data source.

Feeding Grafana

Grafana’s JSON API data source (available as a plugin) expects a JSON array or a newline‑delimited stream. The simplest setup:

  1. Install the plugin (Grafana 10+):

    grafana-cli plugins install grafana-json-datasource
systemctl restart grafana-server

  2. Create a data source in Grafana pointing to the file or a local HTTP endpoint that serves the stream. If you prefer a file, use the File data source type:

    • URL: file:///var/log/syslog_errors.json
    • Format: JSON Lines

  3. Build a panel that queries the field msg and groups by tag. Example query:

    {
  "query": "SELECT msg, tag FROM \"syslog_errors\" GROUP BY tag"
}

  4. Add a time filter using the timestamp field to plot error counts over time.

If you want real‑time updates, run the conversion script as a systemd service that tails /var/log/syslog and writes to a named pipe:

[Unit]
Description=Stream syslog errors to JSON for Grafana
After=network.target

[Service]
ExecStart=/usr/local/bin/syslog_to_json.sh
StandardOutput=append:/var/run/syslog_errors.json
Restart=on-failure

[Install]
WantedBy=multi-user.target

Then point Grafana to /var/run/syslog_errors.json. The pipe ensures the file is always fresh.

Performance and rotation

Disk usage: JSON objects are larger than plain text. For a busy server, a 10 GB syslog file can become 20 GB of JSON. Mitigate by:

  • Compressing the output (gzip -c > errors.json.gz) and using Grafana’s Compressed JSON support.

  • Rotating the JSON file with logrotate:

    /var/log/syslog_errors.json {
    daily
    rotate 7
    compress
    missingok
    create 0640 root adm
}

CPU load: awk + date + jq is lightweight for a few thousand lines per second. If you hit a bottleneck, consider:

  • Using systemd-journald’s native JSON export (journalctl -o json).
  • Writing a small Go or Rust program that parses syslog more efficiently.

Memory: jq -c . streams objects, so it never buffers the whole file. The main memory hog is the awk date

linux  open-source 

See also