Turning a Splunk dashboard into a hands‑on threat‑hunting playbook: a step‑by‑step recipe for real‑world detection agility -

Turning a Splunk Dashboard into a Hands‑On Threat‑Hunting Playbook

Splunk dashboards are great for visualizing data, but they rarely give you a ready‑to‑execute hunting workflow. In a real‑world environment you need a playbook that tells you what to look for, how to filter the data, and what to do when you find something. This post walks through turning an existing Splunk dashboard into a fully documented playbook that can be run by a sysadmin or a SOC analyst on a Linux host. The focus is on practical commands, trade‑offs, and troubleshooting tips that fit the Linux‑security‑first mindset of the Linux Security blog.

Why a Playbook Matters

A dashboard is a static snapshot. A playbook is a living set of instructions that can be executed, reviewed, and updated. In 2026, many teams still rely on manual copy‑paste from dashboards into Splunk queries or into a SIEM‑agnostic tool. That approach is error‑prone and hard to audit. A playbook, on the other hand, can be version‑controlled, reviewed by peers, and run automatically via cron or a lightweight orchestrator like systemd‑timer.

1. Pick a Dashboard to Re‑engineer

Start with a dashboard that already covers a high‑impact event, such as:

Failed SSH logins – a classic indicator of brute‑force attempts.
Unusual outbound DNS queries – often a sign of data exfiltration.
Privilege escalation attempts – e.g., sudo usage outside normal windows.

For this walkthrough we’ll use a Failed SSH logins dashboard that shows:

index=linux sourcetype=sshd
| stats count by src_ip, user, status
| where status="Failed"
| sort -count

The dashboard visualises the top 10 IPs and users with the most failures.

2. Define the Hunting Goal

A playbook must answer three questions:

What – the specific indicator or pattern.
Why – the threat context.
How – the steps to confirm and remediate.

Goal: Detect and block IPs that have more than 50 failed SSH attempts in the last 24 h, then add them to the host’s firewall.

3. Translate the Dashboard into a Reusable Query

The dashboard query is fine for a visual report, but a playbook needs a deterministic, reproducible query. Add a time window and a threshold:

index=linux sourcetype=sshd earliest=-24h@h latest=now
| stats count by src_ip, user, status
| where status="Failed" AND count>50
| table src_ip, user, count

Why this matters: Splunk’s earliest/latest syntax is more robust than relying on the dashboard’s time picker. It also makes the query idempotent when run from a script.

4. Export the Query to a File

Create a file that can be sourced by a shell script:

# /usr/local/bin/hunt_ssh_failures.sh
#!/usr/bin/env bash
set -euo pipefail

SPLUNK_QUERY='index=linux sourcetype=sshd earliest=-24h@h latest=now
| stats count by src_ip, user, status
| where status="Failed" AND count>50
| table src_ip, user, count'

# Run the query via Splunk REST API
TOKEN=$(cat /etc/splunk/api_token)
URL="https://splunk.example.com:8089/services/search/jobs"
RESPONSE=$(curl -sk -u admin:$TOKEN -d "search=$SPLUNK_QUERY" -d "output_mode=json" $URL)
JOB_ID=$(echo $RESPONSE | jq -r '.sid')

Trade‑off: Using the REST API requires a token with search permissions. If you prefer the CLI, splunk search can be used, but it requires the Splunk binary on the host.

5. Poll for Job Completion

Splunk jobs are asynchronous. Poll until the job is done:

while true; do
  STATUS=$(curl -sk -u admin:$TOKEN "$URL/$JOB_ID" | jq -r '.entry[0].content.isDone')
  if [[ "$STATUS" == "true" ]]; then
    break
  fi
  sleep 2
done

Troubleshooting: If the job never finishes, check the Splunk server logs (/opt/splunk/var/log/splunk/splunkd.log) for errors like “search job timed out”.

6. Retrieve Results

RESULTS=$(curl -sk -u admin:$TOKEN "$URL/$JOB_ID/results?output_mode=json")
IP_LIST=$(echo "$RESULTS" | jq -r '.results[].src_ip')

If jq is not installed, you can parse JSON with python -m json.tool, but jq is the de‑facto standard for shell scripts.

7. Block the IPs with iptables

For each IP, add a rule to drop inbound SSH traffic:

for IP in $IP_LIST; do
  iptables -I INPUT -p tcp --dport 22 -s "$IP" -j DROP
  logger -p local0.warn "Blocked SSH from $IP due to >50 failed logins"
done

Caveat: iptables rules are volatile across reboots. Persist them with iptables-save and iptables-restore, or use nftables which is the modern replacement:

nft add rule inet filter input tcp dport 22 ip saddr "$IP" counter drop

Persist with /etc/nftables.conf or systemd units.

8. Notify the Team

Send a Slack or email alert. For Slack, use curl with a webhook:

WEBHOOK="https://hooks.slack.com/services/XXX/YYY/ZZZ"
MESSAGE="Blocked SSH from $IP due to >50 failed logins in last 24h."
curl -X POST -H 'Content-type: application/json' --data "{\"text\":\"$MESSAGE\"}" $WEBHOOK

Tip: Keep the webhook URL in a secure file with chmod 600.

9. Log the Action

Write a structured log entry for audit:

echo "$(date -u +%FT%TZ) BLOCKED_IP $IP COUNT $(echo "$RESULTS" | jq -r ".results[] | select(.src_ip==\"$IP\") | .count")" >> /var/log/ssh_block.log

Use logrotate to keep the log manageable.

10. Wrap It in a systemd Service

Create /etc/systemd/system/hunt-ssh.service:

[Unit]
Description=SSH Failure Hunting Service
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/hunt_ssh_failures.sh
User=root
Group=root
Environment=PATH=/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin

[Install]
WantedBy=multi-user.target

Create a timer /etc/systemd/system/hunt-ssh.timer:

[Unit]
Description=Run SSH failure hunt every 30 minutes

[Timer]
OnBootSec=5min
OnUnitActiveSec=30min
Unit=hunt-ssh.service

[Install]
WantedBy=timers.target

Enable and start:

systemctl enable --now hunt-ssh.timer

Now the playbook runs automatically, logs actions, and blocks offenders.

11. Validate the Playbook

Simulate failures: Use ssh from a test IP with wrong passwords to generate log entries.
Run the timer manually: systemctl start hunt-ssh.service.
Check iptables: iptables -L INPUT -v -n | grep 22.
Review logs: tail -f /var/log/ssh_block.log.

If the IP is blocked, the playbook works. If not, check:

Splunk token permissions.
Network connectivity to Splunk’s REST endpoint.
Firewall rule syntax.

12. Extend the Playbook

12.1. Add a Whitelist

Maintain a file /etc/ssh/whitelist_ips and skip those:

if grep -q "^$IP$" /etc/ssh/whitelist_ips; then
  continue
fi

12.2. Rate‑Limit Instead of Drop

If you want to allow a few attempts before blocking:

iptables -I INPUT -p tcp --dport 22 -s "$IP" -m recent --set
iptables -I INPUT -p tcp --dport 22 -s "$IP" -m recent --update --seconds 300 --hitcount 5 -j DROP

12.3. Integrate with Fail2Ban

Fail2Ban already watches /var/log/auth.log. You can feed the Splunk query results into Fail2Ban’s jail configuration, but the Splunk‑based approach gives you a central view across multiple hosts.

13. Security Trade‑offs

Decision	Benefit	Risk
Use `iptables`	Mature, widely supported	Volatile across reboots
Use `nftables`	Modern, persistent	Requires learning new syntax
Store Splunk token in `/etc/splunk/api_token`	Simple	If file permissions slip, token leaks
Log to `/var/log/ssh_block.log`	Auditable	Log rotation needed

Always audit file permissions and ensure the playbook runs as root only if necessary. If you can run the Splunk query as a non‑root user, do so to reduce blast radius.

14. Common Pitfalls

Symptom	Likely Cause	Fix
No IPs blocked	Splunk query returns empty	Verify `earliest`/`latest` and `status="Failed"`
Splunk job fails	Token expired	Rotate token, update `/etc/splunk/api_token`
Firewall rule not applied	`iptables` not persistent	Use `iptables-save` or switch to `nftables`
Slack alert missing	Webhook URL wrong	Test webhook with `curl -X POST` manually
Script exits with `jq` error	`jq` missing	Install via `apt install jq`

15. Maintenance Checklist

Rotate Splunk token every 90 days.
Review whitelist regularly; remove stale entries.
Check logrotate config for /var/log/ssh_block.log.
Test the timer after system updates.
Audit firewall rules with iptables -L -n -v or nft list ruleset.

16. Going Beyond

Once you have a working playbook, you can:

Add anomaly detection: Use Splunk’s machine‑learning toolkit to flag unusual patterns.
Correlate with other data: Combine SSH failures with failed sudo attempts.
Export to a SIEM: Push the block events to a central SIEM for correlation.

17. Final Thoughts

Turning a Splunk dashboard into a hands‑on playbook is a matter of:

Making the query deterministic.
Automating the retrieval and action steps.
Persisting state and logging for audit.
Integrating with existing Linux tools (iptables, systemd, logging).

The result is a repeatable, auditable process that turns raw data into defensive action. For Linux sysadmins, this approach keeps the focus on the host while leveraging Splunk’s powerful analytics.