Web Server Log Analysis | Blue Team & Defense

Web server logs are the front-line record of every request hitting your web application. Every

attempt, every

XSS

payload, every directory brute-force - it's all in the logs. Learn to read them, and you'll see attacks unfold in real-time.

Think of web server logs like a guest book at your front door, except this guest book records exactly what everyone asked for - including the guy who tried every password or asked for /admin a thousand times. The information is there; you just need to know how to read it.

Two Types of Logs

Access logs record every request (who asked for what).Error logs record problems (what went wrong). Both are valuable for security analysis.

Log Locations and Formats

1Web Server Log Locations:
2 
3APACHE
4├── Access Log: /var/log/apache2/access.log (Debian/Ubuntu)
5│              /var/log/httpd/access_log (RHEL/CentOS)
6├── Error Log:  /var/log/apache2/error.log
7├── Virtual Host logs: /var/log/apache2/mysite-access.log
8└── Config: /etc/apache2/apache2.conf
9 
10NGINX
11├── Access Log: /var/log/nginx/access.log
12├── Error Log:  /var/log/nginx/error.log
13├── Virtual Host logs: /var/log/nginx/mysite-access.log
14└── Config: /etc/nginx/nginx.conf
15 
16IIS (Windows)
17├── Access Log: C:\inetpub\logs\LogFiles\W3SVC1\
18├── Format: u_ex<date>.log
19└── Config: IIS Manager → Logging
20 
21Common Log Format (CLF):
22─────────────────────────────────────────────────────────────────
23192.168.1.50 - admin [15/Jul/2024:14:22:01 +0000] 606070;">#a5d6ff;">"GET /page HTTP/1.1" 200 1234
24 
25Fields:
26├── IP Address: 192.168.1.50
27├── Identity: - (usually blank)
28├── Username: admin (or - if not authenticated)
29├── Timestamp: [15/Jul/2024:14:22:01 +0000]
30├── Request: 606070;">#a5d6ff;">"GET /page HTTP/1.1"
31├── Status Code: 200
32└── Response Size: 1234 bytes
33 
34Combined Log Format (Adds referer and user-agent):
35─────────────────────────────────────────────────────────────────
36192.168.1.50 - - [15/Jul/2024:14:22:01 +0000] 606070;">#a5d6ff;">"GET /page HTTP/1.1" 200 1234 "http://example.com" "Mozilla/5.0..."

Understanding Status Codes

1HTTP Status Codes for Security Analysis:
2 
32XX - SUCCESS
4├── 200 OK          │ Request succeeded
5├── 201 Created     │ Resource created (POST success)
6└── 204 No Content  │ Success but no body
7 
83XX - REDIRECTION
9├── 301 Permanent   │ Resource moved
10├── 302 Found       │ Temporary redirect
11└── 304 Not Modified │ Cached version OK
12 
134XX - CLIENT ERRORS (Attacker Activity!)
14├── 400 Bad Request │ Malformed request (malicious input?)
15├── 401 Unauthorized│ Auth required (login attempt)
16├── 403 Forbidden   │ Access denied (trying protected resources)
17├── 404 Not Found   │ Resource doesn't exist (enumeration?)
18├── 405 Not Allowed │ Wrong HTTP method
19└── 429 Too Many Req│ Rate limited (brute force detected)
20 
215XX - SERVER ERRORS (Possible Exploitation!)
22├── 500 Internal    │ Server error (crash from exploit?)
23├── 502 Bad Gateway │ Backend issue
24├── 503 Unavailable │ Server overloaded (DoS?)
25└── 504 Gateway TO  │ Backend timeout
26 
27Security Analysis by Status Code:
28─────────────────────────────────────────────────────────────────
29Many 401s → Brute force authentication
30Many 403s → Trying to access restricted resources
31Many 404s → Directory/file enumeration
32Many 400s → Malformed input, possibly SQLi/XSS testing
33500 after 200s → Attacker found something exploitable

Detecting Web Attacks

bash

1606070;"># Detecting Common Web Attacks in Logs
2 
3606070;"># 1. SQL INJECTION ATTEMPTS
4grep -iE 606070;">#a5d6ff;">"union.*select|'.*or.*'|'.*and.*'|sleep\(|benchmark\(" access.log
5grep -iE 606070;">#a5d6ff;">"1=1|1'='1|' or ''='" access.log
6grep -iE 606070;">#a5d6ff;">"information_schema|concat\(|load_file" access.log
7 
8606070;"># Example malicious request:
9606070;"># GET /product?id=1' OR '1'='1 HTTP/1.1
10606070;"># GET /search?q='; SELECT * FROM users--
11 
12606070;"># 2. XSS ATTEMPTS
13grep -iE 606070;">#a5d6ff;">"<script|javascript:|onerror=|onload=" access.log
14grep -iE 606070;">#a5d6ff;">"alert\(|document\.cookie|eval\(" access.log
15 
16606070;"># Example malicious request:
17606070;"># GET /search?q=<script>alert('XSS')</script>
18606070;"># GET /profile?name="><img src=x onerror=alert(1)>
19 
20606070;"># 3. PATH TRAVERSAL
21grep -E 606070;">#a5d6ff;">"\.\./|\.\.%2f|%2e%2e" access.log
22grep -E 606070;">#a5d6ff;">"/etc/passwd|/etc/shadow|win\.ini|boot\.ini" access.log
23 
24606070;"># Example malicious request:
25606070;"># GET /files?file=../../../etc/passwd
26606070;"># GET /download?path=....//....//etc/passwd
27 
28606070;"># 4. COMMAND INJECTION
29grep -iE 606070;">#a5d6ff;">"\||;|\$\(|\x60|cmd=|exec=" access.log
30grep -iE 606070;">#a5d6ff;">"\bcat\b|\bls\b|\bid\b|\bwhoami\b" access.log
31 
32606070;"># Example malicious request:
33606070;"># GET /ping?host=127.0.0.1;cat /etc/passwd
34 
35606070;"># 5. DIRECTORY ENUMERATION
36grep -E 606070;">#a5d6ff;">"\.(bak|old|swp|~|backup|sql|config)" access.log
37grep -iE 606070;">#a5d6ff;">"/admin|/wp-admin|/phpmyadmin|/manager" access.log
38grep 606070;">#a5d6ff;">" 404 " access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head
39 
40606070;"># 6. SCANNER/BOT DETECTION
41grep -iE 606070;">#a5d6ff;">"nikto|sqlmap|nmap|dirbuster|gobuster|burp" access.log
42grep -iE 606070;">#a5d6ff;">"nessus|openvas|acunetix|netsparker" access.log

URL Encoding

Attackers often URL-encode payloads to evade detection. %27 is a quote, %3C is <, %2F is /. Your searches should account for both encoded and decoded versions!

User-Agent Analysis

bash

1606070;"># User-Agent Analysis
2 
3606070;"># List all unique user-agents
4awk -F606070;">#a5d6ff;">'"' '{print $6}' access.log | sort | uniq -c | sort -rn
5 
6606070;"># Find known scanners by User-Agent
7grep -iE 606070;">#a5d6ff;">"nikto|sqlmap|nmap|masscan|gobuster|dirbuster" access.log
8grep -iE 606070;">#a5d6ff;">"curl|wget|python|perl|ruby" access.log
9 
10606070;"># Suspicious User-Agents:
11─────────────────────────────────────────────────────────────────
12sqlmap/1.0-dev           │ SQL injection scanner
13Nikto/2.1.6              │ Web vulnerability scanner
14DirBuster-1.0            │ Directory enumeration
15Mozilla/4.0 (compatible;)│ Old/fake, often bots
16-                        │ Empty user-agent = script
17 
18606070;"># Search attacks ignore
19606070;"># Many scanners set specific user-agents
20grep -v 606070;">#a5d6ff;">"Mozilla/5.0" access.log | grep -v "Googlebot" | head -50
21 
22606070;"># Find requests with no user-agent
23awk -F606070;">#a5d6ff;">'"' '$6 == "-" || $6 == "" {print $0}' access.log
24 
25606070;"># Check if attacker is spoofing
26606070;"># Same IP, different user-agents = likely scanner
27awk -F606070;">#a5d6ff;">'"' '{print $1, $6}' access.log | \
28  sort | uniq -c | awk 606070;">#a5d6ff;">'$1 > 1 {print}'

Detecting Reconnaissance

bash

1606070;"># Reconnaissance Detection
2 
3606070;"># 1. DIRECTORY BRUTE FORCE
4606070;"># Many 404s from same IP = enumeration
5awk 606070;">#a5d6ff;">'$9 == 404 {print $1}' access.log | \
6  sort | uniq -c | sort -rn | head -10
7 
8606070;"># 2. WHAT WERE THEY LOOKING FOR?
9606070;"># List 404 URLs by count
10awk 606070;">#a5d6ff;">'$9 == 404 {print $7}' access.log | \
11  sort | uniq -c | sort -rn | head -20
12 
13606070;"># Common targets:
14606070;"># /admin, /wp-admin, /phpmyadmin, /manager
15606070;"># /.git, /.env, /backup, /config
16606070;"># /api/v1, /swagger, /graphql
17 
18606070;"># 3. SPIDER/CRAWLER BEHAVIOR
19606070;"># Too many requests per second = automated
20awk 606070;">#a5d6ff;">'{print $1, $4}' access.log | \
21  awk -F606070;">#a5d6ff;">'[:[]' '{print $1, $3":"$4}' | \
22  sort | uniq -c | sort -rn | head
23 
24606070;"># 4. LOOKING FOR SENSITIVE FILES
25grep -E 606070;">#a5d6ff;">"\.(env|config|ini|sql|bak|log|git)" access.log
26grep -E 606070;">#a5d6ff;">"robots\.txt|sitemap\.xml|crossdomain\.xml" access.log
27 
28606070;"># 5. API ENUMERATION
29grep -E 606070;">#a5d6ff;">"/api/|/v[0-9]/" access.log | \
30  awk 606070;">#a5d6ff;">'{print $7}' | sort | uniq -c | sort -rn
31 
32606070;"># 6. PARAMETER FUZZING
33606070;"># Same URL, many different parameters
34grep 606070;">#a5d6ff;">"login" access.log | \
35  awk -F606070;">#a5d6ff;">'?' '{print $1}' | sort | uniq -c | sort -rn

Error Log Analysis

bash

1606070;"># Error Log Analysis
2 
3606070;"># Apache error log format:
4606070;"># [Mon Jul 15 14:22:01.123456 2024] [error] [pid 1234] [client 192.168.1.50:54321]
5606070;">#   PHP Fatal error: ...
6 
7606070;"># View recent errors
8tail -100 /var/log/apache2/error.log
9tail -100 /var/log/nginx/error.log
10 
11606070;"># PHP errors (often indicate exploitation)
12grep 606070;">#a5d6ff;">"PHP" error.log | tail -50
13 
14606070;"># Critical: SQL errors in logs
15grep -iE 606070;">#a5d6ff;">"mysql|sql|syntax error|query" error.log
16606070;"># Attackers causing SQL errors while testing payloads
17 
18606070;"># File permission errors (access attempt)
19grep -iE 606070;">#a5d6ff;">"permission denied|access denied" error.log
20 
21606070;"># File not found (but with context)
22grep 606070;">#a5d6ff;">"File does not exist" error.log
23 
24606070;"># ModSecurity blocks (if installed)
25grep 606070;">#a5d6ff;">"ModSecurity" error.log
26 
27606070;"># Pattern: Normal error vs Attack indicator
28606070;"># Normal: "File does not exist: /var/www/html/favicon.ico"
29606070;"># Attack: "File does not exist: /var/www/html/../../etc/passwd"
30 
31606070;"># Critical errors timeline
32grep 606070;">#a5d6ff;">"\[crit\]\|\[error\]" error.log | \
33  awk 606070;">#a5d6ff;">'{print $1,$2,$3}' | sort | uniq -c

Correlate Access + Error Logs

When you see errors, check the access log for the same timestamp to see the exact request that caused it. This reveals the attacker's payload!

Log Analysis Tools

bash

1606070;"># Tools for Web Log Analysis
2 
3606070;"># 1. GOACCESS - Real-time visual analyzer
4goaccess access.log -o report.html --log-format=COMBINED
5606070;"># Generates beautiful HTML report
6 
7606070;"># 2. AWK - Flexible log parsing
8606070;"># Requests per IP
9awk 606070;">#a5d6ff;">'{print $1}' access.log | sort | uniq -c | sort -rn | head
10 
11606070;"># Requests per hour
12awk 606070;">#a5d6ff;">'{print $4}' access.log | cut -d: -f1-2 | sort | uniq -c
13 
14606070;"># 3. CUT + SORT + UNIQ - Quick counts
15cut -d606070;">#a5d6ff;">' ' -f1 access.log | sort | uniq -c | sort -rn | head
16 
17606070;"># 4. GREP + AWK combo
18606070;"># Find all POST requests and their targets
19grep 606070;">#a5d6ff;">"POST" access.log | awk '{print $7}' | sort | uniq -c
20 
21606070;"># 5. JQ for JSON logs (Nginx JSON format)
22cat access.json | jq 606070;">#a5d6ff;">'.request_uri' | sort | uniq -c
23 
24606070;"># 6. LOGREP - Regex-based analysis
25606070;"># Great for pattern matching across large log files
26 
27606070;"># 7. CUSTOM SCRIPT
28606070;">#!/bin/bash
29LOG=$1
30echo 606070;">#a5d6ff;">"=== Top 10 IPs ==="
31awk 606070;">#a5d6ff;">'{print $1}' $LOG | sort | uniq -c | sort -rn | head
32echo -e 606070;">#a5d6ff;">"\n=== Top 10 404s ==="
33awk 606070;">#a5d6ff;">'$9==404 {print $7}' $LOG | sort | uniq -c | sort -rn | head
34echo -e 606070;">#a5d6ff;">"\n=== Potential SQLi ==="
35grep -c 606070;">#a5d6ff;">"union\|select\|'--" $LOG
36echo -e 606070;">#a5d6ff;">"\n=== Potential XSS ==="
37grep -c 606070;">#a5d6ff;">"<script\|onerror=\|onload=" $LOG

Investigation Example

bash

1606070;"># Investigation: Web Application Compromise
2 
3606070;"># Scenario: You notice suspicious behavior from a web server
4 
5606070;"># STEP 1: Get the big picture
6wc -l access.log                    606070;"># How many requests?
7awk 606070;">#a5d6ff;">'{print $1}' access.log | sort -u | wc -l  # Unique IPs
8 
9606070;"># STEP 2: Find top talkers
10awk 606070;">#a5d6ff;">'{print $1}' access.log | sort | uniq -c | sort -rn | head -10
11606070;"># Output:
12606070;">#  15234 192.168.1.100  <- Suspicious volume!
13606070;">#    523 10.0.0.50
14606070;">#    412 192.168.1.5
15 
16606070;"># STEP 3: Investigate suspicious IP
17grep 606070;">#a5d6ff;">"192.168.1.100" access.log > suspect.log
18wc -l suspect.log                    606070;"># 15234 requests!
19 
20606070;"># STEP 4: Timeline analysis
21head -5 suspect.log                  606070;"># First request
22tail -5 suspect.log                  606070;"># Last request
23606070;"># Determine attack window
24 
25606070;"># STEP 5: What were they requesting?
26awk 606070;">#a5d6ff;">'{print $7}' suspect.log | sort | uniq -c | sort -rn | head -20
27 
28606070;"># STEP 6: Look for attack payloads
29grep -iE 606070;">#a5d6ff;">"union|select|script|eval|exec" suspect.log
30606070;"># Found SQLi attempts!
31 
32606070;"># STEP 7: Did they succeed?
33awk 606070;">#a5d6ff;">'{print $9}' suspect.log | sort | uniq -c
34606070;"># Output:
35606070;">#  14500 404    <- Enumeration phase
36606070;">#    700 200    <- Found valid pages
37606070;">#     34 500    <- Errors (SQLi working?)
38 
39606070;"># STEP 8: What 200s and 500s hit?
40awk 606070;">#a5d6ff;">'$9 == 200 {print $7}' suspect.log | sort | uniq -c
41awk 606070;">#a5d6ff;">'$9 == 500 {print $7}' suspect.log | sort | uniq -c
42606070;"># Found: /search?q= with SQLi payloads returning 500!
43 
44606070;"># STEP 9: Check if data exfiltrated
45grep 606070;">#a5d6ff;">"192.168.1.100" access.log | grep POST
46606070;"># Any POST requests with large response sizes?
47 
48606070;"># STEP 10: Document findings
49606070;"># IP: 192.168.1.100
50606070;"># Time: 14:00-14:45
51606070;"># Attack: SQLi on /search endpoint
52606070;"># Evidence: 500 errors followed by 200s
53606070;"># Impact: Possible data extraction

Web Log Analysis Methodology

Web Attack Investigation

Identify ScopeHow many requests? Unique IPs?

Find AnomaliesTop IPs, unusual status codes

Isolate SuspectFilter logs to suspicious source

TimelineWhen did attack start and end?

PayloadsSearch for SQLi, XSS, traversal patterns

Success CheckDid any attacks succeed (200/500)?

DocumentRecord findings with evidence

Knowledge Check

Quick Quiz

Question 1 of 3

What does a large number of 404 responses from a single IP typically indicate?

Challenges

Find the SQLi Attack

Challenge

🔥 intermediate

Write a grep command to find all requests in access.log that contain potential SQL injection payloads (UNION, SELECT, OR 1=1, etc.), showing the full request line.

Need a hint? (4 available)

Key Takeaways

Access logs record all requests; error logs show what failed
Status codes tell the story: 404s = enumeration, 500s = exploitation
SQLi payloads contain UNION, SELECT, OR 1=1, and SQL syntax
XSS payloads contain <script>, onerror=, javascript:
User-Agents reveal scanners: sqlmap, nikto, gobuster
Correlate access and error logs for complete picture

Related Lessons

Linux Log Analysis

Network Traffic Analysis

SQL Injection Fundamentals