Splunk Fundamentals

1Splunk Web Interface Overview:
2 
3MAIN AREAS
4─────────────────────────────────────────────────────────────────
5Search Bar        │ Where you type SPL queries
6Time Picker       │ Select time range for search
7Events Tab        │ Raw events matching your search
8Statistics Tab    │ Aggregated/tabular results
9Visualization Tab │ Charts and graphs of results
10Save As           │ Save searches, create alerts
11 
12KEY APPS
13─────────────────────────────────────────────────────────────────
14Search & Reporting│ Main search interface
15Enterprise Security│ Security-specific dashboards (ES license)
16Dashboards        │ Custom visualizations
17Alerts            │ Automated detections
18 
19DATA ORGANIZATION
20─────────────────────────────────────────────────────────────────
21Index             │ Data storage container (like database)
22Source            │ Where data came from (file path, input)
23Sourcetype        │ Type of data (syslog, json, csv)
24Host              │ Machine that generated the data
25Field             │ Parsed key-value pair (src_ip, user)
26 
27Example:
28index=security sourcetype=windows_security host=DC01
29  │           │                    │
30  └── Index   └── Data type        └── Source machine

1| SPL (Search Processing Language) Fundamentals
2 
3| Basic Search - Find events containing a term
4error
5failed
6| Returns all events containing 606070;">#a5d6ff;">"error" or "failed"
7 
8| Field-based search
9status=404
10user=admin
11src_ip=192.168.1.100
12| Returns events where field equals value
13 
14| Boolean operators
15error AND failed           | Both terms
16error OR warning           | Either term
17error NOT timeout          | Exclude term
18(error OR failed) AND user=admin  | Grouping
19 
20| Wildcards
21user=admin*               | Starts with admin
22src_ip=192.168.1.*        | IP in that range
23*password*                | Contains password
24 
25| Time-based
26earliest=-24h             | Last 24 hours
27earliest=-7d latest=-1d   | 7 days to 1 day ago
28earliest=606070;">#a5d6ff;">"07/15/2024:00:00:00" | Specific time
29 
30| Index and sourcetype
31index=security            | Search security index
32sourcetype=syslog         | Only syslog data
33index=* sourcetype=*      | All data (expensive!)
34 
35| Combined Example
36index=security sourcetype=windows_security EventCode=4625
37| Find all failed logins in Windows security logs

1| Essential SPL Commands
2 
3| STATS - Aggregate data
4index=security EventCode=4625
5| stats count by src_ip
6| Returns: count of failed logins per source IP
7 
8| stats with multiple aggregations
9| stats count, dc(user) as unique_users by src_ip
10| dc = distinct count
11 
12| TABLE - Display specific fields
13index=security EventCode=4624
14| table _time, user, src_ip, LogonType
15| Creates clean table output
16 
17| SORT - Order results
18| stats count by src_ip
19| sort -count
20| Descending order (- prefix)
21 
22| HEAD/TAIL - Limit results
23| head 10                   | First 10 results
24| tail 10                   | Last 10 results
25 
26| WHERE - Filter after stats
27| stats count by src_ip
28| where count > 100
29| Only IPs with > 100 events
30 
31| EVAL - Create/modify fields
32| eval severity=if(count>100, 606070;">#a5d6ff;">"high", "low")
33| eval time_hour=strftime(_time, 606070;">#a5d6ff;">"%H")
34| eval fullname=user.606070;">#a5d6ff;">" - ".src_ip
35 
36| REX - Extract fields with regex
37| rex field=message 606070;">#a5d6ff;">"user=(?<extracted_user>\w+)"
38| Extracts user from message field
39 
40| RENAME - Rename fields
41| rename src_ip as 606070;">#a5d6ff;">"Source IP", count as "Total"
42 
43| DEDUP - Remove duplicates
44| dedup src_ip                  | Unique source IPs
45| dedup src_ip user             | Unique src_ip + user combos

1| Security Use Case Searches
2 
3| === AUTHENTICATION ===
4 
5| Failed logins by source IP (brute force detection)
6index=security EventCode=4625
7| stats count by src_ip
8| where count > 10
9| sort -count
10 
11| Failed then successful login (password cracked!)
12index=security (EventCode=4625 OR EventCode=4624)
13| stats count(eval(EventCode=4625)) as failures,
14        count(eval(EventCode=4624)) as success
15        by src_ip, user
16| where failures > 5 AND success > 0
17 
18| Login from multiple locations
19index=security EventCode=4624
20| stats dc(src_ip) as locations by user
21| where locations > 3
22 
23| === POWERSHELL ANALYSIS ===
24 
25| PowerShell execution events
26index=security sourcetype=*powershell* EventCode=4104
27| table _time, host, ScriptBlockText
28| sort -_time
29 
30| Suspicious PowerShell (encoded commands)
31index=security sourcetype=*powershell*
32| search 606070;">#a5d6ff;">"-EncodedCommand" OR "-enc" OR "FromBase64"
33| table _time, host, CommandLine
34 
35| === LATERAL MOVEMENT ===
36 
37| Network logon (Type 3) from workstations
38index=security EventCode=4624 LogonType=3
39| search NOT (src_ip=606070;">#a5d6ff;">"127.0.0.1" OR src_ip="-")
40| stats count by src_ip, dest_host, user
41| sort -count
42 
43| RDP connections
44index=security EventCode=4624 LogonType=10
45| table _time, src_ip, user, dest_host
46 
47| === SERVICE/PROCESS ===
48 
49| New service installed
50index=security EventCode=7045
51| table _time, host, ServiceName, ImagePath
52| sort -_time
53 
54| Suspicious process creation
55index=security EventCode=4688
56| search CommandLine=606070;">#a5d6ff;">"*powershell*" OR
57        CommandLine=606070;">#a5d6ff;">"*cmd.exe*" OR
58        CommandLine=606070;">#a5d6ff;">"*certutil*"
59| table _time, host, ParentProcessName, NewProcessName, CommandLine

1| Time-Based Analysis
2 
3| TIMECHART - Events over time
4index=security EventCode=4625
5| timechart span=1h count
6| Shows failed logins per hour
7 
8| Timechart by field
9index=security EventCode=4625
10| timechart span=1h count by src_ip
11| Per source IP over time (beaconing detection!)
12 
13| Detect beaconing (regular intervals)
14index=network
15| stats count by src_ip, dest_ip
16| where count > 50
17| transaction src_ip dest_ip maxpause=5m
18| eval avg_interval=duration/eventcount
19| where avg_interval > 55 AND avg_interval < 65
20| Regular 60-second intervals = suspicious!
21 
22| Peak detection
23index=security
24| timechart span=1h count
25| predict count
26| Anomaly detection using predict command
27 
28| Hour of day analysis
29index=security EventCode=4624 user=admin
30| eval hour=strftime(_time, 606070;">#a5d6ff;">"%H")
31| stats count by hour
32| sort hour
33| Admin logins by hour (off-hours = suspicious)
34 
35| Day of week analysis
36index=security EventCode=4625
37| eval day=strftime(_time, 606070;">#a5d6ff;">"%A")
38| stats count by day
39| sort -count

1| Subsearches and Joins
2 
3| SUBSEARCH - Search within search
4index=security EventCode=4624
5  [search index=security EventCode=4625
6   | stats count by src_ip
7   | where count > 10
8   | fields src_ip]
9| Find successful logins from IPs that had 10+ failures
10 
11| How it works:
12| 1. Inner search finds brute force IPs
13| 2. Returns list: src_ip=1.2.3.4 OR src_ip=5.6.7.8
14| 3. Outer search finds successful logins from those IPs
15 
16| JOIN - Combine results from two searches
17index=security EventCode=4624
18| join type=inner src_ip
19    [search index=threat_intel
20     | fields src_ip, threat_type]
21| table _time, src_ip, user, threat_type
22| Add threat intel context to logins
23 
24| LOOKUP - Enrich with external data
25| inputlookup ip_reputation.csv
26| Sample: src_ip, reputation, country
27 
28index=security
29| lookup ip_reputation.csv src_ip OUTPUT reputation, country
30| where reputation=606070;">#a5d6ff;">"malicious"
31| Enrich and filter by reputation
32 
33| APPEND - Add results from another search
34index=security EventCode=4625
35| stats count by src_ip
36| append
37    [search index=firewall action=blocked
38     | stats count by src_ip]
39| stats sum(count) as total by src_ip

1Creating Splunk Alerts:
2 
3BASIC ALERT WORKFLOW
4─────────────────────────────────────────────────────────────────
51. Create a search that finds what you want to alert on
62. Test it thoroughly (verify results)
73. Save As → Alert
84. Configure trigger conditions
95. Configure alert actions
10 
11ALERT CONFIGURATION OPTIONS
12─────────────────────────────────────────────────────────────────
13Trigger Conditions:
14├── Number of Results: > 0 (any match triggers)
15├── Number of Results: > 10 (threshold)
16├── Number of Hosts: > 5
17├── Custom condition (use eval)
18 
19Alert Actions:
20├── Add to Triggered Alerts (view in Splunk)
21├── Send Email
22├── Run a Script
23├── Send to Slack/Teams (via webhook)
24├── Create ticket (ServiceNow, Jira)
25└── Log Event (create new event in Splunk)
26 
27Schedule:
28├── Real-time (instant, resource intensive)
29├── Cron schedule (every 5 min, hourly, etc.)
30└── Rolling time window
31 
32EXAMPLE: Brute Force Alert
33─────────────────────────────────────────────────────────────────
34Search:
35index=security EventCode=4625
36| stats count by src_ip
37| where count > 20
38 
39Settings:
40├── Schedule: Every 5 minutes
41├── Time range: Last 5 minutes
42├── Trigger: Number of Results > 0
43├── Throttle: 1 hour per src_ip
44└── Action: Email security team
45 
46EXAMPLE: New Admin Account Alert
47─────────────────────────────────────────────────────────────────
48Search:
49index=security EventCode=4728 TargetGroupName=606070;">#a5d6ff;">"Domain Admins"
50| table _time, MemberName, SubjectUserName
51 
52Settings:
53├── Schedule: Real-time
54├── Trigger: Number of Results > 0
55└── Action: Create ticket, email, triggered alert

1Splunk Dashboards:
2 
3CREATING A DASHBOARD
4─────────────────────────────────────────────────────────────────
51. Run your search
62. Click 606070;">#a5d6ff;">"Save As" → "Dashboard Panel"
73. Create new dashboard or add to existing
84. Choose visualization type
95. Configure panel settings
10 
11VISUALIZATION TYPES
12─────────────────────────────────────────────────────────────────
13Single Value    │ One big number (total count)
14Gauge/Radial    │ Value against a threshold
15Column/Bar Chart│ Compare categories
16Line/Area Chart │ Trends over time
17Pie/Donut Chart │ Proportions
18Table           │ Detailed data
19Map             │ Geographic data
20 
21SECURITY DASHBOARD EXAMPLES
22─────────────────────────────────────────────────────────────────
23Authentication Dashboard:
24├── Failed logins (last 24h) - Single Value
25├── Failed logins over time - Line Chart
26├── Top failed login sources - Bar Chart
27├── Failed login details - Table
28 
29Threat Overview:
30├── Total alerts today - Single Value
31├── Alerts by severity - Pie Chart
32├── Alert trend - Line Chart
33├── Recent critical alerts - Table
34 
35Network Dashboard:
36├── Blocked connections - Single Value
37├── Top blocked IPs - Bar Chart
38├── Traffic volume - Area Chart
39├── Allowed vs Blocked - Stacked Bar
40 
41DASHBOARD TIPS
42─────────────────────────────────────────────────────────────────
43├── Use consistent time ranges
44├── Include drill-down to details
45├── Limit panels (5-10 max for readability)
46├── Use colors meaningfully (red = bad)
47└── Add input controls (dropdowns, time pickers)

What SPL command is used to count events grouped by a field?

Write an SPL query to find source IPs with more than 20 failed login attempts (EventCode=4625) in the security index, showing the count and unique users targeted.