> For the complete documentation index, see [llms.txt](https://muhammed-hatem.gitbook.io/muhammed-hatem/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://muhammed-hatem.gitbook.io/muhammed-hatem/siem-and-soc/elk.md).

# ELK

### <mark style="color:orange;">**What is ELK?**</mark>

**ELK** is an acronym for three powerful open-source tools used for **log management, data analysis, and real-time monitoring**:

1. **Elasticsearch** – A distributed search and analytics engine (stores and indexes data).
2. **Logstash** – A data processing pipeline (collects, parses, and enriches logs).
3. **Kibana** – A visualization dashboard (creates charts, graphs, and alerts).

### <mark style="color:orange;">**How ELK Works ?**</mark>

<figure><img src="/files/sKKGs9G7JW8mfFULvmwl" alt="" width="188"><figcaption></figcaption></figure>

#### <mark style="color:green;">**1. Data Collection**</mark>

* **Sources:** Servers, firewalls, cloud apps, IoT devices.
* **Tools:**
  * **Logstash** (processes logs).
  * **Filebeat** for linux or **winlog-beat** >(lightweight log shipper).

#### <mark style="color:green;">**2. Data Processing (Logstash)**</mark>

* **Parses** logs (e.g., Apache logs → JSON).
* **Filters** data (e.g., extracts IPs, timestamps).
* **Enriches** data (e.g., adds geo-location from IPs).

#### <mark style="color:green;">**3. Storage & Search (Elasticsearch)**</mark>

* Stores logs in a **scalable, distributed database**.
* Enables **fast full-text searches** (like Google for logs).

#### <mark style="color:green;">**4. Visualization (Kibana)**</mark>

* Creates **real-time dashboards** (e.g., attack maps, error trends).
* Sets up **alerts** (e.g., "Notify if 500 errors spike").

## <mark style="color:orange;">**Kibana Query Language (KQL) Guide**</mark>

KQL is the intuitive query syntax used in Kibana for fast, efficient log searching. Here's everything you need to know:

### <mark style="color:blue;">**1. Basic KQL Syntax**</mark>

#### **Field-Based Search**

```kql
field_name:value
```

**Examples:**

```kql
status:200              # Exact match
http.method:"GET"       # Phrase match (quotes for spaces)
response_size > 1000    # Numeric comparison
```

#### **Logical Operators**

| Operator | Description                | Example                          |
| -------- | -------------------------- | -------------------------------- |
| `AND`    | Both conditions must match | `status:500 AND service:payment` |
| `OR`     | Either condition can match | `method:GET OR method:POST`      |
| `NOT`    | Exclude matches            | `NOT status:200`                 |

#### **Wildcards**

```kql
host.name:web-*         # Matches web-1, web-2, etc.
message:*timeout*       # Contains "timeout"
```

### <mark style="color:blue;">**2. Advanced KQL Features**</mark>

#### **Nested Fields**

```kql
user.name:"john"        # For fields like user.name
```

#### **Range Queries**

```kql
response_time:[100 TO 500]  # Between 100-500
bytes_received:>100000      # Greater than
@timestamp >= "now-1d/d"    # Time ranges
```

#### **Regular Expressions**

```kql
host.ip:/192\.168\..*/      # Regex pattern
```

#### **Escaping Special Characters**

```kql
message:"error\\: failed"   # Escape colon with \\
```

### <mark style="color:blue;">**3. Practical KQL Examples**</mark>

#### **Security Investigations**

```log
# Failed SSH logins
event.action:"login_failed" AND source.ip:192.168.* AND @timestamp >= "now-1h"

# Brute force detection
event.action:"login_failed" | stats count by source.ip | where count > 5
```

#### **Application Debugging**

```log
# 5xx errors in API
status:[500 TO 599] AND path:/api/* AND NOT user_agent:"ELB-HealthChecker"

# Slow responses
response_time_ms:>2000 AND service:checkout
```

#### **System Monitoring**

```log
# High memory usage
metricset.name:memory AND system.memory.used.pct:>0.8

# Disk space alerts
system.filesystem.used.pct:>0.9 AND system.filesystem.mount_point:"/var"
```

<br>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://muhammed-hatem.gitbook.io/muhammed-hatem/siem-and-soc/elk.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.