eBPF for Security: Kernel-Level Observability Without Agents

Traditional security tools run in userspace, watching from the outside. eBPF runs in the kernel, seeing everything. No agents, no sidecars, no overhead - just kernel-level visibility.

This guide covers eBPF fundamentals, then dives into three powerful security tools: Cilium, Falco, and Tetragon.

TL;DR

• eBPF = programmable kernel, runs custom code safely in kernel space
• Cilium = eBPF-based networking + network policies + service mesh
• Falco = runtime threat detection via syscall monitoring
• Tetragon = security observability + enforcement from Cilium
• All three run without sidecars or agents in pods

What is eBPF?

eBPF (extended Berkeley Packet Filter) lets you run sandboxed programs in the Linux kernel. Originally for packet filtering, it’s now used for networking, security, tracing, and more.

┌─────────────────────────────────────────────────────────────┐
│                      User Space                              │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────────────┐ │
│  │  App 1  │  │  App 2  │  │  App 3  │  │  eBPF Loader    │ │
│  └─────────┘  └─────────┘  └─────────┘  └────────┬────────┘ │
└──────────────────────────────────────────────────┼──────────┘
                                                   │ load
┌──────────────────────────────────────────────────┼──────────┐
│                      Kernel Space                │          │
│                                                  ▼          │
│  ┌──────────────────────────────────────────────────────┐   │
│  │                    eBPF Verifier                      │   │
│  │  (validates safety: no loops, bounded memory, etc)    │   │
│  └──────────────────────────────────────────────────────┘   │
│                            │                                 │
│     ┌──────────────────────┼──────────────────────┐         │
│     ▼                      ▼                      ▼         │
│  ┌──────┐              ┌──────┐              ┌──────┐       │
│  │kprobe│              │ XDP  │              │ tc   │       │
│  └──────┘              └──────┘              └──────┘       │
│  (syscalls)            (network)             (traffic)      │
└─────────────────────────────────────────────────────────────┘

Why eBPF for Security?

TRADITIONAL AGENTS          eBPF-BASED
==================          ==========
Run in userspace            Run in kernel
Higher overhead             Minimal overhead
Can be bypassed             Can't be bypassed
Per-pod sidecars            Node-level only
Resource hungry             Lightweight

eBPF Hook Points

eBPF can attach to various kernel hook points:

HOOK POINT      USE CASE                    EXAMPLES
==========      ========                    ========
kprobes         Syscall tracing             File access, process exec
tracepoints     Stable kernel events        Scheduler, memory
XDP             Network packet processing   DDoS mitigation
tc              Traffic control             Network policies
LSM             Security decisions          Access control
cgroup          Container resource control  Rate limiting

Cilium: eBPF-Powered Networking

Cilium replaces kube-proxy and provides eBPF-based networking, network policies, and service mesh capabilities.

Install Cilium

# Install Cilium CLI
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --fail --remote-name-all \
  https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin

# Install Cilium (replaces kube-proxy)
cilium install --version 1.15.0

# Verify
cilium status

Network Policies with Cilium

Cilium extends Kubernetes NetworkPolicy with L7 rules:

# cilium-network-policy.yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-server-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: api-server
  
  ingress:
    # Allow from frontend, only to specific paths
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: GET
                path: "/api/v1/public/.*"
              - method: POST
                path: "/api/v1/public/.*"
    
    # Allow from admin, all paths
    - fromEndpoints:
        - matchLabels:
            app: admin-panel
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
  
  egress:
    # Allow to database
    - toEndpoints:
        - matchLabels:
            app: postgres
      toPorts:
        - ports:
            - port: "5432"
              protocol: TCP
    
    # Allow DNS
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
          rules:
            dns:
              - matchPattern: "*.cluster.local"

Cluster-Wide Policies

# Block all egress to external IPs by default
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: default-deny-external
spec:
  endpointSelector: {}
  egressDeny:
    - toEntities:
        - world
---
# Allow specific external services
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: allow-external-apis
spec:
  endpointSelector:
    matchLabels:
      external-access: "true"
  egress:
    - toFQDNs:
        - matchName: api.stripe.com
        - matchName: api.sendgrid.com
        - matchPattern: "*.amazonaws.com"
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP

Hubble: Network Observability

Hubble is Cilium’s observability layer:

# Enable Hubble
cilium hubble enable --ui

# Port-forward Hubble UI
cilium hubble ui

# CLI: observe flows
hubble observe --namespace production

# Filter specific traffic
hubble observe \
  --from-pod production/api-server \
  --to-pod production/postgres \
  --verdict DROPPED

# JSON output for SIEM
hubble observe --output json | jq '.flow.destination'

Falco: Runtime Threat Detection

Falco monitors syscalls to detect suspicious behavior at runtime. It’s like an IDS for your containers.

Install Falco

# Add Helm repo
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

# Install with eBPF driver (recommended)
helm upgrade --install falco falcosecurity/falco \
  --namespace falco --create-namespace \
  --set driver.kind=ebpf \
  --set falcosidekick.enabled=true \
  --set falcosidekick.webui.enabled=true

Falco Rules

Falco uses rules to detect threats:

# custom-rules.yaml
customRules:
  rules-custom.yaml: |-
    # Detect shell spawned in container
    - rule: Shell Spawned in Container
      desc: Detect shell execution in a container
      condition: >
        spawned_process and
        container and
        shell_procs and
        proc.pname != "bash" and
        not shell_spawned_by_allowed_process
      output: >
        Shell spawned in container
        (user=%user.name container=%container.name
        shell=%proc.name parent=%proc.pname
        cmdline=%proc.cmdline image=%container.image.repository)
      priority: WARNING
      tags: [container, shell, mitre_execution]

    # Detect sensitive file access
    - rule: Read Sensitive Files
      desc: Detect reading of sensitive files
      condition: >
        open_read and
        container and
        (fd.name startswith /etc/shadow or
         fd.name startswith /etc/passwd or
         fd.name startswith /root/.ssh or
         fd.name startswith /home/*/.ssh)
      output: >
        Sensitive file read in container
        (user=%user.name file=%fd.name
        container=%container.name
        image=%container.image.repository)
      priority: WARNING
      tags: [filesystem, mitre_credential_access]

    # Detect crypto mining
    - rule: Crypto Mining Detected
      desc: Detect crypto mining activity
      condition: >
        spawned_process and
        container and
        (proc.name in (xmrig, minerd, cpuminer, cgminer) or
         proc.cmdline contains "stratum+tcp" or
         proc.cmdline contains "pool.minergate" or
         proc.cmdline contains "crypto-pool")
      output: >
        Crypto mining detected
        (user=%user.name process=%proc.name
        cmdline=%proc.cmdline container=%container.name)
      priority: CRITICAL
      tags: [cryptomining, mitre_resource_hijacking]

    # Detect reverse shell
    - rule: Reverse Shell
      desc: Detect reverse shell connections
      condition: >
        spawned_process and
        container and
        ((proc.name = "bash" or proc.name = "sh") and
         proc.cmdline contains "/dev/tcp/")
      output: >
        Reverse shell detected
        (user=%user.name cmdline=%proc.cmdline
        container=%container.name)
      priority: CRITICAL
      tags: [network, shell, mitre_execution]

    # Detect kubectl exec
    - rule: Kubectl Exec into Pod
      desc: Detect kubectl exec usage
      condition: >
        spawned_process and
        container and
        proc.pname = "runc" and
        proc.cmdline contains "kubectl exec"
      output: >
        kubectl exec detected
        (user=%user.name container=%container.name
        cmdline=%proc.cmdline)
      priority: NOTICE
      tags: [k8s, mitre_execution]

Falcosidekick: Alert Routing

Route Falco alerts to various destinations:

# falcosidekick-values.yaml
config:
  # Slack alerts
  slack:
    webhookurl: "https://hooks.slack.com/services/xxx"
    outputformat: "all"
    minimumpriority: "warning"

  # Elasticsearch for SIEM
  elasticsearch:
    hostport: "https://elasticsearch.logging:9200"
    index: "falco"
    type: "_doc"
    minimumpriority: "notice"

  # Prometheus metrics
  prometheus:
    extralabels: "env:production,cluster:main"

  # PagerDuty for critical
  pagerduty:
    routingkey: "xxx"
    minimumpriority: "critical"

  # AWS SecurityHub
  aws:
    securityhub:
      accountid: "123456789012"
      region: "eu-west-1"
      minimumpriority: "high"

Tetragon: Security Observability + Enforcement

Tetragon (from Cilium) provides deep observability and the ability to enforce security policies at the kernel level.

Install Tetragon

helm repo add cilium https://helm.cilium.io
helm repo update

helm upgrade --install tetragon cilium/tetragon \
  --namespace kube-system \
  --set tetragon.btf=/sys/kernel/btf/vmlinux

TracingPolicy: Observe and Enforce

# process-execution-policy.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: process-execution
spec:
  kprobes:
    - call: "sys_execve"
      syscall: true
      args:
        - index: 0
          type: "string"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Prefix"
              values:
                - "/bin/"
                - "/usr/bin/"
---
# file-access-policy.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: sensitive-file-access
spec:
  kprobes:
    - call: "fd_install"
      syscall: false
      args:
        - index: 0
          type: "int"
        - index: 1
          type: "file"
      selectors:
        - matchArgs:
            - index: 1
              operator: "Prefix"
              values:
                - "/etc/shadow"
                - "/etc/passwd"
                - "/root/.ssh"
        - matchActions:
            - action: Sigkill  # Kill process accessing sensitive files

Block Actions with Tetragon

# block-crypto-mining.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: block-crypto-mining
spec:
  kprobes:
    - call: "sys_execve"
      syscall: true
      args:
        - index: 0
          type: "string"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Equal"
              values:
                - "/usr/bin/xmrig"
                - "/tmp/xmrig"
                - "/var/tmp/minerd"
          matchActions:
            - action: Sigkill
              argError: -1
---
# block-reverse-shells.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: block-reverse-shell
spec:
  kprobes:
    - call: "sys_connect"
      syscall: true
      args:
        - index: 0
          type: "int"
        - index: 1
          type: "sockaddr"
      selectors:
        - matchArgs:
            - index: 1
              operator: "NotEqual"
              values:
                - "family:AF_INET,addr:10.0.0.0/8"
                - "family:AF_INET,addr:172.16.0.0/12"
                - "family:AF_INET,addr:192.168.0.0/16"
          matchBinaries:
            - operator: "In"
              values:
                - "/bin/bash"
                - "/bin/sh"
          matchActions:
            - action: Sigkill

Tetragon CLI: Real-time Monitoring

# Stream all events
kubectl exec -n kube-system ds/tetragon -c tetragon -- \
  tetra getevents -o compact

# Filter process execution
kubectl exec -n kube-system ds/tetragon -c tetragon -- \
  tetra getevents -o compact --process-exec

# Filter by namespace
kubectl exec -n kube-system ds/tetragon -c tetragon -- \
  tetra getevents -o json | jq 'select(.process.pod.namespace == "production")'

# Export to JSON for analysis
kubectl exec -n kube-system ds/tetragon -c tetragon -- \
  tetra getevents -o json > tetragon-events.json

Combining All Three

For comprehensive security, use all three tools:

┌─────────────────────────────────────────────────────────────┐
│                        Kubernetes                            │
│                                                              │
│  ┌────────────────────────────────────────────────────────┐ │
│  │                       Cilium                            │ │
│  │  • eBPF networking (replace kube-proxy)                 │ │
│  │  • L3/L4/L7 network policies                           │ │
│  │  • Hubble observability                                 │ │
│  └────────────────────────────────────────────────────────┘ │
│                                                              │
│  ┌────────────────────────────────────────────────────────┐ │
│  │                       Tetragon                          │ │
│  │  • Process execution tracking                           │ │
│  │  • File access monitoring                               │ │
│  │  • Enforcement (Sigkill malicious processes)           │ │
│  └────────────────────────────────────────────────────────┘ │
│                                                              │
│  ┌────────────────────────────────────────────────────────┐ │
│  │                        Falco                            │ │
│  │  • Rich rule language for detection                     │ │
│  │  • Alert routing (Slack, SIEM, PagerDuty)              │ │
│  │  • Compliance and audit logging                         │ │
│  └────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

Production Architecture

# Alert flow
Kernel Event
    │
    ├──▶ Tetragon ──▶ Enforce (kill) ──▶ Alert
    │
    ├──▶ Falco ──▶ Detect ──▶ Falcosidekick ──▶ Slack/PagerDuty/SIEM
    │
    └──▶ Cilium/Hubble ──▶ Network flow logs ──▶ Elasticsearch

Troubleshooting

eBPF not loading:

# Check kernel version (need 4.19+, recommend 5.10+)
uname -r

# Check BTF availability
ls -la /sys/kernel/btf/vmlinux

# Check Cilium/Tetragon logs
kubectl logs -n kube-system ds/cilium
kubectl logs -n kube-system ds/tetragon

High CPU from eBPF programs:

# Check eBPF program stats
bpftool prog show

# Check Cilium overhead
cilium metrics list | grep cpu

# Reduce tracing scope in TracingPolicy

Falco missing events:

# Verify driver is loaded
kubectl exec -n falco ds/falco -- falco --version

# Check for dropped events
kubectl logs -n falco ds/falco | grep -i drop

References

• eBPF Docs: https://ebpf.io
• Cilium Docs: https://docs.cilium.io
• Falco Docs: https://falco.org/docs
• Tetragon Docs: https://tetragon.cilium.io
• BPF Performance Tools (book): https://www.brendangregg.com/bpf-performance-tools-book.html

======================================== eBPF + Cilium + Falco + Tetragon

Kernel-level security. Zero sidecars.