Tetragon observes process creation and termination with default configuration
and generates process_exec and process_exit events:
The process_exec events include useful information about the execution of
binaries and related process information. This includes the binary image that
was executed, command-line arguments, the UID context the process was
executed with, the process parent information, the capabilities that a
process had while executed, the process start time, the Kubernetes Pod,
labels and more.
The process_exit events, as the process_exec event shows how and when a
process started, indicate how and when a process is removed. The information
in the event includes the binary image that was executed, command-line
arguments, the UID context the process was executed with, process parent
information, process start time, the status codes and signals on process
exit. Understanding why a process exited and with what status code helps
understand the specifics of that exit.
Both these events include Linux-level metadata (UID, parents, capabilities,
start time, etc.) but also Kubernetes-level metadata (Kubernetes namespace,
labels, name, etc.). This data make the connection between node-level concepts,
the processes, and Kubernetes or container environments.
These events enable a full lifecycle view into a process that can aid an
incident investigation, for example, we can determine if a suspicious process
is still running in a particular environment. For concrete examples of such
events, see the next use case on process execution.
1 - Process execution
Monitor process lifecycle with process_exec and process_exit
This first use case is monitoring process execution, which can be observed with
the Tetragon process_exec and process_exit JSON events.
These events contain the full lifecycle of processes, from fork/exec to
exit, including metadata such as:
Binary name: Defines the name of an executable file
Parent process: Helps to identify process execution anomalies (e.g., if a nodejs app forks a shell, this is suspicious)
Command-line argument: Defines the program runtime behavior
Current working directory: Helps to identify hidden malware execution from a temporary folder, which is a common pattern used in malwares
Kubernetes metadata: Contains pods, labels, and Kubernetes namespaces, which are critical to identify service owners, particularly in a multitenant environments
exec_id: A unique process identifier that correlates all recorded activity of a process
As a first step, let’s start monitoring the events from the xwing pod:
Then in another terminal, let’s kubectl exec into the xwing pod and execute
some example commands:
kubectl exec -it xwing -- /bin/bash
whoami
If you observe, the output in the first terminal should be:
🚀 process default/xwing /bin/bash
🚀 process default/xwing /usr/bin/whoami
💥 exit default/xwing /usr/bin/whoami 0
Here you can see the binary names along with its arguments, the pod info, and
return codes in a compact one-line view of the events.
For more details use the raw JSON events to get detailed information, you can stop
the Tetragon CLI by Crl-C and parse the tetragon.log file by executing:
If we want to monitor execution of Executable and Linkable Format (ELF) or flat binaries
before they are actually executed. Then the process-exec-elf-begin tracing policy is a good first choice.
Note
The process-exec-elf-begin tracing policy, will not report the
different binary format handlers or scripts being executed, but will report
the final ELF or flat binary, like the shebang handler.
To report those another tracing policy can be used.
Before going forward, verify that all pods are up and running, ensure you
deploy our Demo Application to explore the Security Observability Events:
In addition to the Kubernetes Identity and process metadata,
ProcessKprobe
events contain the binary being executed. In the above case they are:
function_name: where we are hooking into the kernel to read the binary that is being executed.
file_arg: that includes the path being executed, and here it is /bin/busybox that is the real
binary being executed, since on the xwing pod the container is running busybox.
The binary /usr/bin/id -> /bin/busybox points to busybox.
Monitor process capabilities and kernel namespace access
Tetragon also provides the ability to check process capabilities and kernel
namespaces access.
This information would help us determine which process or Kubernetes pod has
started or gained access to privileges or host namespaces that it should not
have. This would help us answer questions like:
Which Kubernetes pods are running with CAP_SYS_ADMIN in my cluster?
Which Kubernetes pods have host network or pid namespace access in my
cluster?
Step 1: Enabling Process Credential and Namespace Monitoring