Linux in a Nutshell – Engineer Edition

SECTION 01

Overview — Linux at a Glance

Free, open-source Unix-like kernel + GNU userland. Powers 97% of the top 1M web servers, all top-500 supercomputers, Android, and most cloud infrastructure.

KERNEL DESIGN

Monolithic — subsystems share kernel space; fast IPC

LKM — loadable kernel modules extend without reboot

Preemptive — kernel itself is preemptible (CONFIG_PREEMPT)

SMP — symmetric multiprocessing, NUMA-aware since 2.6

UNIX PHILOSOPHY IN LINUX

🔧 Everything is a file — devices, sockets, pipes, /proc

🔗 Compose small tools — pipes, redirection, shell scripting

📝 Text streams — config, logs, and IPC via text

🔒 Least privilege — UID/GID + capabilities + namespaces

🧩 Silent on success — exit code 0 = OK, non-zero = error

MAJOR DISTRIBUTIONS

RHEL family — RPM/dnf · enterprise
Debian family — APT/dpkg · widespread
Arch / Gentoo — bleeding edge · DIY

KERNEL VERSIONS

Versioned as major.minor.patch (e.g. 6.8.0). LTS kernels maintained 6 years. Check: uname -r. Stable release cycle ~9–10 weeks.

RING ARCHITECTURE

Ring 0 = kernel space (full HW access). Ring 3 = user space (syscall to cross). Context switch cost ≈ 1–5 µs.

SECTION 02

Kernel Architecture

Layered structure from hardware to userland applications — every user request descends through syscall, VFS, and driver subsystems.

User Space

Applications Shell (bash/zsh) GNU Coreutils systemd (PID 1) glibc ring 3 · restricted HW access · communicates via syscalls

Syscall IF

Trap to ring 0 via SYSCALL / INT 0x80 instruction · ~400 syscalls (open, read, write, mmap, clone, execve…) · args in registers rdi/rsi/rdx/rcx/r8/r9

VFS

Virtual FS inode / dentry / file / super_block objects · translates to ext4/xfs/btrfs/tmpfs/procfs/sysfs

Memory Mgr

Page Tables TLB Slab/SLUB OOM Killer mmap · huge pages · KSM

Scheduler

CFS RT (FIFO/RR) DEADLINE time slice ≈ 0.75 ms (CONFIG_HZ=1000) · nice −20..+19 · cgroups v2 weight

Networking

Socket layer TCP/IP stack Netfilter/iptables eBPF XDP sk_buff (skb) as universal packet descriptor

Drivers / HW

Block drivers NIC drivers GPU drivers USB/PCIe bus LKM (.ko) — loadable without reboot · udev hotplug

SECTION 03

Filesystem — FHS & Key Paths

The Filesystem Hierarchy Standard defines the single root / tree. Everything — devices, processes, network sockets — is exposed as a file path.

/

Root of entire filesystem hierarchy — single mount point

/bin /sbin

Essential user & admin binaries (ls, cp, mount, ip)

/usr

Secondary hierarchy — /usr/bin, /usr/lib, /usr/share

/etc

System-wide config files — fstab, passwd, sudoers, hosts, resolv.conf

/home

User home directories — ~ expands to /home/<user>

/var

Variable data — log, spool, cache, lib — grows at runtime

/tmp

Ephemeral temp files — cleared on reboot; often tmpfs (RAM)

/proc

procfs — kernel + process state as virtual files (/proc/<pid>/)

/sys

sysfs — kernel objects, drivers, device attributes exported to userland

/dev

Device nodes — block (sda, nvme0n1), char (tty, null, urandom)

/boot

Kernel image (vmlinuz), initrd/initramfs, GRUB config

/lib /lib64

Shared libraries (glibc, kernel modules in /lib/modules/)

/opt /srv

Optional 3rd-party packages (/opt) and service data (/srv)

/run

Runtime state (PIDs, sockets, tmpfs) — replaces /var/run since systemd

FILE PERMISSIONS

# rwxr-xr-- = 754

chmod 755 script.sh

chmod u+x,g-w file

chown user:group file

umask 022 # default 644/755

# setuid bit: chmod 4755

# sticky bit: chmod +t /tmp

INODE & LINKS

Every file = inode (metadata: size, uid, gid, perms, timestamps, block pointers). Hard link — same inode, same partition. Symlink — separate inode pointing to path, crosses filesystems. stat file shows inode detail.

FS TYPES

ext4 — journal, 16 TiB max file
xfs — 64-bit, parallel I/O
btrfs — CoW, subvolumes, checksums
tmpfs — in-memory, no persistence

SECTION 04

Process & Memory Model

Linux processes are the fundamental execution unit — each with its own virtual address space, file descriptors, and signal handlers. Threads share the same VAS.

fork/exec

fork() → child PID; CoW pages until write · execve() → load new program; preserves PID · clone() → threads (POSIX pthreads)

States

R Running S Sleeping D Disk Wait T Stopped Z Zombie I Idle

VAS Layout

Low → High: text data bss heap ↑ ··· ↓ stack mmap region vDSO kernel ASLR randomises base addrs

Signals

SIGTERM 15 SIGKILL 9 SIGINT 2 SIGHUP 1 SIGCHLD 17 SIGUSR1 10

cgroups v2

Hierarchical resource accounting & limits: cpu.max memory.max blkio pids.max Foundation of Docker/containerd/systemd slices

Namespaces

pid net mnt uts ipc user cgroup time = container isolation primitives

SECTION 05

System KPIs & Performance Metrics

Key observability metrics — what to monitor, healthy targets, and which tools report them.

METRIC	HEALTHY TARGET	WARNING THRESHOLD	TOOL / FILE
CPU Load Average (1m)	< nCPU	> 2× nCPU	uptime · /proc/loadavg · top
CPU %idle	> 20%	< 5% sustained	mpstat -P ALL 1 · /proc/stat
Memory Used %	< 80%	> 90% + swap active	free -h · /proc/meminfo · vmstat
Swap In/Out rate	0 KB/s	> 100 KB/s	vmstat 1 (si/so cols) · sar -W
Disk I/O await (ms)	< 10 ms SSD / <20 HDD	> 100 ms	iostat -x 1 · /proc/diskstats
Disk %util	< 70%	> 90% sustained	iostat -x (last %util col)
Open file descriptors	< 80% of limit	EMFILE errors	lsof \| wc -l · /proc/sys/fs/file-nr
Network RX/TX errors	0	any non-zero trend	ip -s link · /proc/net/dev · ethtool
Context switches/s	< 100k/s	> 500k/s	vmstat 1 (cs col) · perf stat
OOM kill events	0	any occurrence	dmesg \| grep -i oom · journalctl -k

SECTION 06

Administration Workflows

Boot sequence and systemd service management — the two most critical operational workflows.

LINUX BOOT SEQUENCE

1

BIOS/UEFI POST — hardware initialisation, finds boot device, loads bootloader (GRUB2) from MBR/EFI partition.

2

GRUB2 — loads vmlinuz kernel + initramfs into RAM. Passes kernel cmdline (root=, quiet, ro).

3

Kernel init — decompresses, detects hardware, mounts initramfs as /, loads essential drivers, pivots to real root fs.

4

systemd (PID 1) — reads /etc/systemd/system/ units, resolves dependency graph, activates default.target in parallel. systemd-analyze blame shows slowest units.

SYSTEMD SERVICE MANAGEMENT

1

Inspect state — systemctl status nginx shows active/failed, PID, log tail. journalctl -u nginx -f follows live logs.

2

Start / stop / reload — systemctl start|stop|restart|reload nginx. Use reload (SIGHUP) where supported to avoid downtime.

3

Enable / disable on boot — systemctl enable --now nginx creates symlink in wants/ and starts immediately. mask prevents accidental start.

4

Write a unit file — drop /etc/systemd/system/myapp.service with [Unit] [Service] [Install] sections. systemctl daemon-reload then enable.

SECTION 07

Essential Commands Reference

Core commands grouped by function — the daily toolkit of any Linux engineer.

FILES, SEARCH & PIPES

find Recursive file search by name, type, time, size, perms

grep Pattern search; -r recursive, -i case, -v invert

awk Column extraction & transformation — essential for log parsing

sed Stream editor — in-place substitution, line delete, insert

sort/uniq Sort and deduplicate — combine with awk for frequency tables

xargs Convert stdin to arguments; -P n parallel execution

PROCESS & PERFORMANCE

top / htop Live CPU/MEM per-process; htop adds sorting, tree, kill UI

ps Snapshot of processes — aux all users, -ef full, --forest tree

strace Trace syscalls in real time — diagnose hangs, I/O issues

lsof List open files & sockets per process — debug FD leaks

iostat Block device I/O stats — await, %util, rkB/s, wkB/s per disk

perf Hardware counters, CPU cycles, cache misses, branch mispred

NETWORKING

ip Modern iproute2 — replaces ifconfig/route. addr, link, route, neigh

ss Socket statistics — replaces netstat; faster, more detail

tcpdump Packet capture — combine with -A (ASCII) or Wireshark

curl HTTP testing — -v verbose, -I headers, -d POST body, -k insecure

nft / iptables Packet filtering — nftables is the modern replacement for iptables

STORAGE & USERS

df / du df = per-mount usage; du = per-dir size; ncdu for interactive

lsblk Block device tree — shows NAME, SIZE, TYPE, MOUNTPOINT, UUID

rsync Efficient incremental copy — -z compress, --delete mirror

useradd Add user; usermod -aG add to group; visudo for sudoers

journalctl Query systemd journal — -k kernel, -b current boot, -p err

SECTION 08

Values Cheatsheet

Quick-reference cards — file permissions, bash scripting, /proc key files, systemd unit skeleton, networking one-liners, and signal quick-ref.

FILE PERMISSIONS MATRIX

OctalSymbolicMeaning

777rwxrwxrwxall access

755rwxr-xr-xexecutables

644rw-r--r--config files

600rw-------SSH keys

4755rwsr-xr-xsetuid (su/sudo)

1777rwxrwxrwt/tmp sticky

BASH SCRIPT TEMPLATE

#!/usr/bin/env bash

set -euo pipefail

# -e exit on error, -u undef var = err

# -o pipefail pipe fails on any cmd

readonly SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd)

log() { echo "[$(date +%T)] $*" >&2; }

die() { log "ERROR: $*"; exit 1; }

main() {

[[ $# -lt 1 ]] && die "Usage: $0 <arg>"

log "Starting with arg=$1"

}

main "$@"

KEY /proc FILES

/proc/cpuinfoCPU model, cores, flags

/proc/meminfoMemTotal, MemFree, Cached, Buffers

/proc/net/tcpTCP socket table (hex encoded)

/proc/<pid>/fd/Symlinks to every open FD of a process

/proc/<pid>/mapsVirtual memory map of a process

/proc/sys/Tunable kernel params (sysctl mirror)

SYSTEMD UNIT SKELETON

# /etc/systemd/system/myapp.service

[Unit]

Description=My Application

After=network.target

[Service]

Type=simple

User=myuser

WorkingDirectory=/opt/myapp

ExecStart=/opt/myapp/bin/server

Restart=on-failure

RestartSec=5s

StandardOutput=journal

StandardError=journal

[Install]

WantedBy=multi-user.target

NETWORKING ONE-LINERS

# who is listening on port 443

ss -tulnp | grep :443

# capture 50 packets on eth0

tcpdump -i eth0 -c50 -w /tmp/cap.pcap

# check DNS resolution chain

dig +trace example.com @8.8.8.8

# test TCP connectivity

nc -zv 10.0.0.1 22

# route to destination

ip route get 1.1.1.1

# live bandwidth per interface

sar -n DEV 1 5

SYSCTL PERFORMANCE TUNING

# TCP tuning for high concurrency

sysctl -w net.core.somaxconn=65535

sysctl -w net.ipv4.tcp_max_syn_backlog=65535

sysctl -w net.ipv4.tcp_fin_timeout=15

sysctl -w net.ipv4.tcp_tw_reuse=1

# VM memory pressure

sysctl -w vm.swappiness=10 # reduce swap

sysctl -w vm.dirty_ratio=15

# persist in /etc/sysctl.d/99-tune.conf

sysctl --system

Linux Operating System in a Nutshell