Linux Sysadmin Bible

1. Know the machine

Before changing a server, identify its distribution, kernel, user, filesystems, and available resources. /etc/os-release describes the distribution; uname identifies the kernel. A filesystem can run out of bytes or inodes, so checking only capacity is incomplete.

cat /etc/os-release
uname -r
id
pwd
df -h
df -i
free -h

What to expect

Record the distribution/version, effective user, current path, filesystem usage, inode usage, and memory. Output is machine-specific.

The filesystem may have exhausted inodes, hit a quota, become read-only, or denied access. df -i checks one of these hypotheses; follow the actual error.

Watch for: Unused RAM is not the same as available RAM. Linux uses memory for reclaimable caches.

2. Understand ownership and permissions

Files have an owner, group, and permission bits. For a file, r reads, w writes, and x executes. On a directory, x permits traversal, r lists names, and w permits entry changes, subject to other controls. 640 grants owner read/write, group read, and others nothing.

mkdir auwen-permissions-lab
cd auwen-permissions-lab
printf "private note\n" > note.txt
chmod 640 note.txt
ls -l note.txt
stat note.txt

What to expect

The permission string is -rw-r-----, aside from optional ACL or security-context indicators.

Owner: read/write/traverse. Group: read/traverse. Others: none. Parent-directory permissions and ACLs can still affect effective access.

Watch for: chmod 777 is not a general fix. It grants broad access while hiding the actual ownership or application problem.

3. Processes, services, and logs

A process is a running program. systemd manages units, including services. Enabled describes startup configuration; active describes current state. A service can be enabled but failing. Read logs for the current boot to avoid mixing old failures into a new incident.

ps -eo pid,comm,%cpu,%mem --sort=-%cpu
systemctl --failed
systemctl list-units --type=service --state=running
journalctl -b -p warning --no-pager

What to expect

Identify busy processes and any failed units. Some logs require additional privileges; access denied does not mean no logs exist.

Use the service list: many Debian-family systems use ssh.service, while many RPM-family systems use sshd.service. Run systemctl status NAME and journalctl -u NAME -b.

Watch for: Do not restart a remote-access service casually. Validate changes and keep a working console or second connection before touching access configuration.

4. Separate routes, listeners, and names

An interface address does not prove a route exists. A route does not prove a server process is listening. A listening process does not prove a firewall permits traffic. Inspect each layer separately before editing configuration.

ip -brief address
ip route
ss -lnt
getent hosts localhost

What to expect

Identify local addresses, the default route if present, listening TCP ports, and local name resolution.

127.0.0.1 is loopback on each host. Remote access needs an appropriate bind address plus routing and access rules. Decide who should connect before exposing it.

Watch for: Opening a firewall cannot make a nonexistent listener appear. IPv4 and IPv6 listeners may differ.

5. Prove a restore

A backup becomes useful when you can recover the intended files and verify them. This isolated exercise archives one text file, extracts to a different directory, and compares bytes. A live database needs its own consistent backup method.

mkdir auwen-restore-lab
cd auwen-restore-lab
mkdir source restored
printf "recover me\n" > source/note.txt
tar -czf notes.tar.gz source
tar -tzf notes.tar.gz
tar -xzf notes.tar.gz -C restored
cmp source/note.txt restored/source/note.txt

What to expect

The archive listing includes source/note.txt. cmp exits successfully and prints nothing when both files match.

cmp now reports a difference. A service restore also needs permissions, ownership, configuration, dependencies, data consistency, and a functional test.

Watch for: A VM snapshot is not an independent backup. Test recovery onto a separate target and document recovery time and acceptable data loss.

6. Give people identities and files a shared group

Before this lesson: Complete Linux inventory and permissions (lessons 1–2). Use a disposable Ubuntu 24.04 or RHEL 9 VM with an administrator account, console access and a snapshot. Names auwen-lab and auwen-editors must be unused. Outcome: create one local identity and explain why a group change may not affect an already-running shell.

Linux permissions refer to numeric user and group IDs; names are the human-readable lookup layer. A user has a primary group and can have supplementary groups. Files record an owner UID and group GID. Group membership grants only the access allowed by the file and directory mode/ACL; it does not automatically grant administrative power.

Use getent to ask the configured identity lookup services rather than assuming every identity is in a local text file. A missing result before creation is expected; an existing identity is a reason to choose a fresh lab name. useradd is a low-level tool available on both selected distributions; Ubuntu's adduser is a friendlier alternative with different interaction. Do not run both for the same account.

The shared directory below is owned by root and the editors group. Mode 2770 grants owner/group access and sets the directory setgid bit: new entries inherit the directory's group on ordinary Linux filesystems. It does not force group-write permissions on every new file, so the demonstration uses umask 0007. The primary group of the account is distinct from the supplementary editors group.

A running process keeps its current supplementary group list. id auwen-lab asks about the account; id inside an old session describes that process's credentials. After usermod -aG, start a fresh login or test with a newly created sudo -u process. Merely editing a group database does not retroactively rewrite the credentials of every open shell.

# Verify the VM and that these names are unused:
cat /etc/os-release
getent passwd auwen-lab
getent group auwen-editors
# Stop if either lookup found an existing identity.

# Administrator, disposable VM only:
sudo groupadd auwen-editors
sudo useradd -m -U -s /bin/bash auwen-lab
sudo usermod -aG auwen-editors auwen-lab
id auwen-lab
sudo install -d -o root -g auwen-editors -m 2770 /srv/auwen-group-lab
sudo -u auwen-lab sh -c 'umask 0007; printf "%s\n" "group note" > /srv/auwen-group-lab/note.txt'
stat -c '%U %G %a %n' /srv/auwen-group-lab/note.txt
sudo -u auwen-lab cat /srv/auwen-group-lab/note.txt

Run it

Use sudo from the VM's existing administrator account. The lab account initially has no usable password; this example runs a process as that identity through the administrator's sudo access. Do not change your real account or grant auwen-lab sudo/wheel membership. After studying, revert the dedicated VM snapshot to remove the lab identities and directory together.

What to expect

id auwen-lab includes auwen-editors. The new file should show owner auwen-lab, group auwen-editors and mode 660 in a simple filesystem without overriding default ACLs. The final command prints group note. Numeric IDs and primary-group names are machine-specific; do not memorize them from somebody else's screenshot.

sudo chmod 640 /srv/auwen-group-lab/note.txt
# Another editors member can read, but cannot write the file contents
# through these mode bits. The directory is group-writable and
# traversable, so that member may still remove/rename its entry;
# file write permission is not the same as deletion permission.
#
# usermod -G replaces supplementary membership. -aG appends.
# In an existing shell record id; after membership change compare
# id auwen-lab with id in that old shell and in a fresh login.
# A difference between sessions suggests stale process credentials.
# Restore the file mode for later practice:
sudo chmod 660 /srv/auwen-group-lab/note.txt

Watch for: Do not edit /etc/passwd or /etc/group with a casual search-and-replace. Removing an account does not automatically discover or transfer every file it owned. Reusing a numeric UID can expose old files to a new identity. The setgid bit is group inheritance, not sudo access; default ACLs and network filesystems can change effective behavior.

Lesson references

• Ubuntu Server: user management; checked 14 September 2026 ↗
• RHEL 9: managing users and groups ↗
• GNU install manual supplied by Ubuntu 24.04 ↗
• GNU stat manual supplied by Ubuntu 24.04 ↗

7. Read a sudo rule as a permission contract

Before this lesson: Complete users/groups and keep the disposable VM snapshot. Use the existing administrator account and retain a working administrator console. auwen-lab must not already have broad sudo/wheel or other overlapping privileges. Outcome: check a small sudoers rule, test allowed and denied commands, and remove the rule cleanly.

sudo runs an allowed command as another identity after evaluating policy. It is not simply a synonym for 'administrator mode'. A sudoers rule identifies who may run a command, on which host, as which target identity, and with which arguments. Several matching rules can contribute permissions; a narrow new rule does not cancel an existing broad one.

Our deliberately small demonstration grants auwen-lab only /usr/bin/id with no arguments as root. That prints identity information without modifying the system. The empty quoted argument string in sudoers means no arguments are allowed; omitting that restriction would allow any arguments for that command. This toy rule teaches the policy grammar before you authorize an operational command.

First make and check a local file. visudo -cf parses the selected file without installing it. This proves syntax, not that your overall installed policy is appropriately restrictive. The installed configuration must also include /etc/sudoers.d (standard on the target VM configurations). Use visudo for the actual edit, and never force-save invalid policy.

The installed file is root-owned and mode 0440. A drop-in filename without a dot avoids include-directory filename exclusions. No NOPASSWD is used: set a password for the lab account interactively, then start its login shell and test with that password. Passwords are never placed in a lesson, shell history argument or shared script. Fresh sudo authentication can be requested with sudo -k.

# Local scratch fixture, not installed policy:
printf '%s\n' 'auwen-lab ALL=(root) /usr/bin/id ""' > auwen-sudoers-check
visudo -cf auwen-sudoers-check
command -v id

# VM administrator: verify id really is /usr/bin/id.
sudo passwd auwen-lab
sudo visudo -f /etc/sudoers.d/auwen-lab
# In the editor, save exactly:
# auwen-lab ALL=(root) /usr/bin/id ""

sudo chown root:root /etc/sudoers.d/auwen-lab
sudo chmod 0440 /etc/sudoers.d/auwen-lab
sudo visudo -c
sudo -l -U auwen-lab
sudo -iu auwen-lab
# Now inside the lab user's shell:
sudo -k
sudo -l
sudo /usr/bin/id
sudo /usr/bin/id -u
exit

Run it

The first fixture check needs visudo installed but does not need to edit /etc. Perform all subsequent commands only in the disposable VM. Check the command path before writing the rule. When prompted during sudo inside the lab account, enter that account's password. Keep the original administrator console open throughout.

What to expect

The fixture and whole-policy checks report valid syntax. The lab account's sudo -l lists the exact rule. sudo /usr/bin/id succeeds and reports uid=0. Adding -u is denied because arguments are forbidden, provided the account has no other rule granting it. Returning with exit leaves the lab shell; the administrator session remains available.

# Both forms can be syntactically valid:
# /usr/bin/id ""  -> command with no arguments
# /usr/bin/id     -> command with arbitrary arguments
# Therefore a parser cannot decide whether the rule matches your intent.

# VM administrator, remove this lab rule only:
sudo rm -i /etc/sudoers.d/auwen-lab
sudo visudo -c
sudo -l -U auwen-lab
# It should no longer list the grant, unless another rule provides it.
# Revert the dedicated snapshot when the lab is finished.
# Checkpoint: explain user, host, run-as identity, path and argument
# policy in the original rule without saying "it gives admin".

Watch for: A command that opens an editor, runs a shell, accepts plugin paths or executes arbitrary scripts may grant far more power than its friendly name suggests. Syntax validity is not least privilege. Avoid ALL or wildcard arguments as a shortcut. sudo -k clears a credential timestamp; it does not revoke a policy grant. Do not experiment on the only remote account that can repair access.

Lesson references

• Sudo upstream: visudo checking and editing ↗
• Ubuntu 24.04 sudoers manual: command arguments and includedir ↗

8. Inspect a package transaction before accepting it

Before this lesson: Complete inventory and sudo lessons. Use separate disposable Ubuntu 24.04 and RHEL 9 VMs if you want to run both columns. RHEL needs accessible configured repositories; repository/subscription issues are not repaired by pasting random third-party URLs. Outcome: distinguish installed version, candidate version, metadata refresh, transaction preview and actual installation.

A package manager resolves dependencies and records which files a package owns. A repository supplies available versions and metadata. The installed version can differ from the candidate offered by configured repositories. Refreshing metadata changes what the manager knows; it does not itself upgrade all installed software.

On Ubuntu, apt is convenient interactively; apt-get and apt-cache provide interfaces commonly used in scripts. On RHEL 9, dnf manages RPM transactions. dpkg-query and rpm ask the local installed-package database. Neither a repository's package listing nor a successful download proves that a package is installed and functioning.

Work on the harmless tree directory-listing package. Query its state before any changes, refresh metadata in the VM, inspect the candidate and preview the transaction. APT's -s simulates a transaction. DNF's --assumeno declines its confirmation; it can still refresh/cache metadata while resolving the proposal. Read dependency additions and removals, total size and repository origin before accepting.

Record the exact OS release and installed package version after installation. There is intentionally no universal version number in this lesson: repository snapshots, architectures and distribution backports differ. A vendor can fix a vulnerability without adopting the newest upstream version string. A functional smoke test and relevant vendor advisory are more useful than comparing only the first version number.

# UBUNTU 24.04 VM:
cat /etc/os-release
dpkg-query -W tree
sudo apt-get update
apt-cache policy tree
apt-get -s install tree
# Only after reviewing the proposal:
sudo apt-get install tree
dpkg-query -W tree
tree --version

# RHEL 9 VM (run this column on RHEL, not Ubuntu):
cat /etc/os-release
rpm -q tree
sudo dnf makecache
dnf info tree
sudo dnf --assumeno install tree
# Only after reviewing the proposal:
sudo dnf install tree
rpm -q tree
tree --version

Run it

Choose the block for your VM. An initial 'not installed' result is useful baseline information. Make a small directory with a text file and run tree against that directory to confirm the installed command works. Keep confirmation prompts; do not add -y while learning. To undo the whole practice session predictably, revert its VM snapshot.

What to expect

The package database initially reports a version or absence. Metadata and candidate output identify configured sources. Preview describes a transaction without installing tree; DNF may report cancellation/nonzero status after declining. After an accepted install, the package database records a version and tree --version runs. If tree was already installed, the proposal may be a no-op or update rather than a first install.

# Ubuntu:
dpkg-query -S /usr/bin/tree
apt-get -s remove tree
# RHEL:
rpm -qf /usr/bin/tree
sudo dnf --assumeno remove tree

# update refreshed APT metadata, not installed package contents.
# A separately reviewed upgrade/install transaction is required.
# Automatic dependency removal is based on package-manager records,
# not proof that your operational workflows no longer need a tool.
# Review every removal, preserve configuration/data, and verify the
# application after a change. Reinstalling an old package is not
# necessarily an application-data rollback.
# Checkpoint: write a five-line before/change/verify/recover record
# with actual versions from your disposable VM.

Watch for: Do not run APT and DNF blocks on the same machine hoping one works. Repository trust and signature checking are not optional troubleshooting obstacles. Installing a package can run maintainer scripts or start services. A snapshot helps this disposable exercise; it is not an independent production backup. DNF 5/Fedora and newer Ubuntu sudo defaults may differ from the pinned RHEL 9/Ubuntu 24.04 baseline.

Lesson references

• Ubuntu Server: installing and managing packages ↗
• RHEL 9: managing software with DNF ↗
• Ubuntu apt-get manual: simulation ↗