Defense in Depth

No single control stops all attacks. Layer updates, access control, network policy, application security, and monitoring.

Layer Actions
Updates Automated security patches, tested rollouts
Access SSH keys, sudo limits, no root login
Network Firewall default deny, minimal open ports
Application Non-root services, TLS, input validation
Monitoring Centralized logs, alerts, file integrity
Recovery Backups, immutable snapshots, rebuild runbooks

Patching and Minimal Install 

  sudo apt update && sudo apt upgrade -y
sudo apt autoremove -y

# Minimal package install
sudo apt install --no-install-recommends nginx

# Check for pending security updates
apt list --upgradable 2>/dev/null | grep -i security
sudo dnf updateinfo list security    # RHEL/Fedora

Remove unused services:

  systemctl list-unit-files --type=service --state=enabled
sudo systemctl disable --now avahi-daemon.service 2>/dev/null
sudo systemctl disable --now cups.service 2>/dev/null

SSH and Authentication Hardening 

  # /etc/ssh/sshd_config.d/99-hardening.conf
# PermitRootLogin no
# PasswordAuthentication no
# PubkeyAuthentication yes
# MaxAuthTries 3

sudo sshd -t && sudo systemctl reload ssh
  • Ed25519 keys with passphrases
  • Unique keys per environment
  • AllowUsers or AllowGroups to restrict who can SSH

Brute-Force Protection 

  sudo apt install fail2ban
sudo systemctl enable --now fail2ban

sudo fail2ban-client status
sudo fail2ban-client status sshd

# Custom jail /etc/fail2ban/jail.local
# [sshd]
# enabled = true
# maxretry = 3
# bantime = 3600

Cloud WAF and security groups provide network-layer brute-force reduction before traffic hits the host.

Firewall Baseline 

  sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow from 10.0.0.0/8 to any port 22 proto tcp
sudo ufw allow 443/tcp
sudo ufw enable
sudo ufw status verbose

Review listening ports regularly:

  sudo ss -tulpn
sudo lsof -i -P -n | grep LISTEN

Every open port needs an owner and justification in the runbook.

Least Privilege 

  # Dedicated service user
sudo useradd -r -s /usr/sbin/nologin myapp

# Sudo with command limits
echo 'deploy ALL=(ALL) NOPASSWD: /bin/systemctl restart myapp' \
    | sudo tee /etc/sudoers.d/deploy

# Find world-writable files
sudo find /etc /var/www -perm -002 -type f 2>/dev/null

# Find SUID binaries
sudo find / -perm -4000 -type f 2>/dev/null

Secrets: mode 600, owned by service user, never in world-readable git repos.

Audit and Logging 

  # Failed SSH attempts
sudo journalctl -u ssh --grep "Failed"
sudo grep "Failed password" /var/log/auth.log

# auditd for compliance (PCI, SOC2)
sudo apt install auditd
sudo systemctl enable --now auditd
sudo auditctl -l
sudo ausearch -m USER_LOGIN -ts recent

Ship logs to central storage (Loki, ELK, CloudWatch, Splunk) before attackers can tamper with local files. Forward with Fluent Bit, Filebeat, or rsyslog.

File Integrity Monitoring 

  # AIDE — Advanced Intrusion Detection Environment
sudo apt install aide
sudo aideinit
sudo aide --check

# Tripwire or OSSEC for larger deployments

Baseline /etc, /bin, /sbin, /usr/bin — alert on unauthorized changes.

Kernel and sysctl Hardening 

  # /etc/sysctl.d/99-hardening.conf
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
kernel.randomize_va_space = 2
fs.suid_dumpable = 0

# Apply
sudo sysctl --system

Mandatory Access Control

SELinux (RHEL/Fedora) and AppArmor (Ubuntu/Debian) enforce policies beyond Unix permissions:

  # AppArmor
sudo aa-status
sudo aa-enforce /etc/apparmor.d/usr.sbin.nginx

# SELinux
getenforce
sudo setenforce 1
sudo ausearch -m avc -ts recent

Do not disable SELinux/AppArmor to “fix” issues — adjust policies or booleans properly.

Incident Response

  1. Isolate — remove from load balancer, restrict SG/firewall to admin IPs only
  2. Preserve — snapshot disk, copy logs to secure storage
  3. Investigate — journalctl, auth.log, last, history, cron, running processes
  4. Remediate — patch CVE, rotate all keys/passwords, or rebuild from golden image
  5. Post-mortem — document timeline, root cause, detection gaps

If root was compromised, rebuild — do not trust the existing system.

Best Practices

Practice Reason
Immutable infrastructure Compromised VM replaced, not cleaned
Secrets in vault, not disk Rotation and access audit
CIS benchmark baseline Industry-standard hardening checklist
Regular penetration tests Find gaps before attackers do

Common Mistakes

Mistake Consequence
Security through obscurity (non-standard SSH port only) Scanners find it; no real protection
Disabling SELinux globally Removes MAC protection
Local-only logs Attacker covers tracks after root
Shared admin accounts No accountability in audit trail

Production Scenario

Baseline checklist for every new production server:

  #!/usr/bin/env bash
# post-provision-hardening.sh (run via Ansible)
apt update && apt upgrade -y
apt install -y ufw fail2ban unattended-upgrades auditd

ufw default deny incoming && ufw allow 443/tcp && ufw --force enable
sed -i 's/^#*PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl reload ssh
systemctl enable fail2ban auditd unattended-upgrades

# Deploy monitoring agent, log shipper, AIDE baseline

Servers failing CIS scan score below 85% do not join the production pool until remediated.

Baseline every new server with updates, SSH keys, firewall rules, centralized logging, and monitoring before production traffic — hardening is not a one-time task but continuous verification.