SSH: Keys, Config & Real Failures

SSH is simple once you stop treating it as a magic login box. A client reaches a server, verifies the server’s identity, then proves the user’s identity. Most failures tell you exactly which of those stages broke.

🎙️ Published & recorded:

01The key model: lock and proof

Your private key stays on your machine. The public key may be copied freely to servers. During login, the server asks you to prove possession of the private key; the private key is not uploaded. On the server, one line in ~/.ssh/authorized_keys grants one public key access to that account.

# Make a modern key pair; use a passphrase.
ssh-keygen -t ed25519 -a 100 -C "laptop-2026"

~/.ssh/id_ed25519       PRIVATE — never send this
~/.ssh/id_ed25519.pub   PUBLIC — install this on servers

# Show the public key safely:
cat ~/.ssh/id_ed25519.pub
My position
Do not email private keys, reuse one team-wide key, or remove the passphrase for convenience. Generate one key per person and device. Revocation then means deleting one public line, not coordinating a company-wide secret replacement.

02Connect once, then ask verbose mode

An SSH destination has four facts: user, host, port, and identity. Defaults hide three of them, which is pleasant until the wrong default wins. When a connection fails, run the same command with one -v. Read it by stage: network connection, host-key check, then authentication offers.

ssh [email protected]
ssh -p 2222 [email protected]

# Diagnostic view; -vvv is usually noise until -v is insufficient.
ssh -v -p 2222 [email protected]

# Useful lines:
debug1: Connecting to server.example.com [203.0.113.10] port 2222.
debug1: Server host key: ssh-ed25519 SHA256:...
debug1: Offering public key: /home/alice/.ssh/id_ed25519

Be precise about the remote user. Your laptop username is not evidence that the server account has the same name. Cloud images commonly expect ubuntu, ec2-user, or another provisioned account.

03Fix Permission denied (publickey)

This exact message means the network path and SSH service worked. Authentication did not. Do not debug firewalls now. Confirm the remote username, see which key the client offers, force the intended key, then verify that its matching public key is installed for that exact server account.

[email protected]: Permission denied (publickey).

# 1. Does verbose output say "Offering public key"?
ssh -v [email protected]

# 2. Force the intended identity and ignore surprise keys:
ssh -o IdentitiesOnly=yes -i ~/.ssh/work_ed25519 [email protected]

# 3. Compare fingerprints, not filenames:
ssh-keygen -lf ~/.ssh/work_ed25519.pub
# On server console:
ssh-keygen -lf /home/alice/.ssh/authorized_keys
Repair order
Wrong user first, wrong offered key second, missing authorized_keys entry third, permissions fourth, server policy last. Regenerating keys at random destroys evidence and often leaves you with two wrong keys instead of one.

04Permissions: private means private

OpenSSH rejects a private key readable by other users. The server may also ignore authorized_keys if its directory or file is writable by the wrong people. Fix ownership before modes; chmod cannot repair a file owned by root.

WARNING: UNPROTECTED PRIVATE KEY FILE!
Permissions 0644 for 'id_ed25519' are too open.

# Client:
chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_ed25519
chmod 644 ~/.ssh/id_ed25519.pub

# Server, run for the target account:
chown -R alice:alice /home/alice/.ssh
chmod 700 /home/alice/.ssh
chmod 600 /home/alice/.ssh/authorized_keys

On Windows, use the file’s Security settings or icacls to remove inherited access for unrelated users. Copying Unix mode numbers into PowerShell is not a cross-platform fix; the rule is that only the owner should read the private key.

05Host identification changed

The host key identifies the server. A changed key may be an expected rebuild, a recycled IP, or an attacker intercepting the connection. The warning cannot distinguish them, so your job is verification. Never reflexively delete the warning and reconnect.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
Offending ED25519 key in /home/alice/.ssh/known_hosts:12
Host key verification failed.

# 1. Verify the new fingerprint through a trusted channel:
#    cloud console, provider metadata, or an administrator.
# 2. Remove only that host's stale entry:
ssh-keygen -R server.example.com
ssh-keygen -R "[server.example.com]:2222"
# 3. Reconnect and compare the displayed fingerprint.
Security boundary
StrictHostKeyChecking=no is not a fix. It trades an inconvenient warning for silent impersonation. Verification takes a minute; skipping it defeats the part of SSH that tells you which machine received your credentials.

06Connection refused is not a timeout

These failures happen before authentication. “Refused” is a fast answer from the destination: the host is reachable, but nothing accepts that port, or a firewall actively rejects it. A timeout means replies are being dropped or the route is broken. Treating both as “SSH is down” wastes the distinction.

ssh: connect to host server.example.com port 22: Connection refused
→ Check port; check sshd is listening; check service state.
ss -ltn | grep ':22'
systemctl status sshd   # some systems name it ssh

ssh: connect to host server.example.com port 22: Connection timed out
→ Check DNS/IP, VPN, routing, security group, firewall allowlist.
ssh -v -o ConnectTimeout=8 [email protected]

Test the actual hostname and port from the actual client network. A successful test from the server itself proves almost nothing about an inbound firewall. Once the TCP connection succeeds, then return to keys and usernames.

07Put stable facts in SSH config

The command line is for experiments. Stable hosts belong in ~/.ssh/config. Give machines memorable aliases, pin the user, port, and identity, and use ssh -G alias to inspect the fully resolved configuration when inheritance surprises you.

Host reports
    HostName reports.internal.example.com
    User alice
    Port 2222
    IdentityFile ~/.ssh/work_ed25519
    IdentitiesOnly yes

Host *.internal.example.com
    ServerAliveInterval 30
    ServerAliveCountMax 3

# Now:
ssh reports
scp results.csv reports:/srv/import/
ssh -G reports | less
Ordering trap
For each parameter, the first obtained value generally wins. Put specific host blocks before broad wildcards. Do not keep adding duplicate settings at the bottom and expect the last one to override everything.

08Agents and jump hosts

An agent holds unlocked key operations so you do not type the passphrase for every connection. It does not copy the private key. Check what it has loaded when the client offers the wrong identities. For private networks, use a jump host instead of logging into a bastion and storing your key there.

# Inspect and load local agent identities:
ssh-add -l
ssh-add ~/.ssh/work_ed25519
The agent has no identities.  # load one, or specify -i

# Reach an internal host through bastion:
ssh -J [email protected] [email protected]

# Config equivalent:
Host private-app
    HostName 10.0.4.12
    User app
    ProxyJump [email protected]

Avoid agent forwarding by default. A compromised remote host can ask your forwarded agent to sign while your session is open. ProxyJump routes the connection without exposing agent access to the bastion and is the better default.

09Port forwarding without the fog

Local forwarding opens a port on your machine and carries it through SSH to a destination visible from the server. Read -L 8080:db.internal:5432 as: listen locally on 8080, then from the remote side reach db.internal port 5432. Use -N when you need the tunnel, not a shell.

# Local 15432 → database reachable from gateway
ssh -N -L 15432:db.internal:5432 gateway
psql -h 127.0.0.1 -p 15432 appdb

# Local 8080 → remote service bound to its localhost:3000
ssh -N -L 8080:127.0.0.1:3000 app-server
# Browse http://127.0.0.1:8080

# Fail immediately if forwarding cannot be established:
ssh -N -o ExitOnForwardFailure=yes -L 8080:127.0.0.1:3000 app-server
Safe default
Bind forwarded ports to loopback, not 0.0.0.0. The latter can expose an internal database or admin panel to your entire local network. A tunnel is private transport, not automatic access control.

10The SSH failure checklist

Classify the stage before changing anything. One verbose run plus this order usually beats twenty guesses.

□ Did DNS resolve to the expected IP?
□ Is the port correct, reachable, and actually listening?
□ Refused (reachable, closed) or timeout (dropped/unroutable)?
□ Did I verify the server host-key fingerprint?
□ Is the remote username exactly right?
□ Which key does `ssh -v` say it is offering?
□ Does that public key exist in that user's authorized_keys?
□ Are ownership and permissions strict on both sides?
□ What does `ssh -G alias` say after config expansion?
□ Does the agent hold the intended identity?
□ For a tunnel, which machine resolves the destination host?
□ Did I bind only to loopback and use ExitOnForwardFailure?

Keep the stages separate: reach the port, identify the server, authenticate the user, then open a channel or tunnel. SSH’s messages become useful as soon as you stop treating them as one generic login failure.