How I Manage My SSH Keys Across Multiple Servers and GitHub

Managing SSH keys across multiple servers and platforms can get messy fast — especially when you have a homelab, VPS, GitHub repos, and more. I used to juggle one or two keys across everything… until I hit name collisions, agent confusion, and “wrong key” errors.

Now I generate one key per service or server, store them with meaningful filenames, and use an organized ~/.ssh/config to control how each is used.

Here’s how I do it — and how you can too.

My SSH Key Directory Structure

~/.ssh/
├── id_ed25519_homelab
├── id_ed25519_homelab.pub
├── id_ed25519_hetzner
├── id_ed25519_hetzner.pub
├── id_ed25519_github
├── id_ed25519_github.pub
├── config

My Key Setup Process

1. Generate a Separate Key for Each Server

ssh-keygen -t ed25519 -C "docker-01" -f ~/.ssh/id_ed25519_docker_01
ssh-keygen -t ed25519 -C "pve-1" -f ~/.ssh/id_ed25519_pve_1
ssh-keygen -t ed25519 -C "hetzner" -f ~/.ssh/id_ed25519_hetzner_example
ssh-keygen -t ed25519 -C "github"  -f ~/.ssh/id_ed25519_github_example

Tip: You can skip the passphrase for convenience or add one for security.

2. Copy Your Public Keys to the Remote Hosts

ssh-copy-id -i ~/.ssh/id_ed25519_docker_01.pub [email protected]
ssh-copy-id -i ~/.ssh/id_ed25519_pve_1.pub [email protected]
ssh-copy-id -i ~/.ssh/id_ed25519_hetzner.pub root@your-vps-ip

3. Configure Your `~/.ssh/config`

vim ~/.ssh/config

Example:

# Docker server
Host docker-01
  HostName 10.xx.xx.xx
  User myuser
  IdentityFile ~/.ssh/id_ed25519_docker_01

# Proxmox node
Host pve-1
  HostName 10.xx.xx.xx
  User root
  IdentityFile ~/.ssh/id_ed25519_pve_1

# Hetzner VPS
Host hetzner
  HostName 65.21.xx.xx
  User root
  IdentityFile ~/.ssh/id_ed25519_hetzner

# GitHub
Host github.com
  HostName github.com
  User git
  IdentityFile ~/.ssh/id_ed25519_github
  IdentitiesOnly yes

IdentitiesOnly yes prevents SSH from trying other keys — helps avoid confusion when multiple keys are loaded.

4. Add SSH Key to GitHub

4.1. Copy the Public Key

macOS:

pbcopy < ~/.ssh/id_ed25519_github.pub

Linux:

xclip -sel clip < ~/.ssh/id_ed25519_github.pub
# or 
wl-copy < ~/.ssh/id_ed25519_github.pub

4.2. Add to GitHub

Visit GitHub SSH settings
Click “New SSH key”
Give it a meaningful name
Paste the key
Click “Add SSH key”

Test Your Setup

Linux VMs

ssh docker-01
ssh pve-1
ssh hetzner

GitHub

ssh -T [email protected]

If everything’s set up correctly, GitHub will reply:

Hi kjetil! You've successfully authenticated, but GitHub does not provide shell access.

Optional: Disable Password Prompt for sudo

If you’re tired of typing your password every time you use sudo, you can edit the sudoers file:

sudo visudo

Look for this line:

# Allow members of group sudo to execute any command
%sudo  ALL=(ALL:ALL) ALL

Change it to:

# Allow members of group sudo to execute any command
%sudo   ALL=(ALL:ALL) NOPASSWD: ALL

Just be careful — removing password prompts reduces security, especially on multi-user or production systems.

Final Thoughts

This setup has completely decluttered my SSH workflow — no more guessing which key is being used, or why I can’t connect. Whether you’re running a homelab, cloud servers, or just want GitHub SSH access done right, this pattern is clean, scalable, and easy to maintain.

Found this helpful? I share more real-world fixes and homelab tips here on the blog and on GitHub.