If you’re only doing ssh config file organization for a single laptop, one flat ~/.ssh/config with a handful of Host blocks works fine. Add a devops team, a dozen staging boxes, three bastion hosts, and a client’s production fleet, and that same file turns into 300 lines of copy-pasted host names nobody wants to touch, let alone review in a pull request.
The failure mode isn’t dramatic. It’s quiet: someone adds a wildcard block above a specific host and silently breaks that host’s key path, or two people edit the same file and one person’s local dev alias ends up on a shared machine. None of that shows up until someone can’t connect and burns twenty minutes figuring out why.
What actually breaks when one ssh config file gets too big?
Three problems show up once a single config file has to serve more than one person or more than a handful of hosts. First, ordering bugs: OpenSSH reads the file top to bottom and applies the first matching value for each directive, so a broad Host * block placed above a specific host silently shadows it. Second, merge conflicts: a flat file with every host in one place is a magnet for conflicting edits the moment more than one person touches it. Third, no separation between shared and personal settings: team defaults like a shared bastion or agent-forwarding policy end up mixed in with someone’s personal IdentityFile path, so nobody can safely commit the file as-is.
There’s a fourth problem that only shows up months later: no ownership trail. When one file holds every host anyone has ever needed, nobody can tell which entries are still in use, which belong to a decommissioned box, and which were a one-off someone added for a client that churned last quarter. Splitting the file by environment, covered next, fixes the first three problems mechanically and makes the fourth one a matter of deleting a file instead of hunting through hundreds of lines for the right block to remove.
How does the Include directive fix ssh config file organization for devops teams?
OpenSSH’s Include directive lets you split one file into many. ~/.ssh/config keeps only the settings that apply everywhere plus a single Include line; everything host-specific lives in separate files that get pulled in. Include supports glob wildcards, so one line can load an entire directory of host definitions at once.
Base layout: keep ~/.ssh/config for global defaults only
Put connection-wide defaults in the base file: things like ServerAliveInterval, AddKeysToAgent, and a default Host * block. Add the include line at the end, after those defaults:
Host *
ServerAliveInterval 60
AddKeysToAgent yes
ForwardAgent no
Include conf.d/*.confThe conf.d pattern: one file per environment
Create ~/.ssh/conf.d/ and give each environment its own file: work.conf for your employer’s hosts, personal.conf for side projects, clients/acme.conf for a specific client’s fleet. Each of those files should only contain Host blocks, no global settings, because anything placed after a Host keyword is scoped to that host rather than treated as a new default.
Host staging-web1 staging-web2 staging-web3
User deploy
IdentityFile ~/.ssh/id_ed25519_work
Host prod-db
User dba
IdentityFile ~/.ssh/id_ed25519_work
ProxyJump bastion-workName every included file with an explicit .conf extension and match on that exact pattern, Include conf.d/*.conf rather than Include conf.d/*. A bare wildcard also picks up editor backup files (work.conf~) or a stray .bak copy sitting in the same directory, and OpenSSH will try to parse those too.
Order matters: general settings before Include, specific hosts before wildcards
Two ordering rules matter here. Inside the base file, general configuration has to come before the Include line, since Include can itself sit inside a Host or Match block for conditional loading. Across every included file, specific host blocks need to come before any Host * block, because OpenSSH stops at the first match per directive, not the most specific one.
Ordering rules that prevent silent config bugs
This first-match-wins behavior is the single most common source of ssh config bugs. If a wildcard block sits above a specific host, every directive the wildcard sets wins for that host too, and the specific block below only fills in whatever the wildcard left unset. The fix is mechanical: finish every file with exactly one Host * block, and never add a second one further down expecting it to override the first.
Host *
User ubuntu
IdentityFile ~/.ssh/id_ed25519_default
Host prod-db
User dba
IdentityFile ~/.ssh/id_ed25519_workThat example still resolves correctly because IdentityFile is set explicitly on both blocks. Swap in a directive only the wildcard sets, like ProxyJump, and prod-db silently inherits a bastion it was never meant to use. Run ssh -G <host> before committing a config change: it prints the fully resolved settings and catches this before it reaches a teammate.
Grouping hosts that share settings
Instead of repeating the same lines across ten hosts, list them on one Host line and let them share a block: Host staging-web1 staging-web2 staging-web3 followed by the shared User and IdentityFile. Wildcard patterns work the same way for anything matching a naming convention, for example Host *.internal.example.com for every internal DNS name. Group by what actually needs to be identical, usually environment or region, not by whatever order the hosts happened to get added.
A naming convention pays for itself here. Prefixing every host alias with its environment, stg-web1, prod-db, dr-replica, means a single glob like Host stg-* can carry staging-wide settings without listing every box by name, and it stays accurate automatically as hosts get added or retired. Pick one convention per team and enforce it in review, since a mix of naming styles is what makes wildcard grouping unreliable in the first place.
Using ProxyJump instead of nested ProxyCommand for bastion hosts
Older ssh configs route through a bastion with a ProxyCommand line built on nc or a nested ssh -W. Since OpenSSH 7.3, ProxyJump (or the -J flag) does the same job in one readable line, and it stacks across multiple hops if a connection needs to jump through more than one host:
Host bastion-work
HostName bastion.internal.example.com
User jump
Host prod-db
ProxyJump bastion-workAnyone reading prod-db‘s block can see exactly which bastion it goes through, without decoding a shell command embedded inside the config file. Multiple hops chain with a comma-separated list, ProxyJump bastion-work,internal-jump, for infrastructure that requires jumping through two layers before reaching the target host.
Keeping config in sync across a team or across your own devices
Version-controlling the shared conf.d files
The conf.d/ files with team-shared host definitions, bastion addresses, shared ProxyJump chains, environment naming, are safe to commit to a private repo and symlink into ~/.ssh/conf.d/ on every machine. What doesn’t belong in that repo is anything personal: your own IdentityFile path, a local dev alias, or a client’s hostname if the repo is shared more broadly than that one engagement. Keep personal entries in a separate, gitignored file, included after the shared ones so they can still scope settings to a specific host without touching the shared definitions.
The credential-sync half of this problem, keeping the actual keys and passwords consistent once your hosts are reachable from a laptop, a work desktop, and a phone, is a separate concern from the config file layout covered here. termique’s guide on managing SSH credentials across multiple devices covers that side directly.
Where per-host credentials and keys should live instead
Never put a private key’s contents or a plaintext password directly in ~/.ssh/config. The file should only reference a path, like IdentityFile ~/.ssh/id_ed25519_work, or an agent-forwarded identity, so it stays safe to read over someone’s shoulder or paste into a support ticket. For the broader hardening picture beyond file layout, including disabling password auth and locking down sshd_config on the server side, see termique’s complete guide to SSH security.
A minimal ssh config audit checklist for devops teams
Run through this before merging any change to a shared ssh config, or on a recurring schedule if the file hasn’t been touched in a while:
- Does ~/.ssh/config end with exactly one Host * block, and nothing else after it?
- Are all specific Host blocks placed above any wildcard block that could match the same hostname?
- Does Include appear after the global settings, not before them?
- Are shared, team-wide host definitions split out of any file containing personal identities or paths?
- Does every bastion hop use ProxyJump instead of a hand-built ProxyCommand?
- Does ssh -G return the settings you expect, for every host that matters?
Where snippets and audit logs fit once your config is organized
A clean, consistently-named ssh config is also what makes the next layer of organization possible. Once every host has a predictable alias instead of a raw IP typed from memory, the commands you run against those hosts become worth saving and reusing instead of re-typing them or scrollback-searching bash history for the exact flag combination you used last time. termique’s command snippets vs. bash history guide covers why a searchable, named snippet library beats scrolling through shell history once you’re running the same handful of commands across a growing list of hosts.
termique is a free, cross-platform SSH manager (macOS, Windows, Linux, iOS) that keeps host groups, tags, and per-command audit logs alongside the same connections this guide is about organizing by hand in a text file. The free plan covers 3 hosts and unlimited PTY sessions; Pro removes the host limit for $5 a month.
