A Practical Guide to SSH (w/ Video Tutorial)

13 min readJan 9, 2025

This post was made as a companion post for the posted video:

Intro

SSH (Secure Shell) is a cryptographic network protocol originally introduced in 1995 as a secure replacement for earlier remote shells, once people realized security vulnerabilities exist. While many people are familiar with using SSH through Visual Studio Code’s Remote SSH extension, SSH itself is independent software that allows you to securely connect to and control a remote machine via a text-based shell. This article explains how SSH works, why it matters, and how to set it up.

I’ve aimed this at technical undergrad students who may have followed some quick guides or done these steps blindly in the past but never fully understood them. I’ll be using RunPod to set up an example remote server, but this guide is largely platform-agnostic.

01 Why do I care about SSH?

SSH allows you to launch a shell on another computer as if you were physically there and logged in. This is INCREDIBLY helpful and is why it’s one of the backbones of the internet — particularly for developers, more so than your average Joe. If you’re ever working on a server — whether it’s managing a database, hosting a website, or training machine learning models — it’s important to have a strong understanding of SSH as a staple in your workflow.

What is a shell?
A shell is a text-based application that lets you interact with an operating system by running programs, installing packages, manipulating files, and more. macOS uses ZShell (zsh) by default, while most Linux distributions default to BASH. Windows ships with PowerShell, although many developers end up installing Git-BASH or Windows Subsystem for Linux (WSL) (since BASH is kind of the standard across many dev workflows). BASH is the classic shell — ugly but well-established — while a bunch of modern shells try to reimagine that text-based experience. XONSH, for example, combines BASH and Python so you can do things like use Python one minute, then chain shell commands the next.

02 How to think of SSH

SSH relies on asymmetric encryption. These days, Ed25519 is a common algorithm, though older keys might have been generated using RSA. In broad terms, you generate a private key and a public key.

You then copy the public key to any server you want to access. When you attempt an SSH connection, the server checks whether your private key matches its copy of your public key. If they match, you’re granted secure access. If you want extra security, you can protect your private key with a passphrase.

It can be helpful to think of an SSH key as a standardized password-management system, except instead of a password, you have a private key that acts like a super-secret pass ID that proves “I AM JOHN,” while your public key is the server’s “JOHN detector.” If the server is set up for John, it keeps that “JOHN detector” in its authorized keys list. When John’s key arrives, the server can verify, “Yes, that’s John.” This beats typing a password each time, especially if you manage multiple servers. However, for extra security, you can add a passphrase to your SSH key in case someone manages to get ahold of your machine.

03 SSH in Action (Using Pre-installed OpenSSH Client)

03.01 What is OpenSSH

Yes, this is the official logo of the tool that serves as a backbone of the modern internet. And it has a fitting website that looks like it’s from the 90s.

Although SSH is a protocol, OpenSSH is a program that implements it. On Linux and macOS, OpenSSH comes pre-installed, so calling ssh, sshd, ssh-keygen or any related command usually invokes the OpenSSH suite. Windows traditionally didn’t come with SSH by default, which is why many people installed PuTTY, though modern versions of Windows do offer an optional OpenSSH client. Whenever you see someone type ssh in their terminal, it’s likely the pre-installed OpenSSH client.

03.02 OpenSSH Configuration Directory `~/.ssh`

By convention, OpenSSH looks in the ~/.ssh directory for its config files. The tilde ~ is your home directory, and the . in front of a file or folder indicates it’s “hidden” from casual view. “Dotfiles” and “dotfolders” are hidden files/folders in the home directory and are the conventional location way to save text configuration files out of the user’s direct line of sight. If you use VSCode, you can find your JSON settings saved this same way.

You can enable viewing hidden files in macOS Finder with Cmd + Shift + .

If you don’t see ~/.ssh in your home directory, you can create it yourself. After creating the directory, you should run chmod 700 ~/.ssh to ensure that only the owner of that folder can access it. OpenSSH will throw errors if it detects that your key directory is too publicly accessible, because that would be a security hazard.

Nothing is inherently “special” about ~/.ssh except that it’s the default location where OpenSSH (and other SSH-based services, including VSCode Remote Connect) looks for keys and configs if a location is not specified. You could store these files elsewhere if you wanted, but there’s usually no need. (More in Section 4 on why it can be useful to store elsewhere)

# 03.02 OpenSSH Configuration Directory `~/.ssh`

cd ~ # Change to Home Directory
ls -a # List all files including hidden files
mkdir ~/.ssh # Create .ssh dir if it doesn't already exist
chmod 700 ~/.ssh # Make sure directory is only accessible to owner
cd ~/.ssh # Change to ssh config directory

03.03 Generating Private & Public Keys w/ `ssh-keygen`

Before you can connect to a remote server with SSH, you need to generate a key pair. This step is so common that the GitHub documentation is practically the go-to reference.

ssh-keygen -t ed25519 -C "your_email@example.com"

People often say “SSH Key” to refer to either the public or private component of the pair. Both generated keys will have the same name, differing only by the file extension. Typically, the public key ends with .pub, while the private key might not have an extension at all, or it could end in .key, .pem, .ppk, or something else (we love a lack of standards 💜). It’s also sometimes called your “identity file,” since it identifies who’s connecting. The name of the key-pair often defaults to id_<encryption_algorithm>

A private key file (ex: ~/.ssh/id_ed25519, no file extension) might look like:

-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAACmFlczI1Ni1jdHIAAAAGYmNyeXB0AAAAGAAAABA+X5a6nj
...
-----END OPENSSH PRIVATE KEY-----

A public key file ~/.ssh/id_ed25519.pub might look like:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBTgcSQGewYZraT/zg3MHSlkWAVwrurfSbWopYz1RJfk JohnDoe@mit.edu

You’ll see the encryption algorithm (like ssh-ed25519), the key itself, and then some -C comment—your email is often the convention (this can be specified on generation).

Note: You can use ssh-keygen to generate keys for uses other than SSH, since it’s really just a handy way to create public-private key pairs.

At the end of this step you should now have:

~/.ssh/<key_name>.pub
~/.ssh/<key_name>

# 03.03 Generating Private & Public Keys w/ `ssh-keygen`

man ssh-keygen # Read manual pages for ssh-keygen

# Use all the defaults
# Encryption Type (-t) = ed25519
# file (-f) = ~/.ssh/id_ed25519 (and ~/.ssh/id_ed25519.pub)
# Comment (-C) = <your_user>@<your_computer_name>
ssh-keygen

# OR Use custom arguments
# Encryption Type (-t) = rsa
# file (-f) = /Users/gat/project/secrets/ssh/key_name
#  (and /Users/gat/project/secrets/ssh/key_name.pub)
# Comment (-C) = JohnDoe@gmail.com
ssh -t rsa -f /Users/gat/project/secrets/ssh/key_name -C "JohnDoe@gmail.com"

# Take a look at your generated keys
# (They should already be generated with proper permissions)
cat ~/.ssh/id_ed25519
cat ~/.ssh/id_ed25519.pub

03.04 Authorizing your public key w/ `authorized_keys` file

Once you have a key, you need to add your public key to the ~/.ssh/authorized_keys file on the server you want to log into. For services like AWS, you’ll often be prompted to upload your public key via a console or dashboard. For RunPod, you might hop into a temporary web-based terminal (like a Jupyter terminal) and manually edit the ~/.ssh/authorized_keys file. A single computer can store multiple authorized public keys, each granting access to a different user or collaborator.

Some folks use the same key everywhere — this is kind of like using the same password for all your accounts, which isn’t the safest approach. If someone compromises that one key, they can break into every server you’ve configured, so think about generating separate keys for separate projects.

# 03.04 Authorizing your public key w/ `authorized_keys` file

# (While on Remote Machine)

# Append your key to authorized_keys file, creating it if it doesn't exist
echo "ssh-ed25519 pUblIc-keY-yOu-cOpied JohnDoe@mit.edu" >> "~/.ssh/authorized_keys"

# Check authorized_keys file
cat "~/.ssh/authorized_keys"

03.05 Check SSH Daemon is Running w/ `sshd`

Behind the scenes, a program known as the SSH Daemon (sshd) runs in the background and listens for inbound SSH requests on the remote host (typically on port 22, the default for SSH). If sshd isn’t running, nobody can connect.

# 03.05 Check SSH Daemon is Running w/ `sshd`

# (While on Remote Machine)

sshd --help # Check SSH Daemon usage

ps aux | grep sshd # Check SSH Daemon is running

# You might get something like:
# root          70  0.0  0.0  15424  2016 ?        Ss   06:25   0:00 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups
# root        1461  0.0  0.0   3848     0 pts/0    S+   18:37   0:00 grep --color=auto sshd

03.06 Connect w/ `ssh` (ie: 💰 Profit 💰)

Cloud providers typically forward port 22 to another port for people outside their network to connect to. You may need to check your cloud console for the IP address and port. If your RunPod server’s IP is 12.34.56.78 and your identity file (Denoted with flag -i) is ~/.ssh/my_runpod_key, you can connect like so:

ssh -i ~/.ssh/my_runpod_key root@12.34.56.78

RunPod example IP Address. After starting instance, click “Connect”. Here the IP address is `38.128.232.247` and the port is `34215`

Typically, if the username is not clear, you are signing on as root or the administrator. If you’re working with a large team, it’s recommended to make separate users for each key otherwise the intern might have permissions to accidentally delete a database or a hacker could steal crucial information.

known_hosts — A mirror of authorized_keys
Whenever you SSH into a host for the first time, you’ll typically see a prompt that says something like “Are you sure you want to continue connecting (yes/no)?”. When you say yes, this host is added to your ~/.ssh/known_hosts file. This helps prevent man-in-the-middle attacks by storing host fingerprints so you can tell if something fishy is happening, like a malicious server masquerading as your real one

# 03.06 Connect w/ `ssh` (ie: 💰 Profit 💰)

ssh # Read usage for OpenSSH's client
man ssh # Read manual pages for OpenSSH's client

ssh -i ~/.ssh/id_ed25519 -p <port> root@<remote_ip> # Connect to remote machine

# If on RunPod you might get a shell startup prompt like so:
#  ______                 ______            _
# (_____ \               (_____ \          | |
#  _____) ) _   _  ____   _____) )___    __| |
# |  __  / | | | ||  _ \ |  ____// _ \  / _  |
# | |  \ \ | |_| || | | || |    | |_| |( (_| |
# |_|   |_||____/ |_| |_||_|     \___/  \____|
# 
# For detailed documentation and guides, please visit:
# https://docs.runpod.io/ and https://blog.runpod.io/
# 
# root@1ab4225e4d93:/workspace#

03.07 Save time w/ `~/.ssh/`config`` file

After you’ve SSH’d into a bunch of servers, you’ll realize that typing ssh username@IP-address -i /path/to/some/key -p 6789 over and over can be a pain, especially as the number of servers you’re connecting to increases. Fortunately, you can create an SSH config file at ~/.ssh/config (no file extension required). This SSH file format is odd and specific to this file (as opposed to JSON or some other standard language.)

You can add the following:

Host arena_server <Whatever_name_you_want>
    HostName 12.34.56.78 <IP_address_or_domain>
    User root <or_other_user>
    IdentityFile ~/.ssh/key_name <Key_anywhere_on_computer>
    Port 22 <Default_SSH_port>

Now whenever you type ssh arena_server , OpenSSH will use the SSH profile you configured and automatically connect you.

*VSCode has an* *official SSH Config File extension* *that provides syntax highlighting and IntelliSense for this file type*

# 03.07 Save time w/ `~/.ssh/config` file

# (After editing `~/.ssh/config`)
ssh host_name

# OR specify the config file directly
ssh -F ~/.ssh/config host_name

03.08 (RunPod Only) Save time by Authorizing your Key onto New Instances

On RunPod, you can save time by creating a default authorized_keys file which is copied onto every server at the time of launch. This can be found in your settings, but if you’re working within a group, only those with admin privileges can do this.

Sign into console > Left-side panel under “Account / Settings” > SSH Public Keys

04 Manage keys by saving them into your project (My Method)

I like to take a slightly more structured approach for security and reproducibility. First, I create a secrets directory in my project and ensure it’s ignored by version control (by adding it to .gitignore), otherwise anyone with access to the repo will have access to my keys. Then, I generate keys specifically for that project and store them there. I also create a project-specific SSH config.ssh file (the .ssh extension doesn’t change the functionality, but clarifies purpose and VSCode associates it with the proper file type) so that collaborators can pull the repository, drop in their own keys, and tweak the config as needed if the IP address or port changes. (This is especially relevant for services like RunPod where the IP might not always be the same.)

In practice, that might look like:

Creating a PROJECT_ROOT/secrets folder.
Generating an SSH key pair into that folder, e.g. myproject.key and myproject.pub.
Writing a small config file, config.ssh, with all the details.
Importing this config into ~/.ssh/config with Include PROJECT_ROOT/secrets/config.ssh

It’s just my personal method, but I find it keeps everything neat, especially if you’re juggling multiple servers.

05 Noteworthy SSH Tools & Companions

SSH is powerful. Since it gives you control over another machine, it’s often used as a gateway for other types of remote connections.

VSCode Remote SSH. This will be covered in another article, but basically, VSCode can install a lightweight server on the remote machine after SSH-ing into it, letting you code remotely as if it’s local.
SFTP. An SSH-based file transfer protocol for sending files back and forth securely.
Rsync. A specialized tool that synchronizes files between two directories (remote or local). Very handy for backups or deployments.
Remote Desktop Protocol. While not always using SSH directly, it’s another way to connect to a remote machine, but typically over a GUI. Sometimes people will tunnel RDP through SSH for extra security.

That’s about it for now. This should give you a handle on what SSH does, why it’s important, and how to spin up your own keys and config files

06 Conclusion

SSH is a powerful tool that underpins much of modern computing. By understanding how to generate and manage SSH keys, configure the ~/.ssh/config file, and take advantage of companion tools like VSCode Remote Connect and SFTP, you’ll be able to securely and efficiently manage remote servers. Whether you’re hosting a website, training models on a GPU instance, or just tinkering with a home lab, a solid grasp of SSH is invaluable for any technical student or professional.

The following are all the above commands (not meant to be run as a script, I probably need to revisit this article and make this one cohesive example)

# ----------------------------------------
# 03.02 OpenSSH Configuration Directory `~/.ssh`

cd ~ # Change to Home Directory
ls -a # List all files including hidden files
mkdir ~/.ssh # Create .ssh dir if it doesn't already exist
chmod 700 ~/.ssh # Make sure directory is only accessible to owner
cd ~/.ssh # Change to ssh config directory

# ----------------------------------------
# 03.03 Generating Private & Public Keys w/ `ssh-keygen`

man ssh-keygen # Read manual pages for ssh-keygen

# Use all the defaults
# Encryption Type (-t) = ed25519
# file (-f) = ~/.ssh/id_ed25519 (and ~/.ssh/id_ed25519.pub)
# Comment (-C) = <your_user>@<your_computer_name>
ssh-keygen

# OR Use custom arguments
# Encryption Type (-t) = rsa
# file (-f) = /Users/gat/project/secrets/ssh/key_name
#  (and /Users/gat/project/secrets/ssh/key_name.pub)
# Comment (-C) = JohnDoe@gmail.com
ssh -t rsa -f /Users/gat/project/secrets/ssh/key_name -C JohnDoe@gmail.com

# Take a look at your generated keys
# (They should already be generated with proper permissions)
cat ~/.ssh/id_ed25519
cat ~/.ssh/id_ed25519.pub

# ----------------------------------------
# 03.04 Authorizing your public key w/ `authorized_keys` file

# (While on Remote Machine)

# Append your key to authorized_keys file, creating it if it doesn't exist
echo "ssh-ed25519 pUblIc-keY-yOu-cOpied JohnDoe@mit.edu" >> "~/.ssh/authorized_keys"

# Check authorized_keys file
cat ~/.ssh/authorized_keys

# ----------------------------------------
# 03.04 Authorizing your public key w/ `authorized_keys` file

# (While on Remote Machine)

sshd --help # Check SSH Daemon usage

ps aux | grep sshd # Check SSH Daemon is running

# You might get something like:
# root          70  0.0  0.0  15424  2016 ?        Ss   06:25   0:00 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups
# root        1461  0.0  0.0   3848     0 pts/0    S+   18:37   0:00 grep --color=auto sshd

# ----------------------------------------
# 03.05 Check SSH Daemon is Running w/ `sshd`

# (While on Remote Machine)

sshd --help # Check SSH Daemon usage

ps aux | grep sshd # Check SSH Daemon is running

# You might get something like:
# root          70  0.0  0.0  15424  2016 ?        Ss   06:25   0:00 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups
# root        1461  0.0  0.0   3848     0 pts/0    S+   18:37   0:00 grep --color=auto sshd

# ----------------------------------------
# 03.06 Connect w/ `ssh` (ie: 💰 Profit 💰)

ssh # Read usage for OpenSSH's client
man ssh # Read manual pages for OpenSSH's client

ssh -i ~/.ssh/id_ed25519 -p <port> root@<remote_ip> # Connect to remote machine

# If on RunPod you might get a shell startup prompt like so:
#  ______                 ______            _
# (_____ \               (_____ \          | |
#  _____) ) _   _  ____   _____) )___    __| |
# |  __  / | | | ||  _ \ |  ____// _ \  / _  |
# | |  \ \ | |_| || | | || |    | |_| |( (_| |
# |_|   |_||____/ |_| |_||_|     \___/  \____|
# 
# For detailed documentation and guides, please visit:
# https://docs.runpod.io/ and https://blog.runpod.io/
# 
# root@1ab4225e4d93:/workspace#

# ----------------------------------------
# 03.07 Save time w/ `~/.ssh/config` file

# (After editing `~/.ssh/config`)
ssh host_name

# OR specify the config file directly
ssh -F ~/.ssh/config host_name