Skip to content

SSH Programming with Paramiko | Completely Different

OpenSSH is the ubiquitous method of remote access for secure remote-machine login and file transfers. Many people — systems administrators, test automation engineers, web developers and others have to use and interact with it daily. Scripting SSH access and file transfers with Python can be frustrating — but the Paramiko module solves that in a powerful way.

This is a reprint of an article I wrote for Python Magazine as a Completely Different column that was published in the October 2008 issue. I have republished this in its original form, bugs and all

SSH is everywhere. OS X, Linux, Solaris, and even Windows offer OpenSSH servers for remote access and file transfers. It long ago displaced other methods of remote access like telnet and rlogin. While those other systems may still exist, their widespread usage has faded with the rapid adoption of the OpenSSH suite of tools.

OpenSSH itself is actually a suite of tools based on the ssh2 protocol. The suite provides secure remote login tools (ssh), secure file transfer (scp and sftp), and key management tools.

On most operating systems the client-side tools (ssh, scp, sftp) are already installed for users to leverage. Users can also easily install and configure the server-side utilities on systems they want to remotely access.

Many, many people use OpenSSH daily, and many of them spend a lot of time trying to script its usage. Most of these tools and scripts try to wrap the command line executables (ssh, scp, etc) directly. They use things like Pexpect to provide passwords, and try to rationalize and parse the output of the binaries directly.

Having spent a lot of time scripting around the binaries and trying to manage timeouts, standard out/in/error pipes, authentication, arguments and options all through ‘’subprocess”, ”popen2”, etc., I’m here to tell you wrapping command line binaries is prone to error, difficult to test, and painful to maintain.

When you’re in the business of parsing output from command line utilities, watching for exit codes and juggling timeouts, you’re not on a good path. That’s where something like Paramiko comes in.

I discovered Paramiko some time ago. It builds on PyCrypto to provide a Python interface to the SSH2 protocol. The module provides all of the faculties you could ask for, including: ssh-key authentication, ssh shell access, and sftp.

Since discovering Paramiko, my entire paradigm and usage of SSH has changed. Instead of the frustrating experience of shelling-out and hacking around the various kinks with that, I can programmatically access all of the protocols and tools I need in a clean, Pythonic way.

About Paramiko

Paramiko is a pure-Python module and can be easy_install’ed as other typical python modules can. However, PyCrypto is written largely in C, so you may need a compiler to install both depending on your platform.

Paramiko itself has extensive API documentation and an active mailing list. As an added bonus, there’s a Java port of it as well (don’t get me started on controlling SSH within Java) if you need something to achieve the same thing in Java.

Paramiko also offers an implementation of the SSH and SFTP server protocols. It really is feature-rich and complete. I’ve used it in heavily threaded applications as well as in day-to-day maintenance scripts. There’s even an installation and deployment system, named Fabric, that further builds on Paramiko to provide application deployment utilities via SSH.

Getting started

The primary class of the Paramiko API is ”paramiko.SSHClient”. It provides the basic interface you are going to want to use to instantiate server connections and file transfers.

Here’s a simple example:

?View Code PYTHON
1
2
3
4

import paramiko
ssh = paramiko.SSHClient()
ssh.connect('127.0.0.1', username='jesse', 
    password='lol')

This creates a new SSHClient object, and then calls ”connect()” to connect us to the local SSH server. It can’t get much easier than that!

Host Keys

One of the complicating aspects of SSH authentication is host keys. Whenever you make an ssh connection to a remote machine, that host’s key is stored automatically in a file in your home directory called ”.ssh/known_hosts”. If you’ve ever connected to a new host via SSH and seen a message like this:

The authenticity of host 'localhost (::1)' can't be
established.
RSA key fingerprint is
22:fb:16:3c:24:7f:60:99:4f:f4:57:d6:d1:09:9e:28.
Are you sure you want to continue connecting
(yes/no)?

and typed “yes” — you’ve added an entry to the ”known_hosts” file. These keys are important because accepting them implies a level of trust of the host. If the key ever changes or is compromised in some way, your client will refuse to connect without notifying you.

Paramiko enforces this same rule. You must accept and authorize the use and storage of these keys on a per-host basis. Luckily, rather then having to be prompted for each one, or manage each one individually, you can set a magic policy.

The default behavior with an SSHClient object is to refuse to connect to a host (”paramiko.RejectPolicy”) who does not have a key stored in your local ”known_hosts” file. This can become annoying when working in a lab environment where machines come and go and have the operating system reinstalled constantly.

Setting the host key policy takes one method call to the ssh client object (‘’set_missing_host_key_policy()”), which sets the way you want to manage inbound host keys. If you’re lazy like me, you pass in the ”paramiko.AutoAddPolicy()” which will auto-accept unknown keys.

?View Code PYTHON
1
2
3
4
5
6

import paramiko
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(
    paramiko.AutoAddPolicy())
ssh.connect('127.0.0.1', username='jesse', 
    password='lol')

Of course, don’t do this if you’re working with machines you don’t know or trust! Tools built on Paramiko should make this overly liberal policy a configuration option.

Running Simple Commands

So, now that we’re connected, we should try running a command and getting some output.

SSH uses the same type of input, output, and error handles you should be familiar with from other Unix-like applications. Errors are sent to standard error, output goes to standard out, and if you want to send data back to the application, you write it to standard in.

So, the response data from client commands are going to come back in a tuple – (stdin, stdout, stderr) – which are file-like objects you can read from (or write to, in the case of stdin). For example:

?View Code PYTHON
1
2
3
4
5
6
7
8
9

...
>>> ssh.connect('127.0.0.1', username='jesse', 
...    password='lol')
>>> stdin, stdout, stderr = \
...    ssh.exec_command("uptime")
>>> type(stdin)
<class 'paramiko.ChannelFile'>
>>> stdout.readlines()
['13:35  up 11 days,  3:13, 4 users, load averages: 0.14 0.18 0.16\n']

Under the covers, Paramiko has opened a new ”paramiko.Channel” object which represents the secure tunnel to the remote host. The Channel object acts like a normal python socket object. When we call ”exec_command()”, the Channel to the host is opened, and we are handed back ”paramiko.ChannelFile” “file-like” objects which represents the data sent to and from the remote host.

One of the documented nits with the ChannelFile objects paramiko passes back to you is that you need to constantly ”read()” off of the stderr and stdout handles given back to you. If the remote host sends back enough data to fill the buffer, the host will hang waiting for your program to read more. A way around this is to either call ”readlines()” as we did above, or ”read()”. If you need to internally buffer the data, you can also iterate over the object with ”readline()”.

This is the simplest form of connecting and running a command to get the output back. For many sysadmin tasks, this will be invaluable as you need to parse the output of a returned command to find exactly what you need. With Python’s rich string manipulation, this is an easy task. Let’s run something with a lot of output, that also requires a password:

?View Code PYTHON
1
2
3
4

ssh.connect('127.0.0.1', username='jesse', 
   password='lol')
stdin, stdout, stderr = ssh.exec_command(
   "sudo dmesg")

Uh oh. I just called the sudo command. It is going to require me to provide a password interactively with the remote host. No worries:

?View Code PYTHON
1
2
3
4
5
6
7
8
9
10

ssh.connect('127.0.0.1', username='jesse', 
    password='lol')
stdin, stdout, stderr = ssh.exec_command(
    "sudo dmesg")
stdin.write('lol\n')
stdin.flush()
data = stdout.read.splitlines()
for line in data:
    if line.split(':')[0] == 'AirPort':
        print line

There! I logged in remotely and found all messages for my Airport card. The key thing to note here is that I wrote my password to the stdin “file” so that sudo allowed me in.

If you’re wondering, yes, this provides an easy base to create your own interactive shell. You might want to do something like this to make a little custom admin shell using the Python cmd module to administer machines inside of your lab.

Using Paramiko, this is easy. In Listing 1, I outline a basic way to approach this – we wrap the Paramiko manipulation up in the RunCommand methods, allowing the user to add as many hosts as they want, call connect and then run a command.

Listing 1:

?View Code PYTHON
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

#!/usr/bin/python
 
import paramiko
import cmd
 
class RunCommand(cmd.Cmd):
    """ Simple shell to run a command on the host """
 
    prompt = 'ssh > '
 
    def __init__(self):
        cmd.Cmd.__init__(self)
        self.hosts = []
        self.connections = []
 
    def do_add_host(self, args):
        """add_host <host,user,password>
        Add the host to the host list"""
        if args:
            self.hosts.append(args.split(','))
        else:
            print "usage: host <hostip,user,password>"
 
    def do_connect(self, args):
        """Connect to all hosts in the hosts list"""
        for host in self.hosts:
            client = paramiko.SSHClient()
            client.set_missing_host_key_policy(
                paramiko.AutoAddPolicy())
            client.connect(host[0], 
                username=host[1], 
                password=host[2])
            self.connections.append(client)
 
    def do_run(self, command):
        """run <command>
        Execute this command on all hosts in the list"""
        if command:
            for host, conn in zip(self.hosts, self.connections):
                stdin, stdout, stderr = conn.exec_command(command)
                stdin.close()
                for line in stdout.read().splitlines():
                    print 'host: %s: %s' % (host[0], line)
        else:
            print "usage: run <command>"
 
    def do_close(self, args):
        for conn in self.connections:
            conn.close()
 
if __name__ == '__main__':
    RunCommand().cmdloop()

Example output:

ssh > add_host 127.0.0.1,jesse,lol
ssh > connect
ssh > run uptime
host: 127.0.0.1: 14:49  up 11 days,  4:27, 8 users,
load averages: 0.36 0.25 0.19
ssh > close

This is just designed to be a proof-of concept of a pseudo-interactive shell. There are a few improvements you could make should you use it:

- Better printing for multi-line stdout output.
- Handle standard error
- Add in a quit method
- Thread the command execution/data returned.

Like all shells, the sky is the limit when it comes to data visualization. Tools like pssh, OSH, Fabric, etc., all manage the return data differently, and they all have different ways of aggregating the output from different hosts.

File put and get

File manipulation within Paramiko is handled via the SFTP implementation, and, like the ssh client command execution, it’s easy as pie.

We start by instantiating a new paramiko.SSHClient just as before:

?View Code PYTHON
1
2
3
4
5
6

import paramiko
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(
    paramiko.AutoAddPolicy())
ssh.connect('127.0.0.1', username='jesse', 
    password='lol')

This time, we make a call into ”open_sftp()” after we perform the connect to the host. ”open_sftp()” returns a ”paramiko.SFTPClient” client object that supports all of the normal sftp operations (stat, put, get, etc.). In this example, we perform a “get” operation to download the file ”remotefile.py” from the remote system and write it to to the local file, ”localfile.py”.


ftp = ssh.open_sftp()
ftp.get('remotefile.py', 'localfile.py')
ftp.close()

Writing a file to the remote host (a “put” operation) works the exact same way. We just transpose the local and remote arguments:

?View Code PYTHON
1
2
3

ftp = ssh.open_sftp()
ftp.get('localfile.py', 'remotefile.py')
ftp.close()

The nice thing about the sftp client implementation that Paramiko provides is that it support things like stat, chmod, chown, etc. Obviously these might act differently depending on the remote server because some servers do not implement all of the protocol, but even so they’re incredibly useful.

You could easily write functions like ”glob.glob()” to transverse a remote directory tree looking for a particular filename pattern. You could also search based on permissions, size, etc.

One thing to note, however, and this bit me a few times: sftp as a protocol is slightly more restrictive than something like normal secure copy (scp). SCP allows you to use Unix wild cards in the file name when grabbing a file from the remote machine. SFTP, on the other hand, expects the full explicit path to the file you want to download. An example of this is:

?View Code PYTHON
1

ftp.get('*.py', '.')

In most cases, this would mean “download all files with .py” to the local directory on my machine. SFTP is unhappy with this formulation, though (see Listing 2). I learned this the hard way, after I spent several hours pulling apart the sftp client implementation out of frustration.

Listing 2:

?View Code PYTHON
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

>>> ftp.get("./*.py", '.')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.5/site-packages/paramiko/sftp_client.py", 
    line 567, in get
    fr = self.file(remotepath, 'rb')
  File "/Library/Python/2.5/site-packages/paramiko/sftp_client.py", 
    line 238, in open
    t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
  File "/Library/Python/2.5/site-packages/paramiko/sftp_client.py", 
    line 589, in _request
    return self._read_response(num)
  File "/Library/Python/2.5/site-packages/paramiko/sftp_client.py", 
    line 636, in _read_response
    self._convert_status(msg)
  File "/Library/Python/2.5/site-packages/paramiko/sftp_client.py", 
    line 662, in _convert_status
    raise IOError(errno.ENOENT, text)
IOError: [Errno 2] No such file

In Closing

I hope I’ve shown you enough to really dig into Paramiko. It’s one of the gems from the Python community that helps me on a daily basis. I can do remote administration programmatically, write test plugins that perform remote operations easily, and a lot more, all without needing to install extra daemons on the remote machines.

SSH is everywhere, and sooner or later you’re going to need to write a program that interacts with it. Why not save yourself the trouble now and give Paramiko a look?

Related Links

  • Martin
    One word: great!
  • I forgot the related links section, so I added it
  • Brian Cain
    Ok, I'm sold. I love batteries-included, so when can we have it as a standard python module?
  • I don't think it will ever be standard, in addition to requiring
    pycrypto which is 3rd party it probably makes more sense as a 3rd
    party module
  • This has been a great series of posts. Thanks for republishing them here.

    I suspect I'll be using Paramiko soon to forward a couple of ports automatically for me (SFTP, git, svn). Hmm, I once wrote (and still use) a little uploader script that would be great to convert to using SFTP instead of FTP...
  • John
    I am having problems working with paramiko ssh and keys. Platform is Windows trying to connect to openSSH on AIX server.

    Code is

    import os
    paramiko.util.log_to_file('Z://paramiko.log')
    host = "1.1.1.11"
    port = 22
    user = "myuser"
    privatekeyfile = "Z:/opensshkey"
    mykey = paramiko.RSAKey.from_private_key_file(privatekeyfile)
    public_host_key = paramiko.RSAKey(data=str(mykey))
    myssh = paramiko.SSHClient() ;
    #myssh.get_host_keys().add(host, 'ssh-rsa', public_host_key)
    myssh.load_system_host_keys()

    #myssh.connect(host,port,username=user,key_filename=privatekeyfile)
    myssh.connect(host,port,username=user)
    status , msg = myssh.command('ls -al')
    myssh.close()

    I get error unknown server 1.1.1.11

    Any ideas?

    I am able to connect via paramiko sftp but not SSHClient
  • riskable
    Your example of how to use sudo over Paramiko is great (thank you!) but it causes problems when the user in question either has NOPASSWD set or their sudo timestamp is still active/valid. Through much trial and error (googling yielded nothing) I have come up with a good solution:

    stdin, stdout, stderr = ssh.exec_command("sudo -S %s" % command)
    if stdout.channel.closed is False: # If stdout is still open then sudo is asking us for a password
    stdin.write('%s\n' % password)
    stdin.flush()

    ...and that's it! If you try write the password with stdin.write() when sudo isn't expecting one Paramiko will throw a really ambiguous socket.error exception, "Socket is closed". This is because exec_command() method closes all channels (stdin, stdout, stderr) when execution is complete. So there you have it =)

    FYI: When I first tried to solve this problem I just ran the following command before any subsequent sudo execution:

    stdin, stdout, stderr = ssh.exec_command("sudo -k")

    "sudo -k" kills your sudo timestamp and forces you to re-auth the next time you run sudo. It works fine except in the case where the user has NOPASSWD set in the sudoers file =).

    Also, your example should be changed to use "sudo -S". This forces the password to be read from stdin (i.e. so you don't need a pty). It will work without it on standard Linux/OpenSSH hosts but I've run into trouble on other platforms and with other SSH servers.

    -Riskable
    http://riskable.com
    "I have a license to kill -9"
  • riskable
    I know it has been a while since this was posted but I thought I'd follow up and say that thanks to this example and my subsequent discovery (that I posted above) I was able to put together the SSH Power Tool (SSHPT). It is hosted at Google Code:

    http://code.google.com/p/sshpt/

    In the source code is a real-life usage of sudo over Paramiko (works great in production).

    -Riskable
    http://riskable.com
    "Windows admins click away at the branches of evil.
    Linux admins hack at the root $."
  • Thanks, it looks pretty cool at first glance - it's GPL 3 so I probably won't touch it, but I'll pass it on to other people
  • Rachael
    Minor thing that probably no one else noticed, but when you talk about putting and getting, you say "ftp.get" both times, but if you're putting the command is actually "ftp.put". I'm probably an idiot for not picking up on that, but it is mildly confusing.
  • xav.vijay
    Hi



    I followed ur example for paramiko and my code is like this..

    import paramiko
    import os

    ssh = paramiko.SSHClient()
    ssh.load_system_host_keys()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.connect('server-name',22,'root','password')
    stdin, stdout, stderr = ssh.exec_command("ssh -V")
    stdout.readlines()

    But, I am getting the following error!! ... I am not sure why...Can u please help!!
    Traceback (most recent call last):
    File "C:\Documents and Settings\aannas01\My Documents\Downloads\ssh-ver.py", line 18, in
    in <module>
    ssh.connect('server-name',22,'root','password')
    File "C:\Python26\lib\site-packages\paramiko\client.py", line 309, in connect
    self._auth(username, password, pkey, key_filename, allow_agent, look_for_keys)
    File "C:\Python26\lib\site-packages\paramiko\client.py", line 463, in _auth
    raise saved_exception
    BadAuthenticationType: Bad authentication type (allowed_types=['publickey', 'keyboard-interactive'])

    I have no clue why the error is asking for a different set of parameter. I am on a Windows XP machine, and trying to login to a Linux machine...
  • I'm curious about something mentioned above... you seem to discourage use of pexpect for driving ssh sessions, but you launched into issues with 'subprocess' without specific comments about why pexpect isn't a winner for you. Have you had bad experiences with pexpect, and if so please elaborate...

    Having used pexpect to drive cisco routers for the last 3 years, I have to say it's been nothing but pleasant to me.

    Thoughts?
    \m
  • Personally, I've found any tool which waits for specific output from
    commands to be terribly brittle. Output from remote sources can (and
    does) frequently change - sure, something expect-like does the job if
    you can count on the same output over and over again.

    I know pexpect has gotten better since the time I used it, which
    admittedly was awhile ago and at this point I go out of my way to
    avoid it/expect-like tools unless I simply can't. I'm also much more a
    fan of tackling things with tools like paramiko, which is more
    programmatic/reliable in my mind.
  • Actually something expect-like doesn't care what the output is if you code the same way you do with paramiko. pexpect.expect takes a list of args... that list can be regular expressions or perhaps pexpect.TIMEOUT or pexpect.EOF. Based on that which one in that list matches, you make a decision; this is not an issue of a small decision tree. I'm ok with people using other tools, but lets be fair to the tools available. pexpect is a much more generalized solution than paramiko; paramiko gives you more granular visibility into stderr than pexpect does, but paramiko is limited to ssh.
  • Given the article was about SSH programming, and paramiko is for SSH
    programming - I think it's a much better fundamental solution to SSH
    programming than pexpect. I've had plenty of expect-like scripts, and
    expect-base script break because assumption on output, regexes, etc
    broke. I agree that paramiko isn't a generalized tool, but for the
    problem it tries to solve it is the best tool (in my opinion).

    If we were talking a general "watch something and react" tool, then
    yeah - pexpect makes sense, but we're not.
  • hima
    sudo over paramiko doesn't work. It gives exception that " U require a tty to run sudo".
    My server is having requiretty = true . Due to security problem, this setting cant be changed. I will have to handle it through paramiko.
    Pls Help........
  • You should email the paramiko mailing list for support
blog comments powered by Disqus