By default ssh runs on TCP port 22. If you have a ssh configured to listen on a non standard port, you may need a special option to make rsync connect to that port. I was writing a quick backup script for this worpress site, and ran into this issue. In my case I was trying to rsync to a remote server with ssh listening on 4590. You have to give rsync a special ssh option:

# rsync -a --rsh='ssh -p 4590' /srv/www/wp-uploads/ backupsite.com:/backups/wp/wp-uplodas