sftp and sshfs ask me for my password (and work properly after I enter it), whereas ssh and scp don't ask for a password and just work. When I force the use of public key authentication instead of password authentication, like this
sshfs me@server:dir mountpoint -o PreferredAuthentications=publickey
or this
sshfs me@server:dir mountpoint -o PasswordAuthentication=no -o PubkeyAuthentication=yes
the connection is closed ("read: Connection reset by peer").

This is true for all servers I have access to. What could be the problem?