DUH! Estava bem no arquivo de configuração ansible (/etc/ansible/ansible.cfg)
# if set, always use this private key file for authentication, same as
# if passing --private-key to ansible or ansible-playbook
private_key_file = /home/<username>/.ssh/id_rsa
Agora tudo parece funcionar
10.0.0.5 | success >> {
"changed": false,
"ping": "pong"
}