ssh-add tem a opção -c , o que torna as chaves adicionadas sujeitas a confirmação antes do uso. De man ssh-add :
-c Indicates that added identities should be subject to confirmation before being used for authentication. Confirmation is performed by
ssh-askpass(1). Successful confirmation is signaled by a zero exit status fromssh-askpass(1), rather than text entered into the requester.
Deve atender às suas necessidades.