De link
When the IIS6 SMTP Server module looks for a certificate to use for TLS encryption, it seems checks the 'Local_Machine\my' store. I'm not sure where the OpenSSL certificate was placed by the system, but if it's not in the Personal certificates section, then the SMTP server won't be able to find it, and will therefore provide the error listed here.
If you have IIS7 installed on this server, the easiest way to go about securing this would be to select the 'Server' node in the IIS7 manager, select 'Certificates', and 'Create a self-signed certificate...' This will place a copy of the certificate in both the 'Local_Machine\my' store, as well as the Trusted Roots store.
I ran through a quick test of this: clearing out the Personal store provided the error mentioned above, but reloading the 'Properties' of the SMTP server after creating the self-signed certificate through IIS showed the certificate present, and allowed it to be secured.
Hope this helps!
Note: To check what's in a given certificate store, load up MMC (mmc.exe in the 'Run' box), under 'File', select 'Add/Remove Snap-in'. Under the snap-in list, select 'Certificates', and then choose Local Machine. Once back at the MMC listing, under certificates, you can check 'Personal\Certificates' to see its content.