você tem um duplo --
na frente de aes256
. sua linha exata com apenas um único -
funciona para mim:
$ openssl genrsa -aes256 -out server-sec.key 4096
é isso que a ajuda do comando diz:
$ openssl genrsa -h
usage: genrsa [args] [numbits]
-des encrypt the generated key with DES in cbc mode
-des3 encrypt the generated key with DES in ede cbc mode (168 bit key)
-seed
encrypt PEM output with cbc seed
-aes128, -aes192, -aes256
encrypt PEM output with cbc aes
-camellia128, -camellia192, -camellia256
encrypt PEM output with cbc camellia
-out file output the key to 'file
-passout arg output file pass phrase source
-f4 use F4 (0x10001) for the E value
-3 use 3 for the E value
-engine e use engine e, possibly a hardware device.
-rand file:file:...
load the file (or the files in the directory) into
the random number generator