perl regexes são bons aqui:
$ initial='1_[genus_specie_1]_characters:0.2654682758,(((((((((((((((2_[genus_specie_2]_characters:0.0379334280,54_[genus_specie_2]_characters:0.0605802067)/1/100:0.0121248674,(3_[genus_specie_3]_characters:0.0206432295,4_[genus_specie_4]_characters:0.0141250479)/1/100:0.0647820408)/1/100:0.0235327264,30_[genus_specie_5]_characters'
$ expected='genus_specie_1:0.2654682758,(((((((((((((((genus_specie_2:0.0379334280,genus_specie_2:0.0605802067)/1/100:0.0121248674,(genus_specie_3:0.0206432295,genus_specie_4:0.0141250479)/1/100:0.0647820408)/1/100:0.0235327264,genus_specie_5'
$ result=$( perl -pe 's/\d+_\[(.+?)\]_.*?(?=:|$)/$1/g' <<<"$initial" )
$ [[ $result = $expected ]] && echo yes
yes
Isso usa quantificadores não-vorazes ( .*?
) e um look-ahead ( (?=:|$)
)