Na verdade, é por causa do presente :
When you install a new farm with your own credentials, you are
automatically Site Collection administrator of the Central
administration application.
Because I had installed my farm (with Powershell) with another
‘installation’ account, then that ‘installation’ account was Site
Collection administrator.
After I put my own account to the Site Collection administrator group,
I could remove the SharePoint group without any problems.
I was a little confused because when I want to delete that SharePoint
group without the necessary permissions, you don’t get any warning or
error message.
Resumindo, ser um administrador de conjuntos de sites é o nível mínimo de permissões necessário.