Eu finalmente resolvi isso extraindo a fonte do kernel, editando as opções de configuração, recriando o tarball. Parece assim:
# Extract the Kconfig file to change the kernel options
/bin/tar xfv /usr/src/rpm/SOURCES/linux-${vanilla_kernel}.tar linux-${vanilla_kernel}/arch/x86/Kconfig
# Change the options
/bin/sed -i '/ARCH_HIBERNATION_POSSIBLE/!b;n;c\tdef_bool n' linux-${vanilla_kernel}/arch/x86/Kconfig
/bin/sed -i '/ARCH_SUSPEND_POSSIBLE/!b;n;c\tdef_bool n' linux-${vanilla_kernel}/arch/x86/Kconfig
/bin/sed -i '/Support for hot-pluggable CPUs/!b;n;c\t\tdefault n' linux-${vanilla_kernel}/arch/x86/Kconfig
# Add the new Kconfig to the tarball
# There will now be two of these and the last one wins
/bin/tar rf /usr/src/rpm/SOURCES/linux-${vanilla_kernel}.tar linux-${vanilla_kernel}/arch/x86/Kconfig
# Change the build id to include our suffix
/bin/sed -i 's/\(buildid.*\)/\.mybuild/' /usr/src/rpm/SPECS/kernel.spec
# Build the new kernel RPM
/usr/bin/rpmbuild -bb /usr/src/rpm/SPECS/kernel.spec
Isso resulta em um novo conjunto de RPM e posso confirmar que o CONFIG_HOTPLUG_CPU não está ativado.