Você deve conseguir fazer isso com vários < Diretório > blocos. Aqui está um link para a documentação do apache aplicável:
Acho que os principais itens do seu caso são:
What to use When
Choosing between filesystem containers and webspace containers is actually quite easy. When applying directives to objects that reside in the filesystem always use <Directory> or <Files>. When applying directives to objects that do not reside in the filesystem (such as a webpage generated from a database), use <Location>.
It is important to never use <Location> when trying to restrict access to objects in the filesystem. This is because many different webspace locations (URLs) could map to the same filesystem location, allowing your restrictions to be circumvented.
Também link - especificamente:
Apart from <Directory>, each group is processed in the order that they appear in the configuration files. <Directory> (group 1 above) is processed in the order shortest directory component to longest. So for example, <Directory /var/web/dir> will be processed before <Directory /var/web/dir/subdir>. If multiple <Directory> sections apply to the same directory they are processed in the configuration file order. Configurations included via the Include directive will be treated as if they were inside the including file at the location of the Include directive.
Acho que isso funcionará:
<Directory "/srv/www/example.com">
AllowOverride All
Options FollowSymlinks -Indexes
Order deny,allow
Allow from 127.0.0.0/8 1.2.3.4 5.6.7.8
# our IP
Allow from 4.3.2.1
# PayPal IPN
Allow from 216.113.191.33
# Deny from all
AuthType Basic
AuthName "Restricted Files"
AuthUserFile /srv/www/htdocs/.htpasswd
Require valid-user
Satisfy Any
</Directory>
<Directory "/srv/www/example.com/url/to/config.xml">
Order allow,deny
Allow from all
</Directory>