I was able to play BD ISOs in XBMC using PowerDVD10 as my external player. Here are the instructions if anybody is interested:
I try to give you a more detailled description of what you have to do, to play Blu-ray discs and iso-files using PowerDVD:
1. Create a new folder on drive C:, name it 'For_XBMC', DON'T use whitespaces, use underlines instead!
You should now have a folder 'C:\For_XBMC'.
2. Use Notepad or any other text editor to create a batch file, that will be used to play Blu-ray discs using Power DVD. It should only contain this single line:
"C:\Program Files\CyberLink\PowerDVD10\PowerDVD10.exe" /fullscreen e:
If the drive letter of your Blu-ray disc drive is not 'E:' and/or PowerDVD was installed into another directory, change the line above accordingly. Be sure to enclose the path in double quotes. Save the file into the previously created folder, name it 'PlayBD.bat'.
You should now have a file 'C:\For_XBMC\PlayBD.bat'.
3. Insert a Blue-ray or DVD into your optical drive, wait a few moments, if PowerDVD starts via windows autoplay, quit PowerDVD. Now double click the previously created PlayBD.bat. PowerDVD should start now, playing the BD/DVD. If yes, proceed to the next step, otherwise double check the path and driveletter in 'PlayBD.bat'.
4. Check which drive letter is assigned to Virtual CloneDrive. (You can do this by double clicking an *.iso-file in the Windows-Explorer.)
5. Use Notepad or any other text editor to create another batch file, that will be used to play Blu-ray iso-files using Power DVD. It should contain these three lines:
"C:\Program Files\Elaborate Bytes\VirtualCloneDrive\vcdmount.exe" /d=0 %1
"C:\Program Files\CyberLink\PowerDVD10\PowerDVD10.exe" /fullscreen f:
"C:\Program Files\Elaborate Bytes\VirtualCloneDrive\vcdmount.exe" /d=0 /u
If the drive letter of Virtual CloneDrive is not 'F:' and/or PowerDVD was installed into another directory, change line 2 accordingly. If VirtualCloneDrive was installed into another directory, change the pathes in lines 1 and 3 accordingly. Be sure to enclose all pathes in double quotes.
Save the file into the previously created folder, name it 'PlayBDiso.bat'.
You should now have two files: 'C:\For_XBMC\PlayBD.bat' and 'C:\For_XBMC\PlayBDiso.bat'.
6. For testing purposes, reopen PlayBDiso.bat and replace the %1 (at the end of the first line) with the path to one of your Blu-ray iso-files, e. g. "C:\Movies\The Lord Of The Ring.iso".
Now the first line of PlayBDiso.bat should look similiar to
"C:\Program Files\Elaborate Bytes\VirtualCloneDrive\vcdmount.exe" /d=0 "C:\Movies\The Lord Of The Ring.iso"
Leave lines 2 and 3 unchanged. Save the file into 'C:\For_XBMC\', name it 'Test.bat'. Double click 'Test.bat', the iso-file should now play in PowerDVD. If yes, congratulations, halfway there . If no, double check all pathes and the use of double quotes.
7. Use Notepad or any other text editor to create or modify the 'playercorefactory.xml' in the userdata directory.
IMPORTANT!!! DONT'T modify the 'playercorefactory.xml' in the XBMC program directory (e. g. 'C:\Program Files\XBMC\system\playercorefactory.xml')!!!
You MUST create/modify the 'playercorefactory.xml' in your userdata directory (running Windows 7, this is 'C:\Users\<YOUR_USERNAME>\AppData\Roaming\XBMC\use rdata'), with <YOUR_USERNAME> being your user name
It should contain these lines:
<playercorefactory>
<players>
<player name="PowerDVD (BD.iso)" type="ExternalPlayer" audio="false" video="true">
<filename>C:\For_XBMC\PlayBDiso.bat</filename>
<args>"{1}"</args>
<hidexbmc>true</hidexbmc>
</player>
<player name="PowerDVD (BD)" type="ExternalPlayer" audio="false" video="true">
<filename>C:\For_XBMC\PlayBD.bat</filename>
<hidexbmc>true</hidexbmc>
</player>
</players>
<rules action="prepend">
<!-- Blu-rays -->
<rule name="Blu-Ray" protocols="bd" player="PowerDVD (BD)"/>
<rule filetypes="iso" filename=".*BD.*" player="PowerDVD (BD.iso)"/>
</rules>
</playercorefactory>
Save it in the userdata directory (see above), name it 'playercorefactory.xml'.
8. For testing purposes, open 'playercorefactory.xml' with InternetExplorer. It should open in IE with syntax highlighting, without any error messages.
9. Rename your Blu-ray iso-files using the naming scheme <NAME_OF_THE_MOVIE>.BD.iso, e. g. 'The Lord Of The Ring.BD.iso'
10. Run XBMC, Blu-ray disc and iso-files should now automatically be played using PowerDVD. Additionally the 'pay with...' context menu should now contain the menu items 'PowerDVD (BD)' and 'PowerDVD (BD.iso)' which you can use to manually play an optical disc or iso-file using PowerDVD.
I hope this could help someone.