Update:
I connected the xtreamer to the AVR using both HDMI and optical.
I then assigned it so that the AVR gets it's audio from the optical connection, not from HDMI.
So now if I power on in this sequence... TV+xtreamer. I get the audio coming out of the TV.
And if I then power up the AVR, the sound is properly recognized (DD5.1/DTS etc) and audio to the TV cuts out.
If I power off the AVR, the audio goes back just to the TV.
I'm satisfied with this setup for now. Now my family can just power on what they wanna use without too much complication and when I use it, I just power on the AVR to get the sound I want.