To add, graphics memory does not get pooled in use unless you're using split/scissor frame rendering. This is normally for 3D workloads via DirectX, OpenGL, Vulkan or Mantle. Asymmetric Crossfire took full advantage of this mode before it became mainstream, as it was easier to proportion work between GPUs this way.
Going back to topic, you're only able to consume the 4 GB graphics memory in your dedicated GPU for video. You can always choose to disable the APU's integrated GPU (if your BIOS allows you to), if you want to stop reserving memory for it (this will also prevent you from using it on HSA-optimized applications).