You're misunderstanding the purpose of the HDMI modulator. It takes a HDMI feed from the STB (in your case the HD51) and distributes it to other TVs via a coax connection. The AV plug in your last post may work into a conventional SCART/AV modulator but not into this particular one. The F sockets on the modulator are for a terrestrial aerial feed input and the output to whatever TVs you are feeding. A separate cheap STB with no tuners could take a stream from your main HD51 over CAT 5 cable or possibly WiFi if your router supports it. This would be more flexible than the HDMI modulator which limits you to viewing whatever is tuned on the HD51.
As regards the AV port on the HD51, I've never seen it referenced in the ViX setup. I don't know if it works on a ViX image as I've never tried it. It's a bit redundant on a HD/UHD receiver as composite AV is very low quality. That said, my other two receivers (GB Quad+ and MB Twin) support simultaneous output from HDMI/Component and Composite or SCART.
The F sockets on the modulator are for RF (radio frequency) signals. AV feeds are baseband analogue and are not RF, so no chance of feeding them into those F sockets.