The instrument is run with a cable from port 1 to port 2.
The cable has a male SMA to female SMA connection and
calibration was made at that point.
A relay was then inserted as a DUT at the calibrated point.
If the relay is opened for a short time while port 2 is active S12 would go from essentially zero to -50 dB. That is fine and expected. It is hower totally unexpected that S21 changes by about 0.1 dB at the same time.
I do not understand what the instrument is doing and why so I have decided to run this instrument without calibration because in its factory preset state the instrument behaves completely as expected. The relay only affects the two S parameters one would expect.