Beijing had no appetite for starting a war over Taiwan "its military could not win".

In the past, it was clear Beijing had no appetite for starting a war over Taiwan its military could not win. (Jared M. McKinney and Peter Harris)
What is the antecedent of the relative clause in Italics?

There's an elided "that" there.

...Beijing had no appetite for starting a war over Taiwan that its military could not win. The antecedent is "a war over Taiwan".

The antecedent for "its" is Beijing. Think about it. Why would China be reluctant to start a war that it thought Taiwan could not win?

Agreed as far as 'its" is concerned. However the OP was asking about the antecedent of the entire relative clause, which is "its military could not win".