The main reason for the empty seats is that a large percentage are given
away free to sponsors and national (as in "other nations") Olympic and
sports organisations.

The various sporting associations are allocated seats so their
officials (mostly volunteers) and competitors can get to see some of
the Olympics as well.

Obviously, at the start of the games, many of those are still busy
with their own events but, as the games progress and many of the early
events are concluded, they can find time to take in the rest of the
games and the problem of empty seats is then lessened.