We could meet up at Victoria since I'm going there anyway, but I
rather not have him change trains with all the luggage.

Understandable, but there is the option of changing from the Piccadilly Line
to the District Line (for Victoria) at Hammersmith or Barons Court where the
interchange is a few yards on the level across an island platform. That
would at least make his incoming journey bearable with luggage even if it
doesn't help with the later journey.