AFAICS there's no way you could be liable for any sort of
penalty fare, either.

Not on the automated system, no. But if there happened to be an on-train
inspection the situation would change.

Would it, though? You'd have touched in at the start of your journey,
and would be travelling on a "reasonable" route to your destination,
so I can't see what you'd be doing wrong.

You are right. If you are using Oyster card to pay as you go and your card is
inspected while travelling on a train all they are going to do is look for the record
of you "touching in".