Sometimes, I really, really, really despise this Oyster system.

To be fair, it's probably not the system but the people implementing it
that's the problem. I'm often in the opposite situation to you, where I'd
rather have Customer Charter refunds issued directly to my Oyster Card but
instead receive paper vouchers through the post which I then have to take to
a ticket office and get them credited to the card there.