Anyone familiar with hymns has an advantage in knowing "O" from "Oh". The hymn-literate person may have an intuitive descriptive understanding of this.
"O" introduces a direct address (for fancy types, the vocative case):
I am very tired of swimming about here, O Mouse!
O dearest Jesus, what law hast Thou broken?
O Kitchen! my kitchen!
"Oh" is an interjection.
Oh, I'm not particular as to size.
Oh, that I had a thousand voices to praise my God with thousand tongues!
--but oh! We were all such earnest students!
The O- section of the first line index of LSB is a helpful diagnostic resource. Then for fun, compare it with TLH's. ;)
Oh, who cares?