“It is a glorious honor [to be able] to meet someone despite them being prime minister.”
The expectation is that someone like a prime minister would be inaccessible. The fact that the actual situation is not exactly undesirable may give the sentence an ironic twist.
The explanation is correct, but I think the "despite" in the header feels awkward, as it would often be structurally unnatural to translate the Japanese examples into English that way. I think the prime minister example above feels odd because most of the examples of ～ともあろう seem to be cases of A being well regarded and B being a disappointing or undesirable outcome. However, ～ともあろう is more broadly used to express surprise or disappointment in light of how the subject is regarded: that is, despite my expectation of A, B resulted. A is often, but not necessarily, a person, group, or position of high status, respect, or responsibility, while B expresses a surprising or undesirable outcome. The key is really the contrast with the expectation set by A’s standing. A doesn't have to be positive and B doesn't have to be negative. For example, 彼女ともあろう人が、こんなに寛大な対応をしてくれるなんて、さすがだね。(It's impressive that she, of all people, would respond so generously.) In this case, the expectation of A is negative, and B is contrary to expectations but positive.
