When you refer to something that happened in the past, be sure to transition properly so your readers don’t get confused. This is done by using past perfect tense, i.e. “He HAD wondered years ago, whether blah blah blah.”

After that, switch back to simple past so you don’t create a different distraction by having too many “had” whatevers.  Then be sure to close the flashback by using perfect past again, cluing your reader that it’s now back to the present story action. Otherwise, your reader can get lost in time. In some books I’ve encountered I’ve had to read a few paragraphs over several times before I figured out what was past and what was present.

As always, throwing a reader out of the story is a huge faux pas!