Denial

In my case, I was deathly afraid of digging into the reason why I had this neurotic attachment to someone I hardly knew. I'm not saying your husband is the same, but in my case sexual abuse at a young age caused dissociation of this part of my life. After confession of everything, I came to a "locked box" (my abuse memories) I was almost more fearful to open than I feared God. Add to this the abusers use of keeping secrets that was imprinted on my brain and exposing things to the light was very difficult for me. In this suppressed part of me, I was confusing the person with my long-lost abuser.
If I hadn't confessed every shameful thing I was aware of in my life I had ever done to my wife, I believe I would never had recovered the memory of my abuse, it's the only way to be free.