I think you may be right! I also have family and friends who consider fanfiction my slightly peculiar hobby, but they also engage with stories and fictional worlds and try to figure out how they work and what might come next. At least some stories are alive for them, even if they don't feel a need to add to the text. (Now I am trying to figure out what makes the difference between people who simply engage with stories and people who actively create stories -- not-writers vs. writers, as you say.)

Anyway, as I said above to katta, I think that never properly engaging with a story to the point that it becomes real in one's mind would make for a rather grab and drab experience of the world.