That detachment is part of the process of "Enlightenment".
I have also been struggling with my fear of it.
I realized this is why:
I had come to think that Desire=Pleasure.....
See, for years I desired something, went after it, and sometimes got it. Then there was a brief satiation before the desire returned.
So this set up a misunderstanding that I needed to have desire or else I would never experience pleasure/happiness/joy.
But this is my conditioning.
That detachment is about a stillness, remaining non-reactive in the face of life's ups and downs.
It is necessary in order to maintain connectedness with the identity of the True Self, the Witness, the Eternal self.
It is necessary to recognize that we are not our desires....
We come to identify so closely with our desires that we think that losing them is losing the self.
I became celibate for a while....partially because it TERRIFIED me. I realized I had some unbeneficial attachments to sex if the thought of not getting it brought up so much resistance, terror (literally), sadness, frustration, anger.
Meanwhile, much later on, I have come to understand that rising above the desire and become detached actually makes room for real, constant joy that is not contingent upon the satiation of animal desires.....
I'm still working this through, but it's getting better and less scary.
Ra Un Nefer Amen (who writes about Kamitic--Egyptian--wisdom) explains that Ausar Man (God-realized man) is like a zombie....dead to this world. But this doesn't mean Ausar man cannot experience joy. On the contrary, Ausar man, being FREE, is in constant joy.
Also, Ausar man does not become disconnected from his fellow-man. On the contrary, because he experiences the deeper, eternal connectedness, Ausar man is MORE connected to humanity.
I would love to build on this with people (and hear refutations) because this concept is really still work-in-progress for me.