You know how when you sometimes smell things and it reminds you of a certain VERY vivid memory? Well, this is what HPPD is. Something you experience later reminds you of some thought you had in your altered state of mind, and you partially enter that state of mind for a moment. You're remembering; it's just that this particular memory happened when your perceptions were completely altered. Doesn't last for but a few moments, and it always goes away for good after a little while. It doesn't interfere with everyday life (even though there are occasionally visual effects), and is more of an annoyance than anything. Most people never experience any symptoms of HPPD.
Couldn't have put it better myself.
Last summer my friends and I did a fair amount of acid (we got some really primo shit from Niagara Falls, Ontario if that helps you, Dankdude) and had an equal number of mushroom trips. Suffice to say we've experienced no blood clots in the brain, even after the night we all ate a quarter ounce of top shelf shrooms each. Things got really intense some nights, but we're all fine now.
I've had a number of acid flashbacks since then, but nothing serious. Bottom line if you can handle tripping on acid for upwards of eight hours, you can certainly handle a fleeting flashback. My experience with these flashbacks was that my mind was acting almost exactly as it was while on the drug (racing from one conclusion to another, feeling caged up screaming for escape is the best I can describe it) for anywhere from a few seconds to one or two minutes but there were no sensory hallucinations whatsoever. I attribute the flashbacks more to the powerful memories of the experiences than the toxicity or quantity of the drug.
As time progresses these flashbacks are generally becoming more and more sporadic and lasting for shorter periods each instance.
From my readings of Erowid, I'm led to believe that MOST recreational drugs (at least the ones I'm inclined to ingest) are of lower toxicity than cigarettes, alcohol and even fast food. Mushrooms, THC, mescaline, LSD all need to be taken in RIDICULOUS amounts to be physically harmful to the average Joe. That said, heavy acid use can lead to psychological stress and possibly even damage IMO.
As far as Ecstasy goes I read a case a year ago about this guy in England who had a 25 pill a day habit. He kicked it almost cold turkey from what I remember with a short relapse but he's been clean for a few years and is being monitored by doctors. They've so far found no evidence of any lasting damage other than seemingly random short-term memory failure less severe than that experienced by a chronic pothead.