Gutter build up as soil/soil enricher

So today I had the sudden urge to clean out the gutters, and it looks like no one has ever cleaned them before. Note that this is an old house and I just moved in. As I was cleaning out all the crap from the gutter, I began to wonder if I could use it as soil, or ar a soil enricher. The gutters collect tons of leaves from nearby trees and the occasional orange from the neighbors orange tree. There were a few worms in the gutter and it looked like really rich soil. Has anyone tried to use this stuff before? I figured it would work because it is basically just compost. What do you all think about it?

Ask Finshaggy bout this? He is an expert on alternative methods...

I would take all the stuff from the gutters,properly compost it then it could be safe to use.
But there are better options