i would mix in some perlite for drainage. Make sure it is fully composted to that you dont get nutrient burn. Worm castigns are a good fert for little plants as they keep them healthy and dont burn. Real good with the bacteria too, so is compost. Did you buy a big bag? Those bags are diferent from that full black rich compost! I would mix some soil too.
IDk, what others will say, I bet you could but for a better mix add that other stuff
add some dolomite lime, 2 tblspoons per gallon
and some perlite
and some peat
and worm castings if you can find em or chicken poop
and a little comercial soil
MY SECRET RECIEPE, DONT TELL ANYBODY
it all depends on the compost pile: what's in it? how long has it 'cooked'? I wish I could grow on compost piles(especially my biodynamic compost)...we always have kitchen waste trying to grow- and doing an amazing job at that!...even on unfinished compost...and worm castings? hah! try a baitshop per shovel of the worms already there...it's insane-how many worms!