A layer of Goop® may work better than the paint. It gives something for the crushed rock to hang on to. I predict the crushed rock will be significantly lighter colored than the original stone. As they say, "Spot test in an inconspicuous area first".
Instead of pounding rock, let Nature do the work for you by taking a few cubic inches of dirt from the area and putting it in a pail full of water. Swish it around a bit and pour off the water while the silt and organic debris is still in suspension. Dry the remainder and that should be somewhat representative of the rock of the area.