You need to build a proper foundation for any surface that you are going to put in the dirt. Dig everything out to a depth of 6", level the bottom as you dig it up. Lay down a layer of stone 3/4 and smaller to a depth of about two inches, put in a barrier layer (landscape cloth), above this layer fill in with about 2 to 4 inches of sand or rock dust, keep it level. Set pavers on this and then fill in the cracks with the same material, sand or rock dust.