Oh, I would take out the brick and replace with large rocks. When you do it though make sure you cut enough edge between the rock and the start of the grass so it's easier to edge and maintain a barrier. You can lay out some plastic in the space to minimize weeds and keep them from taking too deep a root if they do grow.