Several gardeners have reported that tossing citrus peelings around the perimeter will repel cats. Orange oil sprays will also sometimes work. Not always - remember that we're dealing with cats here. The other techniques is to use a much that has sharp points or a ruff texture. Try things like blackberry and rose clippings or lava gravel. The lava might be the best choice because besides keeping the cats out, it helps plant production.
A commercial product is Cat Scram.
The other option sounds gross but it works. Soak some of the problem animal's poop in water and then spray or drench the dark tea in or around the problem area. The animals will avoid the space. This technique also works for dogs.