I did a little digging for you and found nothing that indicates mint would be the cause of your problem. However, I did find that a variety of other factors can cause fruit trees to abort their crop. Insects, inconsistent watering, not enough chilling during the winter (depends on the variety), and lack of trace nutrients.
Best program is the organic program, which makes the conditions more consistent, builds the tree's own system and health, and helps to keep the soil in top condition to produce the healthiest trees and the best fruits. As I said, my tree improved dramatically with some simple, proper care and is thriving this year. I predict both my family and our neighborhood squirrels will be happy!
Good luck my friend!