The short answer: Everything is radioactive, to some degree, so the question should be, is it too radioactive to use? I think the short answer is easily "don't worry, go ahead and use it."
The longer answer: Granite is a general term for a type of rock found around the world that is composed of various igneous minerals (quartz, feldpars, hornblend, etc.) that cooled slowly enough that minerals in the magma were able to crystalize but the movement wasn't finished and many types of minerals formed and mixed as the area cooled. Metamorphic minerals such as mica and garnet are sometimes found mixed in granite because the magma can be extruded into a metamorphic area and remix the native minerals into the igneous minerals. Pegmatite is simlar to granite, igneous crystals mixed together, but it's identifying feature is that it has much larger crystals. Contrast granite to lava flows and basalt, where the molten material cooled so quickly that no crystals had a chance to form.
I'd hazard a guess that if there was uranium in an area where rock was being mined, then the mining activity would default to mining for the heavy metal uranium, and tailings from uranium mining would not be sold as a consumer product. Uranium is usually found in what is called "massive" or a dense, non-crystalline matrix referred to as "pitchblend" because of it's color and streak. My elderly college copy of Dana's Minerals states that is is most likely found in the massive form, though a crystal will turn up occasionally.
"Small amounts of uraninite have been found at many localities associated both with pegmatites and with the ore minerals of silver, lead, and copper. Thus in the United States isolated crystals are found in the pegmatites of New England and North Carolina" (223).
This doesn't mean it can't happen, but it is very unlikely. I hope there are a few safeguards in place in quality control in mining, that radioactive ore isn't mixed in with a consumer product like decomposed granite.