Is your intention to eventually plant the tree in the ground? If so, then I wouldn't leave it in the pot past a season. The roots will begin to circle and will continue to do so when planted in the ground....which is not a good thing.

I have had a ginko tree in a large terra cotta pot for 8 years. It goes where I go. If you love it, leave it in there. I also have a miniature crape myrtle in a beautiful red pot and it looks like a bonsai, since it's been trimmed. Trees don't have to be in the ground, just be sure you give it as much root space as possible and take care of it. If eventually you want to put a tree in the ground, buy another one.