Sweetbay Magnolia, Magnolia virginiana
If I had only one plant I could choose for its perfume for our zone 6 climate, I would pick the Sweet bay Magnolia aka Swamp or Swampbay Magnolia. The tropical scent is similar to a tuberose and plumeria (Frangipani) combined. The scent wafts around the side of the house it's planted on and you catch drifts of the perfume as you walk by or work in the garden near it. The tree also has a lovely form with open framework and the smallest leaves of the Magnolia family.
It grows naturally along riverbanks or in wet areas so it would thrive if you have a wet spot in your yard. As usual, I went overboard so we have 4 small trees now. I was so taken with the perfume of the first tree, I bought 3 more when the trees went on 1/2 off sale in the fall. The first tree is the one that's blooming now with more than 20 flower buds set to bloom. The other trees only planted last fall have very few buds but they are placed all around the house so it will be great when they bloom - even one flower is enough to experience its essence. The buds don't open all at once so you also have a long period of blooming.
The tree is evergreen in warmer climes but here in NJ it is deciduous. Still, a very attractive tree in my opinion.