We returned from the south of France 4 days ago. We have visited Provence at least a dozen times, maybe more. We generally rent self-catering accommodation, though we are always on the lookout for good restaurants too. We tend to follow your general approach: find a base for several days and radiate outward from there on day-trips.
The Luberon is an obvious candidate, but I would avoid places too close to Avignon -- that's a fairly densely settled area, with a lot of traffic. Very different from when I first stayed there, in 1970! If you go a bit further east - east of Gordes, anyway - you can find less spoiled, more genuinely rural spots without sacrificing your access to the picture-postcard villages of Menerbes, Oppede-le-Vieux, Lacoste, Bonnieux, St Saturnin-les-Apt, Venasque and so on.
We also returned this time to the Fontaine de Vaucluse, which I found a very cheery, lively place worth a short visit.
We tend to avoid the big tourist draws: Gordes, Roussillon, Isle sur la Sorgue. We made an exception for Avignon and were very glad we did. We did not tour the Palais des Papes but concentrated on strolling the back streets of this remarkable city. If you go, do NOT miss the Rue des Teinturiers, lined with plane trees bordering a XIXth C canal.
If you like hiking, you're only about 45 minutes from the Gorges de la Nesque, which are both visually impressive and physically challenging. South of the Luberon valley (about 45 minutes) are the Gorges du Regalon - a slit canyon with cliffs about 100 or 150 feet high and a minimum width of about 5 or 10 feet in some spots. Not for the squeamish.
FWIW, we rented a gite for 4 nights at the Domaine des Peyre outside Gordes - a large and lavish vineyard property that we had all to ourselves for 2 of the 4 nights.
_______________________________________
Another spot we consider an ideal base is the vicinity of Tarascon/ Beaucaire.
The chateau of Tarascon is an extraordinary gem; otherwise, there is not a great amount to see in these twin cities facing each other across the Rhone. We stayed twice just outside Tarascon in a gite at the Chateau la Panisse, an olive-producing property with a grand main house.
From there, it was a short drive to the Alpilles and Les Baux; to Arles and the Abbaye de Montmajour; to the troglodyte Abbaye de St Roman. Nimes, Pont du Gard and Uzes are less than an hour away.
Then there's the whole of the Camargue at your doorstep, including the bull-fighting centre of Les Saintes Marie de la Mer.
The popular tourist town of St Remy is also nearby but has never interested me, so we skip it.
_______________________________________
The Drome is an unfairly neglected "departement" whose southern half, the Drome Provencale, is my favourite region of all. It welcomes few N American visitors and generally moves at a slower pace than the urbanized countryside of the Luberon or the Aix countryside. It is gorgeous and well equipped with restaurants, weekly markets and other amenities
The Vercors mountains are a highly visible backdrop and there are lots of rolling hills but there are also verdant plains of vineyards, a pleasure to drive. Among my favourite places:
La Garde-Adhemar (where we spent 5 nights -- our 3rd stay at "Chez Loubet", a superb rental)
In the same general area, near the Rhone: the chateau ruins at Allan, St Paul-trois-chateaux, Grignan, Suze-la-Rousse, Rochegude
A bit further north: Marsanne, Mirmande, Grane, Saou, Bourdeaux, Comps, and the medieval quarter of La Begude de Mazenc
In the east of the region: Nyons, Dieulefit, Poet-Laval, Buis les Baronnies
The Drome is also a good base for reaching the Gorges de l'Ardeche just across the Rhone.