Windham/Andover is near the sea coast in NH so we recommend exploring Newburyport, MA or Portsmouth, NH if you are looking for a cute town to visit. Or of course, if it isn't too cold, you can visit the beach in Hampton, NH.
Newburyport
36 Merrimac St, Newburyport, MA 01950
Newburyport is a beautiful New England town. Great for wandering in and out of little stores!
Portsmouth, NH
14 Hancock St, Portsmouth, NH 03801
Also full of fun funky stores, Portsmouth also has a few great historical sites like Strawberry
Banke.