Port Townsend
Port Townsend (called PT by the locals) has transcended its hard working shipping-port history to become a classy tourist destination. It’s a cute town and the best thing to do here is to just hang out and walk round. PT's entire business district has been restored and was declared a national landmark in the late 1970's. The downtown waterfront and business district are home to lots of art galleries and coffee shops, while the neighborhoods are filled with Victorian architecture.
The Rose Theater, built in 1907, is a great place to see a movie on a rainy day.
Accomodations:
Many of the local mansions have been converted to bed and breakfasts, making PT B&B central. In winter or the low season, you can often find good bargains. However, in the summer, the rates soar higher than the nearby mountains.
A great budget option is the Olympic Hostel in nearby Fort Wooden State Park. Consider also staying in Port Angeles or taking advantage of the great camping options that are close by.
Personally, we’ve also stayed at the Manresa Castle (again, check for cheap rates in the off season) and highly recommend it. The lounge is great for a drink.
Dungeness Recreation Area
This often overlooked recreation area near the town of Sequim is worth a stop. It is home to the longest natural sand spit in the U.S. (the Dungeness Spit) and is a great place to go hiking.
Hurricane Ridge
When you stand on Hurricane Ridge, it feels like you might be on top of the world. Ninety-five percent of Olympic National Park has been designated Wilderness and, when it's clear out, it seems like you can see most of it from the top of ridge. The highway from Port Angeles is a windy 17 miles long that gains close to a mile in altitude. If you don’t have time to go backpacking, you might just drive to the top of the Ridge, where you can pick up lots of short or long day hikes [pdf]. Most trails go along the ridge and offer fabulous views of the Olympic Mountains, Strait of Juan de Fuca, and Vancouver Island - if you get a clear day. However, it's not unusual for the ridge to be socked in by clouds. In winter, there can be treacherous driving, but there’s actually a cool snowboarding/ski area at the top of the ridge (surely, one of the very few chairlifts inside a national park).