I don't need to wear both, the jacket is fine for anything over 45F, the liner is pretty poofy though, one of the reasons I'm not inclined to wear it. Bulkiness is a real concern, especially when it restricts your movements - although I'm not certain a sweater is any better, especially if the arms bunch up under the jacket.
I suppose I'll stick with the liner, it's not like I need a jacket at work.