Points when refinancing

I am not a giant fan of paying discount points to lower your interest rate in most cases. If you have the money today, and expect to be there past the break even point, it isn't necessarily bad, especially as there may be a sweet spot that gets you more bang for the buck on any given day. On the other hand, many people pay points, then are not either in the home long enough to recover, or refinance too soon to recover. Plus, for many, the money spent on points might have been better spend on just reducing the original loan amount. At the end of the day, it isn't bad or good. Just a tool in the toolbox that each person needs to decide for themselves. I lend in SD, MN, and WI, and can be found at iMortgageJoe.com - NMLS274132

It is not a really good idea to pay down points when refinancing unless you plan to be in that house for a long time. Why not use the buy down money and make a couple extra payments to the principal amount of the mortgage, so you will not be paying back as much interest over the time of the loan?

when you plan to keep the loan for a long time

Pretty much what they said. In general if it is 0.5 point or more to get 0.125 better rate it is barely or not worth the cost; if it is less then your "payback" period is shorter. I've personally refinanced and paid points to get the better rate, then ended up selling a few years (or less) later and didn't keep the home long enough to get the benefit of the additional cost.