You can feel temperature. When you walk into your kitchen and touch the granite countertop and the wooden cutting board at the same time people will say the granite is colder than the wood.

I suppose this comes from the fact that when you touch different materials some feel colder than others do, so people just assume that they must be feeling a difference in temperature.
You don't actually feel the temperature of an object. You feel the rate at which heat is conducted either towards or away from you.

In this picture people were given a block of aluminum and a block of plastic. They all said that the aluminum was colder. They were then asked which would melt a block of ice faster. They thought that the plastic would melt the block faster since it was warmer.

external image ?auth=co&loc=en_US&id=74380&part=2

When ice was placed on the blocks it melted a lot faster on the aluminum than it did on the plastic. This is because the aluminum is a better thermal conductor than the plastic is. Therefore it conducts heat into the ice faster than the plastic does, thus melting the ice faster.