A recent video posted onto the Google Webmaster Central Youtube Channel and highlighted by Search Engine Land shows Matt Cutts setting out how Google determines page speed.
Within the video, Matt Cutts explains that around 1 out of 100 searches carried out on Google are affected by slow page speed and this amounts to roughly 1 in every 1000 websites that have an issue with site speed.
If your website is slow then Google isn't the only one who will potentially penalise it, as your site visitors are unlikely to wait around either, which will directly affect the number of enquiries or sales that you receive.
A quick test using a free tool such as Web Page Test will highlight if you have an issue and ongoing monitoring can alert you when your site slows down.
Here is the Matt Cutts video in full.