I think site speed is really a factor (ever since!) in having good SERP at Google; but, there are at least 100+ other factors to consider to rank well in Google. In short, don't panic so much about your site speed, but pay enough (not so much!) attention about your site speed. How you get my point.
I find this hard, because how Google can determine if your website is slow or fast? Another thing is what of ranking method does Google use in terms of speed of a website?