Not all, but most of it is from infrared, with a little under half of all light recieved from the sun being infrared.* But yes, we would lose our only source of renewable energy and eventually either freeze or starve to death.

*Based on a quick skim of a Wikipedia article