I brought up the exact same objection when Kurzweil spoke at my college. He pointed out that the installed cost-per-watt and total installed capacity of PV exhibits an exponential trend over the ~30 years of available data. He extrapolated that within 20 years, 100% of the world's energy could be provided by photovoltaic panels.