Actually that would be very difficult. It would require huge amounts of energy that we don't have. A solar panel that is well situated takes about 2 years to generate enough energy to offset the energy it took to create it.
We now get ~ 1 % of kWh from solar PV now. To get to 1000x, lets estimate 100x each year for 10 years. We would need 3x the electrical energy we produce now just to make those panels and supply the power we use now. That's not feasible in any way.
Yes, you could scale up slowly, using the energy produced by the solar towards making more panels. But silicon foundries need steady, reliable power 24/7/365 - so we are back to the storage question. And it would be a slow crawl.
Certainly not "hardly difficult".
And "today's number" was produced over many years (though I think most were created in maybe the past 4-5 years?).
Note that while Elon is putting solar panels all over the Gigafactory - he isn't disconnecting from the grid - he needs that fossil fuel and nuke and hydro supply for stability.
-ERD50