Incidentally, unlike in olden times when we had to beg, borrow, steal, or sometimes buy the FAA's publications, they're all available online now at faa.gov. Look under the tab, "Regulations and Policies", then click "Handbooks and Manuals". The information in the Airplane Flying Handbook, the Pilots Handbook of Aeronautical Knowledge, and the Student Pilot Guide is invaluable and necessary. You'll also see the AIM (Aeronautical Information Manual, once called and still occasionally called the Airmen Information Manual). There are gobs of other publications there, also, but those I listed will get you started. At some point before you take your checkride, you'll want to carefully review the Private Pilot Standard Test Guide, usually referred to as the PTS--you must be able to do everything in it applicable to a private pilot to the standards described in it.
I suspect you'll also seek advice on which student pilot materials to buy from private vendors. Each pilot tends to gravitate toward the ones he/she used. I like the King Schools materials, although admittedly John and Martha have hokey humor--but they get the job done well. I also like the Kershner manuals--easy reads, and very informative. Rod Machado's materials are good, although his humor while different from the Kings' gets old after awhile.
Other books to consider for your aviation library: Stick & Rudder, by Wolfgang Langwiesche, the single most valuable and understandable book on aerodynamics out there; Instrument Flying by Richard Taylor and Weather Flying by Bob Buck, both valuable for private pilot students and almost necessities for instrument pilot students. There are many others which are useful and helpful, too.
You're embarking on a process that is a lot of work--nobody ever gave away a private certificate. There's a huge amount of information that you must learn. But it's a lot of fun, too.
Cary