“It will not be allowed to be touched by anybody, destroyed, ripped down by stupid people in order to build something in a slightly different shape.” For instance, he suggested a future White House occupant could decide: “Let's not have an oval anymore, let’s make it square.”
So, fun fact: The only reason the Oval Office is oval is because it used to be in that curved bit above the columns in the very center of the building, and when Taft oversaw the expansion of the West Wing (initially started under Teddy Roosevelt), he wanted his new office to be a copy of the old one. The West Wing was completely rebuilt during the Hoover administration after it was damaged in a fire, and the current location of the Oval Office is in a corner where it would look pretty dumb if the entire East and West Wings weren't cropped out of nearly all photos of the place.
Oh yeah, and this sent me down quite the rabbit hole where I learned, among other things, that the entire White House has been gutted and rebuilt essentially from scratch at least twice, most recently within my parents' lifetime, so the idea that there's anything sacrosanct about its current incarnation is ridiculous.