The novel YOLT was still fresh in your mind, having been released only 3 years earlier. ... But then you get a villain in a volcano and scenes set in outerspace and a film that had NO resemblance whatsoever to the book.

Whoa, whoa, whoa, whoa. Stop right there. YOU ONLY LIVE TWICE is actually
MUCH more faithful to the spirit of the Fleming novel than it's usually made out to be (as well as a far better film than it's usually portrayed as by both critics and fans - quite why its vastly inferior remake, THE SPY WHO LOVED ME, gets all the glory while YOLT is often relegated to the dustbin of 007 history is beyond me).
Don't let the fact that YOLT boasts "a villain in a volcano and scenes set in outerspace" blind you to the ways in which the film either follows the source novel or (when it can't follow it) pays sly homage to it. As Mister Asterix puts it, there is much more of Fleming's novel in the film than meets the eye.
Let's address this business of sly homages first:
In the book, Bond becomes a father (without knowing it). For obvious reasons, the movie couldn't use that bit. Oh, sure, it
could have ended with a "several months later" shot of the pregnant Kissy Suzuki, but, well, it wouldn't really have worked as well as it does on the printed page, Fleming having created a moving but doomed relationship between the amnesiac 007 and the Japanese woman. So what did the filmmakers do instead? Well, they gave Bond a different kind of life-changing romantic experience: marriage! (Okay, he gets hitched purely to help his cover, but, hey, getting hitched is getting hitched!)
Here's something else that "kind of" references the novel: Fleming's Henderson is not an effeminate, artsy-fartsy Englishman, but a rough, tough, foul-mouthed, beer-swilling, ultra-macho Aussie (is there any other kind?

). Hard to believe that the filmmakers didn't deliberately make the screen Henderson more or less the polar opposite of the literary one, as a lil' joke aimed at viewers who had read the novel.
But forget the astronauts and so on - for the most part, film follows book fairly closely (Bond goes to Japan - Bond meets Tiger - Bond becomes Japanese - Bond hooks up with Kissy - Bond and Kissy set off by boat for the foreign baddie's remote lair), and the filmmakers occasionally do an
outstanding job of conjuring the atmosphere of the novel (think of the fishing village scenes). Much of Fleming's "You Only Live Twice" is travelogue, and an enthusiatic (indeed, almost reverential) examination of Japanese culture - happily, the same is true of the Eon film.
Impossible (obviously) to know what Fleming would have thought of this picture, but it seems likely that he would have enjoyed the sophisticated humour injected - one presumes - by his good friend Roald Dahl. Some regard it as a goof when Henderson gives Bond a stirred-not-shaken vodka martini and asks him if he's got it right, and 007 doesn't reply: "Well, no, actually, I like them shaken-not-stirred." No way is it a goof - even more amusing than the fact that Henderson gets it wrong is Bond's immediate, unthinking decision to play by the rules of politeness observed by upper class English society, which dictate that it would be terribly bad form for him to do anything other than agree that his favourite tipple had been prepared perfectly by his host. *Sigh* If only the jokes in the current Bond films were as subtle. Nowadays it's all crude quips about being a cunning linguist and knowing where to put cigars.
I sometimes think YOLT has replaced GOLDFINGER as the Bond prototype. Austin Powers, and affectionate send-up of Bond, parodies YOLT much more than it does any other Bond film.

I agree that YOLT has replaced GOLDFINGER as the Bond prototype - did so a long, long time ago, many years before Austin Powers was even a glimmer in Mike Myers' eye. Which makes the collective unconcerned shrugging over YOLT all the more baffling.
BTW, aren't the fight scenes in YOLT the best of the entire series? Bond has never come across as a tougher customer than when duking it out with the Japanese thug and with Hans.
As for MOONRAKER, I'm one of those Bond fans who firmly believes it's an absolutely splendid 007 outing rather than a stinking embarrassment that ought to be, well, shot into outer space. But it doesn't touch YOU ONLY LIVE TWICE.
You know what? I'm beginning to believe there's a strong and convincing case to be made that, just as "You Only Live Twice" is the best Bond novel, YOU ONLY LIVE TWICE is the best Bond film. YOLT boasts:
- Connery, impossibly charismatic even in a performance often slammed as lazy.
- Arguably the finest cinematography of the series (and best use of widescreen).
- Arguably the the most impressive sets of the series.
- Arguably the best score and title song of the series.
- Probably the best use of locations of the series, making for a film rich in travelogue detail.
- Arguably the greatest "epic feel" of the series.
- An "iconic" Blofeld (who talks like a Fleming Bond villain ought to talk).
- The "element of the bizarre" (obscure Japanese poisons, piranha fish, etc.).
- Excellent casting.
- Superb action scenes.
- Sophisticated wit, snobbery and connoisseurship.
- A fair amount of sadism and eroticism.
- A refusal to take itself seriously (I find THUNDERBALL a bit po-faced).
Etc. The best Bond flick ever? Why not?