Brosnan was 49 when DAD released and he looked both out of shape and too grey to be playing Bond. Also, we don't want another Senior citizen Bond do we?
To be fair, Brosnan never really had a build like Craig. He wasn't entirely or even out of shape out all across four films - he had more of just an average man's build in comparison to Craig's bulky build in CR and more Lean build in QOS&SF (I'm also assuming Spectre). For 49, he looks pretty good in DAD, especially considering it was his last film. Moore, however, didn't look so good...