Her suicide in the novel is a pill overdose and she leaves a note for Bond confessing she had been working as a double agent.
"I knew it would be the end of our love if I told you. I realized that I could either wait to be killed by SMERSH and perhaps get you killed too, or I could kill myself."
She was being blackmailed by Quantum (or Smersh in the novel), she wanted to save the life of her Algerian (or in the book, Polish) boyfriend, and she probably didn't want to draw Bond further in and hurt him. One other reason. She probably thought that, eventually, she'd be found out by MI6 and be facing a long time in "clink". In the film she was the one who entered the password - her name - to transfer the money. At that point only she, Bond and the bank - and Quantum? - knew where it was. It seems she had access to it, and short of the bad guys hacking into the bank and moving it electronically, or Bond himself drawing it all out - unlikely - only one person could have handed it over to the villains.
In the film, Vesper's the only one who knows where the money is as she's the one who enters the account number. The 'organization' doesn't have it, that's why they send Gettler on her trail. MI6 doesn't have it, as M calls Bond to ask why it hasn't been transferred yet. Bond then can't access his account and realizes his trust in her has been betrayed. (Also, in QoS, Mr. White intimates that they would have used Vesper to trap Bond into a similar blackmail--"We'd have had you too. I think you would have done anything for her.")
If this is the case, how come she leaves Mr White's number for Bond on her cell-phone? The ending doesn't quite stack up convincingly for me. As soon as Le Chiffre's has been disposed of and Mathias is out of the picture - the film loses it's way.
In the novel's suicide note, she explains that her boyfriend was caught by the Russians and leaves the contact information--number and address--they used. In the film, M tells Bond the boyfriend connection as there is no note, and Vesper leaves her phone for Bond to find Mr. White's number. I think this is the text she sends to herself on her phone while pretending to contact her employer in the hotel room that final morning. M also tells Bond that's why Vesper saved his life by attaching the defibrillator to him in his car. This was after she wouldn't authorize the $5M re-buy-in. She's further plunged into the web when the CIA ponies up the money. Bond is the only one that can beat Le Chiffre, and now the organization has their bets covered regardless of who wins. If Bond wins, they kidnap Vesper, if Le Chiffre wins they get their money back (and then presumably kill Le Chiffre anyway since he can't be trusted.) "She made a deal with them in order to spare your life," M explains. "She must have known she was on her way to her death."
Two layers the films (CR and QoS) add over the novel is the film makes explicit the boyfriend (Yussef) is an agent of the enemy (that may be the case with the novel's Polish boyfriend but Fleming doesn't elaborate on it.) And the film Vesper has a direct hand in the financial transactions of the winnings whereas not in the novel to my recollection. Why she didn't hand it over is up in the air. She knows it would directly finance terrorism? She's in love with Bond but doesn't know what to do with the money? Whoever she gives it to, MI6 or the organization, she fears they will come after her, even after Bond kills most of the baddies in the Venetian house (but not Mr. White.) So suicide she figured was her only option (and drowning is a very painful way to go.)
"I'm afraid I'm a complicated woman," the movie Vesper told Bond at his rehab.
One final thought, the book's nature of evil speech concludes with Mathis telling Bond to "surround himself with human beings...they are easier to fight for than principles." But with Vesper's suicide he finds that is not true and vows to "go after the threat behind the spies, the threat that made them spy." In the film, it is the organization that trapped Vesper into her predicament. In the novel it is SMERSH, a "cold weapon of death and revenge." Hence I interpret the fall of a house in Venice in the film as a metaphor that Bond will never be allowed to settle into a personal domestic life. His profession is his life.