I am quite convinced that Bond and Silva know each other, in fact, I think they know each other well.
I don’t think this is about them being casualties of a former assignment, in that they were ‘necessary losses’ and it’s some sort of revenge for that. All 00 agents know perfectly well that they will be sacrificed for the greater good of the mission and the UK. The big picture. They know the responsibility that M holds as the person who may have to make that choice. Hurt feelings, ‘I thought she loved me more than that’ etc could not possibly be what’s behind this.
What she has done must be much worse and much more fundamentally questionable than that. Remember, Bond’s loyalty to her will be called into question by a revelation from her past. So, what could that be, given the above?
I agree that the line ‘Her two survivors, this is what she made us.’ Is key.
Other things that I consider important are…
Bond’s reaction to ‘Skyfall’, in the word association. Coupled with the word being used at all in the evaluation process.
Bond taking M to Skyfall and knowing that Silva will follow.
And...
Big spoiler, The filmmakers have gone to a lot of trouble to make this the case, it has to be of vital significance.
M's past misdemeanor and Silva must have a strong connection with Bond’s family and Skyfall Lodge.