When Fleming first started writing the books, he wanted to create a bland dull character. Â There were two reasons for this - he wanted to write spy adventure stories, but he was tired of larger-than-life characters that were the mainstay of British adventure fiction. Instead, he wanted a character who was nothing but a blunt instrument of government policy. Â Fleming also realised his character would be more popular if it was a blank canvas that readers could project themselves onto and fantasise about being.
As the books became more popular, their (white, male, English) readers responded to an English hero who - in a time when Britain was losing its empire, was far less important than it was 50 years previously, had just gotten put of food rationing nearly a decade after WW2 had ended, etc - went to exotic places, ate the best food, drank the best wines, met beautiful women. So Bond became more heroic and more about English virtues like bravery, determination, comparisons to St George, etc. Â He represented an England who was down, but not out and still fighting. (It was only late in the series when the movies started being made that Fleming retconned Bond to give him a Scottish father - a tribute to Connery - and a Swiss mother.) Â He was white because at the time England was overwhelmingly white, and so were his readers.
Fifty years on, England has changed. It has become more open, less sexist, more multicultural, less privileged, less classist, more socially mobile, more questioning, more confident. Â Bond's fans include a lot of women, and a lot of non-English people. Â And Bond has changed with it - the open racism and smoking of the books has gone, and the benzedrene. Bond is still a sexist womanizer, but he's shown to be out of step, and he doesn't dismiss women as useless as he did in the books. Â England will continue to change, and Bond will continue to change with it, and those changes could possibly (and IMO comfortably) include casting a non-white actor in the role.
I see the backlash against the merest possibility of this is not about "being true to the character created by Fleming". It's more about the part of the audience that is white and male wanting to project themselves onto Bond's blank canvas - girls, gadgets, fast cars, beautiful locations, no commitments, all while being being impossibly cool. These fans don't want anything that messes with this - see the hand-wringing at the time of Casino Royale, when the news broke they were casting a blond actor and giving him a Ford Mondeo to drive. Even though non-white Bond fans have been dreaming about being like James Bond for 50 years, these fans are worried that having a black actor in the role will break their fantasy.