I suppose that anyone with the title of Olympics Minister is bound to go over the top. But Tessa Jowell excelled herself on Wednesday when she described the London Olympic Games as “gold dust... economic gold in a time of economic need”. This was two days after The Times had revealed the hushing up by ministers of a detailed Downing Street report in 2002, which forecast that there would be little economic upside to staging the Games, and no impact on the numbers of people playing sport. Ministers were told that their best hope was that the Games would be a morale-boosting party. They made the bid three months later.

To describe a project costing £9.3 billion as “gold dust” is the kind of idiocy people spout once they become courtiers to the International Olympic Committee. The Lords of the Rings lounge unconcerned on their feather beds of TV rights while economies plummet. But it is too late for them to pull their funding. Now that it has become convenient to present the staging of the Games as counter-cyclical investment, London should make sure that it builds only what we need.

The new mantra is “legacy”. The idea that community sport would benefit has faded since the siphoning off of £550 million from local sports projects. So has the hope of a tourist bonanza, since it has become clear that sports tourism blights other kinds of tourism, and that no city has ever seen an increase in visitors as a result of the Olympics. So now regeneration, the chance to revitalise a huge swath of London, is the prize.

If we get it right, Olympic advocates tell me, and create a new district where people want to buy homes, London might be quids in after 20 years, as long as the public authorities don't sell up too fast. If we get it wrong, we will be paying the bills for ages. And there is no doubt who will pay. The retreat of the private sector has become a stampede.

Turning the desolate Lea Valley, with its muddy river and grim pylons, into a thriving district is an enticing prospect. I have had a minor role in two initiatives to regenerate that area, neither of which came to much. The question is whether it can be done, and at what price. The challenge is how to cut the staggering cost of the Games, while also putting East London on the map.

It might help to admit that regeneration is a square peg that doesn't fit neatly into the five round holes of the Olympics. There are some very able people working on this project, but they make hilarious contortions to justify the IOC's demands. The argument made for building a velodrome in the Olympic park, for example, is that Manchester's velodrome, which has hosted track cycling in the Commonwealth Games and the UCI world championships, has helped the careers of British cyclists such as Chris Boardman to take off.

But surely that means we should be hosting the cycling events in Manchester? The argument made for the aquatics centre is that it will host a million swims a year after the Games - mostly by schoolchildren. Whether they will turn up seems partly to depend on whether the budget can extend to building a school on the site. But how many children need two 50-metre pools?

Boris Johnson has come up with the ingenious idea of luring a leading university into the vast undercroft of the even vaster athletics stadium. This would populate the area with students, and could even become a national sporting academy. That could stop the area sinking back into desolation. But this is still horse after cart. Can't athletics take place at Wembley? Does the Olympic park need a “permanent media centre”?

London's new mayor clearly dislikes vanity projects. He was sympathetic to shooting taking place at Bisley - which already stages international events - rather than in a new Olympic ground at Woolwich. That equestrian events could go to Badminton or Windsor, rather than Greenwich. But he seems to have meekly accepted a KPMG study that claims that these alternatives are too expensive, partly because the IOC would require “satellite villages” to be built for athletes. That is defeatist. Rowers competing at Eton Dorney are going to shack up at Royal Holloway College. It's not the athletes who are prissy, it's the IOC, which has not one “Lightning” Bolt or Phelps on its 105-strong board.

The case for building six permanent venues looks thin. The case for satellite villages is nil. I am told, by someone who built them, that parts of Sydney's Olympic village were “posh Portakabins”. I am told, by someone else who is building part of London's Olympic village, that he fears “eight-storey blocks of luxury flats marooned in an incomplete suburbia”. Stratford station is close, and there will be a Westfield shopping centre with the biggest John Lewis and Marks & Spencer in Britain, but this is still a long way from central London. And there are clear worries about the disjointed nature of an Olympic park built around venues whose after-uses are unconvincing.

Regeneration is not just about buildings, however. It is about assembling enough land in one place to make a difference; cleaning it up ready for development; putting in transport connections and spreading economic life from the mega-projects far into deprived communities. I am proud to have led part of the South Bank regeneration in London in the 1990s. But one of the biggest challenges on the South Bank, Tyneside and elsewhere has been spreading the regeneration benefits beyond the vicinity of the big riverside venues. This challenge is nowhere near resolved in the Olympic thinking.

The good news is that London's Olympic project has already banked the most vital regeneration ingredients. Huge tracts of land have been acquired. Hideous pylons are being dismantled. Roads and bridges are connecting the two sides of the valley. And all before most of the vanity projects are under way. So London is in a strong position to call the IOC's bluff and rethink. We need more bread and less five-ring circus.

Times Recommends

• Punish savers and make them spend
• A loose cannon all too likely to be fired
• Wealth will bloom again