Visitors to Torquay would have little idea, as they pull in to the train station, that a battle is raging behind the omnipresent palm trees. It seems so unlikely. The balmy air of Torquay, with its bucket-and-spade families, Regency facades, and ice cream ("made with real clotted cream") frosted by feuding? Torquay, home of Agatha Christie, where present-day Miss Marples take seaside constitutionals? Torquay, where sun-deprived British flesh seared vivid from our very own tropics hides itself beneath garishly captioned fake designer clothes?

Yet pause a minute, and reflect. This was where John Cleese was inspired to write Fawlty Towers after staying at a local hotel. Think of Basil laying into his Mini with the bough of a tree, and you will realise that the town's placid exterior may conceal all sorts of dark fantasies.

I only hope that Jan Siegieda, the newly arrived head of tourism for the English Riviera, does not fall victim to any Basil-type rage. Jan is not content for the Riviera, which runs the length of Torbay and encloses the towns of Torquay, Paignton and Brixham, to rest on its reputation. Jan is a hardened operator. Having earned his spurs buffing the images of Margate, Ramsgate and Broadstairs, he has been brought in to bring the Riviera upmarket. He is surely the man to do it. Anyone who can stand up to a Margate café owner and tell her that her chipped mugs are below par is going to clean up when it comes to the cream-tea brigade.

Not everyone welcomes his approach, however. Jan has spent four months booking incognito into local bed and breakfasts. He was eventually rumbled and subsequently hit the headlines in the local Herald Express newspaper. "Only once was I offered coffee in a cafetière," he is quoted as saying. Other complaints include grapefruit being served in – horrors – clingfilm. His comments on "flock wallpaper and colours that didn't go together" are followed by the information that "Mr Siegieda ... has a fine arts background." Who is he, this foreign-sounding johnny, with his fancy, arty ideas? Who is he to tell the b&b-running burghers of Torbay that a fried egg cooked in the same pan as sausages and bacon isn't, actually, a vegetarian breakfast?

On the second day of a weekend trip to the English Riviera, we shared a taxi with Jan, who told us of an incident earlier that afternoon when he had taken a film crew to a fairly grand hotel. Cream teas had been requested, but arrived without the tea. When tea was finally forthcoming, only two teacups were in evidence, instead of the four required. Jan went back and had words with the manager, informing him that he wouldn't be bringing media guests to his establishment again unless he smartened up his act.

When we dropped Jan off, the taxi driver made it clear he wasn't impressed. "Only two teacups, oh, I am sorry," he said. "Sack the manager. Knock the whole bloody building down. What does this guy think he's doing? He's supposed to be covering these things up, that's what he's paid for, not telling the press about them. And he looks like a Seventies porn star." This is rather unfair to Jan (who might just pass for Burt Reynolds in The Cannonball Run if he grew a moustache, but that's beside the point). However, this kind of response comes as no surprise to him. Transforming Torbay into an area that deserves to be called the English Riviera is not going to be an overnight task, as he acknowledges.

After picking us up from the station, Jan took us to Mulberry House, a charming little hotel and restaurant in Torquay run by Lesley Cooper. "When I'm swimming out at sea," said Lesley, "I look back at the shore and say, 'What's the South of France got on this?'" Now Lesley's establishment is very sweet, and cheap at the price, but not all Torquay is quite like Mulberry House. No doubt many fine fish suppers are served at the Abbey Sands café and The Frying Scotsman, but they have yet to trouble the Michelin Guide with their presence.

This is Jan's problem. There are delightful parts to the English Riviera. But the front is dominated by traditional seaside tat. After a glorious afternoon's sailing across the bay from Brixham harbour to Torquay, we found ourselves in a scene familiar to anyone who has ever been to an English coastal resort. Crop-haired lads with peroxide still wet from their mums' washbasins wander the town, while their older cousins, beefy and bedangled with chunky jewellery, wheel toddlers past the chippies and Mr Whippy emporia. The Princess Theatre on the seafront advertises Joe Pasquale and The Nolans, with "special guest star Jimmy Cricket".

On Sundays there are other treats familiar from those old television variety shows – Bobby Davro, Cannon and Ball, Little and Large. Their faces, a little lined now, a little heavier, smile with the forced gaiety they have worn from Great Yarmouth to Blackpool for the past 30 years. On a Sunday evening at 11.30pm a queue snaked its way impatiently down the street as teenagers energised by alcopops and the heat waited to get into The Venue, an establishment even Peter Stringfellow might hesitate to grace. Parts of Torquay are definitely more Benidorm than Biarritz.

But Jan's project is not doomed. On Sunday morning we drove through Torquay, past the caravan sites and houses called "Montana" in Paignton, and arrived at Brixham. Now we were getting closer to Riviera country. Fresh catches landed at the bustling port, their contents barely given time to stop wriggling before they were thrown on to charcoal grills and served from shacks only feet away from the shore. Two minutes away, up the winding roads of Station Hill where window boxes on either side brush sills in the middle, you really could be in a Provençal town.

Brixham is where William of Orange landed on his way to claim the throne. One local pointed out the place where the Dutchman supposedly fell off his boat and broke his nose. I checked this unusual story with a historian who, after consulting various tomes, offered the opinion that "scholars differ" on the event. True story or not, Brixham has the underdeveloped appeal of the town that has not yet been discovered. It's a proper working port, not just a tourist destination, so it feels real.

Later that afternoon, we checked out the Art in Action in Torquay's Princess Gardens, part of the town's Aquaculture festival. Close to shops selling such novelties as "Daisy's Dollop – a pat made especially for you" (black fudge in shredded green tissue) and "West Country Seagull Poo" (chocolate raisins) was a man hewing a lump of stone. The sculptor, Nigel Watson, sported an earring made of a bottle-stopper-sized hunk of cork which distended his whole lobe. Passers-by were invited to chisel away (at the stone, not the cork) while he tidied up afterwards. Gradually, something resembling an Easter Island statue emerged. The whole operation was conducted to quite creditable cover versions of "Kiss" and "Superstition" performed by Duke Jordan and the Scorchers, a local band on a stage the other side of the gardens.

On the same stage that evening was a show by The Bohemians, a Queen tribute band. The Aquaculture leaflet proudly boasts that "The Bohemians are the only Queen tribute to portray a natural resemblance to the four Queen members, so no false wigs or make-up are ever used!". Sadly, we will have to catch them another day, as we headed instead for a concert at Dartington Hall near Totnes.

Half an hour's drive from Torbay, Dartington Hall's International Summer School draws a different crowd from The Bohemians. The hall itself stands like the refectory of an Oxford college overlooking a quad, lower buildings housing earnest students either side of a verdant lawn. We had gone to see Keith Tippett, an avant garde pianist highly respected in the classical and jazz worlds. His hour of "spontaneous composition" almost defied description. Taking rolling, repeated themes at the lower end of the keyboard, he interspersed flurries of impressionist jazz, later inserting objects on to the strings so it sounded as though he had let loose into the piano first a sack of beetles, then some lonesome wasps. The audience left slowly, gradually comprehending what they had heard. It was, quite literally, stunning.

We returned, quiet still from the concert, to the Orestone Manor Hotel, a friendly gem of a place in Maidencombe, just north of Torquay. All the rooms are individually decorated and the Horsley Room, in which we stayed, is the house's original master bedroom. From our four-poster bed an expanse led to a TV and sofa area, and a window table perfect, we found, on which to play Connect Four. The teenagers queuing for the Venue seemed a million miles away from cosy Orestone Manor, which avoids the wall-to-wall carpet and formulaic service of larger hotels. As the sun streamed through the windows next morning, the sea beckoning in the bay below, while quite exceptionally good croissants were consumed, we certainly felt this deserved the title "English Riviera".

In the final few hours before we had to take the train back to London we visited another place that Jan was quite right to recommend. At Cockington Court, horse-drawn carriages bring visitors to a country house turned into a tea room and arts centre. Past an array of impressive modern sculptures produced by pupils at a local school we found the stables, where glass was blown and worked into shapes that would cost five times the price in Harvey Nicks. Then it was time to go.

Driving by the green where a grand prix – a pedal-race grand prix – had taken place at the weekend, it was immediate what a place of contrasts Torbay is, and what a gulf there is between those who visit for a week at a b&b for under £100 and those who go to Dartington. But we were shown enough to see what Torbay could be – a cheerful seaside resort creeping its way upmarket. Jan has a hard fight on his hands, and comparisons with the real Riviera are always going to be to Torbay's disadvantage. But if he has his way, the area will soon be one in which Agatha Christie's other creation, Hercule Poirot, would be happy to visit to recharge the "little grey cells".