The great challenge when writing a memory novel – in which a narrative of now frames or entwines with a narrative of then – is that both the present and historic storylines need to be equally compelling. Each must have its internal logic, its believable characters and a denouement that is emotionally satisfying. All too often in such novels the account of the past carries a greater charge than that of the present; inevitably, perhaps, since there is usually some horror festering in the past narrative's most shadowy corners. As in many biographies, accounts of childhood have an awkward way of making adult material seem bland or predictable by comparison.

Trezza Azzopardi's third book is a memory novel with bells on: she shows two major and two minor characters drawn to pick apart long-ago events which have left all of them spiritually scarred. Kenneth Earl, a wealthy "sixty-several" music lover, is facing the early onset of something like senility.

For all that he is still a dapper dresser, feeds himself and manages a rambling country house on his own, his memory is going and with it his grasp on day-to-day practicalities. Without informing his bossily protective son or would-be girlfriend, he advertises for someone to help him annotate his vast LP collection to remind him why each record is there, why he bought it, what associations it carries for him.

Maggie, the young woman who lands the job, has been to the house before, when she was four years old. Something terrible happened to her there.

She manages well at first, parrying Kenneth's flirtatious sallies, challenging his moments of emotional cowardice and encouraging him in the sort of eccentricity his son would abhor. She works on his notes but also begins some writing of her own, using the exercise book and expensive fountain pen he gives her to write what becomes, in effect, the novel within the novel.

This tells of her bucolic, unconventional girlhood as the unexpected daughter in a hippie commune started in a nearby riverside cottage by her father, a judge's drop-out "artist" son. Her mother and her real and surrogate fathers teach her songs and the names of wild flowers, and her days pass in a more or less blissful haze of sunshine, cannabis smoke, bad art and naturism, gawped at by the sinister "river man" and his young assistant. Which is where the nasty bit comes in, and where she flees the house just as something like love was growing between her and her employer.

Azzopardi is a good sensual writer, especially strong on smells good and bad and their emotional colours, and at evoking sensations like river mud between one's toes. However, this novel is a house built on shaky foundations. If what should be an overwhelming story of trauma confronted and healed touches the heart less than it should it's because the premise – Kenneth and his wish to annotate a record collection – feels willed and artificial. The man at its heart lacks the kind of emotional context that makes a reader identify and care.

Patrick Gale's 'Gentleman's Relish' is published by Fourth Estate