For the custard
Spread one half of the sponge with raspberry jam and the other with apricot jam. Cut each half into four, and put in a dish in which they will just fit, alternating raspberry with apricot. Pour over the brandy and sweet sherry, and leave to macerate for 2 hours.
To make the custard, whisk the egg yolks with the caster sugar until creamy. Bring the milk to the boil. Pour it over the egg yolks, whisking constantly, then put the bowl over a pan of simmering water and whisk until thick. Stir in the vanilla essence. Pour over the sponge and refrigerate overnight.
To serve, whip the cream until stiff, and spoon over the top.