Put the sponge cubes in a large serving bowl. Combine the jam, brandy and 50ml/2fl oz water, and sprinkle over the sponge.
Add the jelly crystals to the boiling water, and stir until dissolved. Pour into a rectangular dish and refrigerate until set.
Cut the jelly into cubes. Scatter over the sponge. Peel and slice the bananas. Top the jelly with the sliced bananas and the custard.
Decorate the trifle with the whipped cream, almonds and passion fruit. Chill until needed.