Add the water, rosemary, and sugar to a pot over medium heat and bring to a boil. Stir to dissolve the sugar.
Cut an X into the bottom of each peach, just cutting through the skin. Add the peaches into the boiling water.
Let the peaches boil for 5 minutes, or until they are fork tender. Test with a fork making sure they don't become too mushy.
Remove the peaches from the water and allow them to cool.
Use a knife to peel the skin from the peaches. They should peel easily.
Cut the peaches in half, twisting to separate them. Remove the pits.
Add the raspberries to a food processor along with the sugar if using and blend until they are a pureed.
Pour the puree into a fine mesh sieve. Use a spatula to force the puree through and remove the seeds.
To plate, drizzle some of the raspberry puree onto a plate or serving glass. Top with the peaches. Add a scoop or two of vanilla ice cream. Sprinkle sliced almonds over the top and a pinch of black pepper if desired.