The exe setup is created with INNO SETUP and that is a good place to go for information. Currently I don't know how to do that. Otherwise you could just delete the shortcuts after the installation.
EXE installer: on the page with the additional tasks, uncheck Create desktop icons (or only PDF24 Toolbox or PDF24 Launcher below it). Then no desktop icons are created. The start menu entry is always created by the EXE installer.
MSI package: with the property DESKTOPICONS=No no desktop icons are created. The MSI package is also the better choice for an installation without user interaction.