During the production of milk chocolate, its sensory impression can vary greatly depending on the raw materials, processing parameters and storage conditions applied. In most parts of Europe, a neutral fresh-milk note is preferred in milk chocolates, while a distinct caramel note is preferred in the USA and Great Britain. It is already known from practical experience that sensory problems can arise during the production and processing of milk chocolate with a fresh, milky impression. An unwanted caramel aroma and grittiness in the mouthfeel have been reported to occur. To date, the influence of the type of milk powder used (spray or roller dried, fat content, water content, and physical state of lactose) in combination with the influence of conching parameters on the sensory properties and behavior during storage of the chocolate has not been clarified. Therefore, the statistical design of experiment was applied to produce a series of different milk chocolates by using different spray-dried milk powders and conching parameters. The freshly produced chocolates were then characterized physically, aroma-analytically and by human sensory evaluation. The amorphous lactose content, bound fat content, water content, as well as infinite viscosity and yield point of the different chocolates were determined. Marker substances and the organic volatile compound (VOC) profile of the different chocolates were evaluated by gas chromatography-ion mobility spectrometry. Sensory evaluations of selected chocolates were performed by sorted napping. Generally, the results showed a clear influence of the milk powders and the manufacturing parameters used on the physical characteristics, the VOC profiles and the sensory fresh milk impression of the chocolates.