Using femtosecond upconversion we investigate the time and wavelength structure of infrared supercontinuum generation. It is shown that radiation is scattered into higher order spatial modes (HOMs) when generating a supercontinuum using fibers that are not single-moded, such as a step-index ZBLAN fiber. As a consequence of intermodal scattering and the difference in group velocity for the modes, the supercontinuum splits up spatially and temporally. Experimental results indicate that a significant part of the radiation propagates in HOMs. Conventional simulations of supercontinuum generation do not include scattering into HOMs, and including this provides an extra degree of freedom for tailoring supercontinuum sources.