By methods similar to those used by L. Jeffrey [L. C. Jeffrey, Chern-Simons-Witten invariants of lens spaces and torus bundles, and the semiclassical approximation, Commun. Math. Phys. 147 (1992) 563-604], we compute the quantum SU(N)-invariants for mapping tori of trace 2 homeomorphisms of a genus 1 surface when N = 2, 3 and discuss their asymptotics. In particular, we obtain directly a proof of a version of Witten's asymptotic expansion conjecture for these 3-manifolds. We further prove the growth rate conjecture for these 3-manifolds in the SU(2) case, where we also allow the 3-manifolds to contain certain knots. In this case we also discuss trace -2 homeomorphisms, obtaining - in combination with Jeffrey's results - a proof of the asymptotic expansion conjecture for all torus bundles.