The positions and velocities of galaxies in the Local Group (LG) measure the gravitational field within it. This is mostly due to the Milky Way (MW) and Andromeda (M31). We constrain their masses using distance and radial velocity (RV) measurements of 32 LG galaxies. To do this, we follow the trajectories of many simulated particles starting on a pure Hubble flow at redshift 9. For each observed galaxy, we obtain a trajectory which today is at the same position. Its final velocity is the model prediction for the velocity of that galaxy. Unlike previous simulations based on spherical symmetry, ours are axisymmetric and include gravity from Centaurus A. We find the total LG mass is , with 0.14±0.07 of this being in the MW. We approximately account for IC 342, M81, the Great Attractor and the Large Magellanic Cloud. No plausible set of initial conditions yields a good match to the RVs of our sample of LG galaxies. Observed RVs systematically exceed those predicted by the best-fitting Lambda Cold Dark Matter (ΛCDM) model, with a typical disagreement of km/s and a maximum of 110±13 km/s for DDO 99. Interactions between LG dwarf galaxies cannot easily explain this. One possibility is a past close flyby of the MW and M31. This arises in some modified gravity theories but not in ΛCDM. Gravitational slingshot encounters of material in the LG with either of these massive fast-moving galaxies could plausibly explain why some non-satellite LG galaxies are moving away from us even faster than a pure Hubble flow.