2012A&A...546A..38L

HR4796 is a young, early A-type star harboring a well structured debris disk, shaped as a ring with sharp inner edges. The inner edge might be shaped by a yet unseen planet inside the ring; the outer one is not well understood. The star forms together with the M-type star HR4796B, a binary system, with a projected separation of ≃560AU. Our aim is to explore the surroundings of HR4796A and B, both in terms of extended or point-like structures. Adaptive optics images at L'-band were obtained with NaCo in Angular Differential Mode and with Sparse Aperture Masking (SAM). We analyse the data as well as the artefacts that can be produced by ADI reduction on an extended structure with a shape similar to that of HR4796A dust ring. We determine constraints on the presence of companions using SAM and ADI on HR4796A, and ADI on HR4796B. We also performed dynamical simulations of a disk of planetesimals and dust produced by collisions, perturbed by a planet located close to the disk outer edge. The disk ring around HR4796A is well resolved. We highlight the potential effects of ADI reduction of the observed disk shape and surface brightness distribution, and side-to-side asymmetries. We produce 2D maps of planet detection limits. No planet is detected around the star, with masses as low as 3.5M_Jup at 0.5'' (58AU) and less than 3M_Jup in the 0.8-1'' range along the semi-major axis. We exclude massive brown dwarfs at separations as close as 60mas (4.5AU) from the star thanks to SAM data. The detection limits obtained allow us to exclude a possible close companion to HR4796A as the origin of the offset of the ring center with respect to the star; they also allow to put interesting constraints on the (mass, separation) of any planet possibly responsible for the inner disk steep edge. Using detailed dynamical simulations, we show that a giant planet orbiting outside the ring could sharpen the disk outer edge and reproduce the STIS images published by Schneider et al. (2009). Finally, no planets are detected around HR4796B with limits well below 1M_Jup at 0.5'' (35AU).