The many similarities between the prompt emission pulses in gamma-ray bursts (GRBs) and X-ray flares during the fast decay and afterglow (AG) phases of GRBs suggest a common origin. In the cannonball (CB) model of GRBs, this common origin is mass accretion episodes of fall-back matter on a newly born compact object. The prompt emission pulses are produced by a bipolar jet of highly relativistic plasmoids (CBs) ejected in the early, major episodes of mass accretion. As the accretion material is consumed, one may expect the engine's activity to weaken. X-ray flares ending the prompt emission and during the AG phase are produced in such delayed episodes of mass accretion. The common engine, environment, and radiation mechanisms (inverse Compton scattering and synchrotron radiation) produce their observed similarities. Flares in both long GRBs and short hard gamma-ray bursts (SHBs) can also be produced by bipolar ejections of CBs following a phase transition in compact objects due to loss of angular momentum and/or cooling. Optical flares, however, are mostly produced in collisions of CBs with massive stellar winds/ejecta or with density bumps along their path. In this paper, we show that the master formulae of the CB model of GRBs and SHBs, which reproduce very well their prompt emission pulses and their smooth AGs, seem to reproduce also very well the light curves and spectral evolution of the prominent X-ray and optical flares that are well sampled.