KIC 8462852 is a superficially ordinary main sequence F star for which Kepler detected an unusual series of brief dimming events. We obtain accurate relative photometry of KIC 8462852 from the Kepler full-frame images, finding that the brightness of KIC 8462852 monotonically decreased over the four years it was observed by Kepler. Over the first ∼1000 days KIC 8462852 faded approximately linearly at a rate of 0.341 ± 0.041% yr–1, for a total decline of 0.9%. KIC 8462852 then dimmed much more rapidly in the next ∼200 days, with its flux dropping by more than 2%. For the final ∼200 days of Kepler photometry the magnitude remained approximately constant, although the data are also consistent with the decline rate measured for the first 2.7 years. Of a sample of 193 nearby comparison stars and 355 stars with similar stellar parameters, none exhibit the rapid decline by >2% or the cumulative fading by 3% of KIC 8462852. Moreover, of these comparison stars, only one changes brightness as quickly as the 0.341% yr–1 measured for KIC 8462852 during the first three years of the Kepler mission. We examine whether the rapid decline could be caused by a cloud of transiting circumstellar material, finding that while such a cloud could evade detection in submillimeter observations, the transit ingress and duration cannot be explained by a simple cloud model. Moreover, this model cannot account for the observed longer-term dimming. No known or proposed stellar phenomena can fully explain all aspects of the observed light curve.