Hearthstone stood with no maintence in the years after 1995. Local students often broke in to explore the castle and use it as a place to hang out and party. Graffitti soon covered the walls and bottles littered the ground. Initiatives to restore the castle cropped up every few years, but just like decades before, nothing ever got accomplished. At one point a chain-link fence was put up to keep people out of the castle, but soon after it was installed a hole was cut in it.
By 2008 the majority of the roof had collapsed, completely exposing the castle to the elements. The interior then deteriorated rapidly, and the castle was almost entirely rubble.
In 2016 it was determined that the cost to repair the castle would be over $15 million, which was more than the city could ask the taxpayers to cover. A plan was devised to clear the castle of rubble, take the walls down to about 15 feet, and turn the property into a walled garden.