@textfiles @internetarchive ok that helps. How do you get to archived sites that have robots.txt changed to noindex in 2014?