Really expert in HtmlUnit and other Web TestUnits - Error checking/debug and finalization of a grab engine

I have a JAVA engine that utilize HtmlUnit to grab files (swf, flv, jpg, gif) and content/infos of javascript fragmentss of files inside or called by web pages. I need to render these pages in order to can grab this content and save locally (folder for files, records for infos). Actually the engine stops in certain cases due maybe to some HtmlUnit bug, maybe to some not well coded routines, maybe for some not knowed problems. Actually I have around 15000 urls to cheeck and debug to can run this engine 24h a day with NO ERROR. I have very short time to run a test.. and suggestion about how to solve the erros problems.