One of my research areas is examining the role of recordkeeping and documentation in environmental regulations. A research tactic I frequently use is to sift through hundreds of PDFs at once. Large numbers of PDFs are posted on many environmental regulatory websites, but there isn’t a lot of information about what’s in them or how big they are, or where the juicy stuff is. If this sounds daunting – well, it is! But I’ve come up with a few tricks to help me sort out what is useful and what isn’t.

Step 1: Download

The first thing is figuring out how to download a zillion PDFs at once. For this, I recommend getting a bulk downloader add-on for your browser (here is an example). This will scan through every URL on a page that ends with .pdf, which indicates that the URL is likely a downloadable PDF. A bulk downloader prevents the need from clicking on every single link, which can be a lot when you’re on a page with dozens of PDFs like this one from the Mine Safety and Health Administration.

Step 2: Text Recognition

Once you download all the files you want, I like to place them into a dedicated folder. This is because even though theoretically most government agency PDFs should have optical character recognition (OCR), the actual practice is very inconsistent. OCRed text is critical for PDF searching because it allows you to do a keyword search within a single file or across multiple files at once. Currently, there is not widely-available OCR functionality for cursive or handwriting, just typeface.

Adobe Acrobat has a useful function (under Tools > Text Recognition > In Multiple Files) where you can run the OCR function across everything in a specific folder. This can take a while, but at least there is a progress bar that’ll show you how long it takes – which could be a while, considering that many government environmental regulation records can be a hundred pages for each file. Using the Adobe Acrobat tool also allows you to keep or modify original file names. I like to downsample the files to 600 dpi – it takes longer than the lower dpi measures, but I think it enables better keyword searches later on.

Step 3: Dig into the PDF files!

There is a free software program called PDF-XChange Viewer, which you can use to do keyword searches over large amounts of PDFs. You can also run a similar search within Adobe Acrobat, but I find that the searching takes far longer and the results are presented less tidily than with PDF-XChange Viewer. Supposedly you can also run batch OCR with this program, but I haven’t tried it.

The example demonstrates how I wanted to find PDFs from coal mine inspection safety reports that mention the word “map.” The results show me that of the dozens of documents I searched in my dedicated folder, there were 187 hits for the word “map” across 14 of the documents. I can get an idea from the left-hand preview pane what the keyword is like in context, and then click on that to see the actual PDF in the right-hand window.

This step helps me pull out the PDFs that I need to analyze more deeply, thus saving me the headache of opening up every single PDF in case it might have something of interest.

Survey of Federal Archives project (Ohio History Connection)

With the Green New Deal in the news, there is renewed interest in President Franklin Roosevelt’s original New Deal. The New Deal was a series of federal programs meant to combat the major issues of the Depression, including soil loss, stabilizing the banking system, and providing jobs and relief for the unemployed. One of the famous programs of the New Deal was the Works Progress Administration, popularly known as the WPA. The WPA was responsible for constructing many of the significant buildings, roads, and infrastructure that the American public still uses. But the WPA also employed many writers, musicians, playwrights, artists, historians, clerks, and other unemployed white-collar professionals.

One of the most comprehensive, but least known programs to come out of the WPA was the Historical Records Survey. Originally envisioned by American archivist TR Schellenberg, it was expanded into a workable program by Luther Evans, who would go on to become the Librarian of Congress and UNESCO director. The Historical Records Survey had two major programs: a survey of federal records located in offices outside of the Washington DC area, and a survey of state and local records. During its initial operation, the Historical Records Survey was part of the Federal Writers Project, which was known for producing travel guides for 48 states and many large cities, as well as compiling narratives of ex-slaves.

The Historical Records Survey lasted between 1935 and 1943. Its largest achievement was surveying county records – of the 3,066 counties in existence at the time of the survey, fieldwork was completed for 90% of them. WPA workers carried out the field work by going to county courts and administrative agencies to determine what kinds of records existed, where they were located, and a short description of the records. The field work also generated significant information about the history of the states and their counties. In some areas, municipal records surveys were also completed, such as for the city of Cleveland. Although there had been some attempts to survey America’s local and state records before (mainly through the efforts of the American Historical Association’s Public Archives Commission), the WPA Historical Records Survey was a significant advance in trying to establish a comprehensive picture of the overall condition of America’s public records.

Unfortunately like many worthy archival projects, the Historical Records Survey had some significant setbacks. Only 20% of county inventories were published. At least 27 county inventories were published in the state of Ohio, including for major counties such as Cuyahoga, Franklin, Hamilton, Lucas, and Summit. In the guide to the Hamilton County records, the introduction stated there would be a guide issued for every county. Perhaps the suspension of the Historical Records Survey in 1943 ended the publication of these guides. The remaining records from the Historical Records Survey of Ohio can be found at the Ohio History Connection and the Western Reserve Historical Society.

In fact, the fate of the multitude of records generated by the field workers of Historical Records Survey across the country varied wildly. Many of the records ended up in universities and state and local historical societies. But in some cases, they did not and in fact met a fairly tragic fate – when archivist and National Archives employee Leonard Rapport went searching for Maine’s Historical Records Survey field records in the 1970s, he eventually found that they had been dumped into a bay.

If you would like to learn more about the Historical Records Survey, I recommend Loretta Hefner’s 1980 guide to the unpublished records of the Historical Records Survey, and Sargent Child and Dorothy Holmes’ inventory of publications associated with the Survey (Z1223.Z7 C52, ARB Reference). For additional study, Marguerite Bloxom’s guide Pickaxe and Pencil contains an extensive bibliography arranged by WPA program.