Steven Kang
Research Mentor: Devin Judge-Lord
Mentor Department: Not Available, Public Policy
Author(s): Steven Kang, Devin Judge-Lord
Session: Session 3 (11:00 AM – 11:50 AM)
Presentation Type: Poster 70
Abstract
Large-scale datasets of corporations are often used by social scientists to study corporate behavior, such as lobbying. However, these datasets can contain ambiguous or conflicting organization names, creating a major obstacle to linking records across sources and finding these entities in unstructured text. This project aims to simplify research on corporate behavior by developing a software package that will allow users to easily identify corporation names in unstructured text, where they may go by different names and there are no unique identifiers. The first step was to build an extensive crosswalk lookup table of corporations with an automated pipeline to match organizations that go by different names in large datasets of companies, which ranged in size from 5,000 to over 800,000 records. These datasets sometimes contained missing values for certain columns, duplicated records with different names (but the same unique identifier), and they each had different columns as well. Then, several string similarity–matching techniques were implemented and evaluated using Python, particularly comparing a previously used Damerau–Levenshtein distance–based method against the RapidFuzz library’s Token Set Ratio algorithm. To create the “corporations†package, an existing package called “regextableâ€, which extracts regular-expression-based pattern matches, was used as a dependency to match input data from a user with corporations from the crosswalk table. Two modes were implemented in the extract() function of the “corporations†package, allowing a user to either match entities directly against the crosswalk table or first identify entities within longer text before performing the matching step. The result of this corporations package allows anyone to quickly identify corporations within different kinds of text, reducing the time required for manual searching and improving the reliability of analyses involving corporate entities.
