Legislators, Stockholders, Civil Right Groups, and a CEO Seek Limits on AI Face Recognition Technology

Following the tragic killings of journalists and staff inside the Capital Gazette offices in Annapolis, Maryland, in late June, local police acknowledged that the alleged shooter’s identity was determined using a facial recognition technology widely deployed by Maryland law enforcement personnel.  According to DataWorks Plus, the company contracted to support the Maryland Image Repository System (MIRS) used by Anne Arundel County Police in its investigation, its technology uses face templates derived from facial landmark points extracted from image face data to digitally compare faces to a large database of known faces.  More recent technology, relying on artificial intelligence models, have led to even better and faster image and video analysis used by federal and state law enforcement for facial recognition purposes.  AI-based models can process images and video captured by personal smartphones, laptops, home or business surveillance cameras, drones, and government surveillance cameras, including body-worn cameras used by law enforcement personnel, making it much easier to remotely identify and track objects and people in near-real time.

Recently, facial recognition use cases have led to privacy and civil liberties groups to speak out about potential abuses, with a growing vocal backlash aimed at body-worn cameras and facial recognition technology used in law enforcement surveillance.  Much of the concern centers around the lack of transparency in the use of the technology, potential issues of bias, and the effectiveness of the technology itself.  This has spurred state legislators in several states to seek to impose oversight, transparency, accountability, and other limitations on the tech’s uses.  Some within the tech industry itself have even gone so far as to place self-imposed limits on uses of their software for face data collection and surveillance activities.

Maryland and California are examples of two states whose legislators have targeted law enforcement’s use of facial recognition in surveillance.  In California, state legislators took a recent step toward regulating the technology when SB-1186 was passed by its Senate on May 25, 2018.  In remarks accompanying the bill, legislators concluded that “decisions about whether to use ‘surveillance technology’ for data collection and how to use and store the information collected should not be made by the agencies that would operate the technology, but by the elected bodies that are directly accountable to the residents in their communities who should also have opportunities to review the decision of whether or not to use surveillance technologies.”

If enacted, the California law would require, beginning July 1, 2019, law enforcement to submit a proposed Surveillance Use Policy to an elected governing body, made available to the public, to obtain approval for the use of specific surveillance technologies and the information collected by those technologies.  “Surveillance technology” is defined in the bill to include any electronic device or system with the capacity to monitor and collect audio, visual, locational, thermal, or similar information on any individual or group. This includes, drones with cameras or monitoring capabilities, automated license plate recognition systems, closed-circuit cameras/televisions, International Mobile Subscriber Identity (IMSI) trackers, global positioning system (GPS) technology, software designed to monitor social media services or forecast criminal activity or criminality, radio frequency identification (RFID) technology, body-worn cameras, biometric identification hardware or software, and facial recognition hardware or software.

The bill would prohibit a law enforcement agency from selling, sharing, or transferring information gathered by surveillance technology, except to another law enforcement agency. The bill would provide that any person could bring an action for injunctive relief to prevent a violation of the law and, if successful, could recover reasonable attorney’s fees and costs.  The bill would also establish procedures to ensure that the collection, use, maintenance, sharing, and dissemination of information or data collected with surveillance technology is consistent with respect for individual privacy and civil liberties, and that any approved policy be publicly available on the approved agency’s Internet web site.

With the relatively slow pace of legislative action, at least compared to the speed at which face recognition technology is advancing, some within the tech community have taken matters into their own hands.  Brian Brakeen, for example, CEO of Miami-based facial recognition software company Kairos, recently decided that his company’s AI software will not be made available to any government, “be it America or another nation’s.”  In a TechCrunch opinion published June 24, 2018, Brakeen said, “Whether or not you believe government surveillance is okay using commercial facial recognition in law enforcement is irresponsible and dangerous” because it “opens the door for gross misconduct by the morally corrupt.”  His position is rooted in the knowledge of how advanced AI models like his are created: “[Facial recognition] software is only as smart as the information it’s fed; if that’s predominantly images of, for example, African Americans that are ‘suspect,’ it could quickly learn to simply classify the black man as a categorized threat.”

Kairos is not alone in calling for limits.  A coalition of organizations against facial recognition surveillance published a letter on May 22, 2018, to Amazon’s CEO, Jeff Bezos, in which the signatories demanded that “Amazon stop powering a government surveillance infrastructure that poses a grave threat to customers and communities across the country. Amazon should not be in the business of providing surveillance systems like Rekognition to the government.”  The organizations–civil liberties, academic, religious, and others–alleged that “Amazon Rekognition is primed for abuse in the hands of governments. This product poses a grave threat to communities,” they wrote, “including people of color and immigrants….”

Amazon’s Rekognition system, first announced in late 2016., is a cloud-based platform for performing image and video analysis without the user needing a background in machine learning, a type of AI.  Among its many uses today, Rekognition reportedly allows a user to conduct near real-time automated face recognition, analysis, and face comparisons (assessing the likelihood that faces in different images are the same person), using machine learning models.

A few weeks after the coalition letter dropped, another group, this one a collection of individual and organizational Amazon shareholders, issued a similar letter to Bezos.  In it, the shareholders alleged that “[w]hile Rekognition may be intended to enhance some law enforcement activities, we are deeply concerned it may ultimately violate civil and human rights.”  Several Microsoft employees took a similar stand against Microsoft’s role in its software used by government agencies.

As long as questions surrounding transparency, accountability, and fairness in the use of face recognition technology in law enforcement continue to be raised, tech companies, legislators, and stakeholders will likely continue to react in ways that address immediate concerns.  This may prove effective in the short-term, but no one today can say what AI-based facial detection and recognition technologies will look like in the future or to what extent the technology will be used by law enforcement personnel.

New York City Task Force to Consider Algorithmic Harm

One might hear discussions about backpropagation, activation functions, and gradient descent when visiting an artificial intelligence company. But more recently, terms like bias and harm associated with AI models and products have entered tech’s vernacular. These issues also have the attention of many outside of the tech world following reports of AI systems performing better for some users than for others when making life-altering decisions about prison sentences, creditworthiness, and job hiring, among others.

Considering the recent number of accepted conference papers about algorithmic bias, AI technologists, ethicists, and lawyers seems to be proactively addressing the issue by sharing with each other various technical and other solutions. At the same time, at least one legislative body–the New York City Council–has decided to explore ways to regulate AI technology with an unstated goal of rooting out bias (or at least revealing its presence) by making AI systems more transparent.

New York City’s passage of the “Automated decision systems used by agencies” law (NYC Local Law No. 49 of 2018, effective January 11, 2018), creates a task force under the aegis of Mayor de Blasio’s office. The task force will convene no later than early May 2018 for the purpose of identifying automated decision systems used by New York City government agencies, developing procedures for identifying and remedying harm, developing a process for public review, and assessing the feasibility of archiving automated decision systems and relevant data.

The law defines an “automated decision system” as:

“a computerized implementations of algorithms, including those derived from machine learning or other data processing or artificial intelligence techniques, which are used to make or assist in making decisions.”

The law defines an “agency automated decision system” as:

“an automated decision system used by an agency to make or assist in making decisions concerning rules, policies or actions implemented that impact the public.”

While the law does not specifically call out bias, the source of algorithmic unfairness and harm can be traced in large part to biases in the data used to train algorithmic models. Data can be inherently biased when it reflects the implicit values of a limited number of people involved in its collection and labelling, or when the data chosen for a project does not represent a full cross-section of society (which is partly the result of copyright and other restrictions on access to proprietary data sets, and the ease of access to older or limited data sets where groups of people may be unrepresented or underrepresented). A machine algorithm trained on this data will “learn” the biases, and can perpetuate bias when it is asked to make decisions.

Some argue that making algorithmic black boxes more transparent is key to understanding whether an algorithm is perpetuating bias. The New York City task force could recommend that software companies that provide automated decision systems to New York City agencies make their systems transparent by disclosing details about their models (including source code) and producing the data used to create their models.

Several stakeholders have already expressed concerns about disclosing algorithms and data to regulators. What local agency, for example, would have the resources to evaluate complex AI software systems? And how will source code and data, which may embody trade secrets and include personal information, be safeguarded from inadvertent public disclosure? And what recourse will model developers have before agencies turn over algorithms (and the underlying source code and data) in response to Freedom of Information requests and court-issued subpoenas?

Others have expressed concerns that regulating at the local level may lead to disparate and varying standards and requirements, placing a huge burden on companies. For example, New York City may impose standards different from those imposed by other local governments. Already, companies are having to deal with different state regulations governing AI-infused autonomous vehicles, and will soon have to contend with European Union regulations concerning algorithmic data (GDPR Art. 22; effective May 2018) that may be different than those imposed locally.

Before their job is done, New York City’s task force will likely hear from many stakeholders, each with their own special interests. In the end, the task force’s recommendations, especially those on how to remedy harm, will receive careful scrutiny, and not just by local stakeholders, but also by policymakers far removed from New York City, because as AI technology impacts on society grow, the pressure to regulate AI systems on a national basis is likely to grow.

Information and/or references used for this post came from the following:

NYC Local Law No. 49 of 2018 (available at here) and various hearing transcripts

Letter to Mayor Bill de Blasio, Jan. 22, 2018, from AI Now and others (available here)

EU General Data Protection Regulations (GDPR), Art. 22 (“Automated Individual Decision-Making, Including Profiling”), effective May 2018.

Dixon et. al “Measuring and Mitigating Unintended Bias in Text Classification”; AAAI 2018 (accepted paper).

W. Wallach and G. Marchant, “An Agile Ethical/Legal Model for the International and National Governance of AI and Robotics”; AAAI 2018 (accepted paper).

D. Tobey, “Software Malpractice in the Age of AI: A Guide for the Wary Tech Company”; AAAI 2018 (accepted paper).