Aquatic toxicity part II: precision and accuracy of classifications

Thursday, December 17, 2020

by Evan Peters

INTRODUCTION

California State Toxicity is a notoriously difficult waste classification to master for retailers and waste professionals.

Classifying State Toxicity correctly can serve as a major benefit to waste generators by reducing regulated waste volumes, hauling costs, and associated fees

California’s Department of Toxic Substances Control (DTSC) has published aquatic toxicity results for several hundred consumer products. The study evaluated the products’aquatic toxicity and assigned labels of “Pass” to products that are non-toxic to the aquatic environment and “Fail” to products that are toxic to the aquatic environment.

While the traditional aquatic toxicity test is a widely accepted testing standard, it necessitates the usage of live animal testing on fish. More and more, this practice is being viewed as unethical, costly and unnecessary by manufacturers and consumers alike.

Computational methods of toxicology testing are increasingly being considered as more effective and humane alternatives to traditional methods.

To verify this claim, Smarter Sorting applied an automated toxicity calculation to the same DTSC dataset and evaluated the efficacy of our calculations in determining California aquatic toxicity.

The results are promising:

Precision = 100%
Recall = 77%
Specificity = 100%
Balanced accuracy = 89%

Keep these numbers top of mind – we’ll be addressing them soon and explaining exactly what they mean for the aquatic toxicity determination, and more importantly, for fish.

BACKGROUND

Determining State Toxicity in California requires the evaluation of many criteria, which include carcinogenicity, toxicity, flammability, and corrosivity to name a few.

The DTSC outlines the myriad ways in which a waste qualifies as a California State Toxic on their website. One of these criteria is called Acute Aquatic Toxic, where a “waste is hazardous by aquatic toxicity if a 96-hour LC50 is less than 500 mg/L.”

The DTSC does not provide guidance on the methodological evaluation of aquatic toxicity as it pertains to consumer products. Whether computational methods are permitted or not is left unanswered. The text suggests that each and every consumer product should undergo live animal testing to evaluate the aquatic toxicity and determine if the product is hazardous in California.

Live animal testing a substance for aquatic toxicity is not a trivial process. The EPA claims that each test could cost well over $14,000. These tests take time and resources that many retail businesses simply do not have.

When faced with onerous or complicated state hazardous criteria, many retailers will skip the hazardous evaluation process all together. Instead, they opt to just consider all potentially hazardous waste as hazardous. In this manner, many retailers are over-regulating their waste streams, and overpaying for expensive waste treatment in the process.

Smarter Sorting has automated the calculation of toxicity-based state regulations in the past with great success. In our Computer Bits vs. California Fish article, we demonstrated the difficulties associated with determining aquatic toxicity classifications for consumer products, and the accuracy of estimating toxicity calculations in a sample product.

We also demonstrated how Smarter Sorting’s ability to make complex state regulatory calculations typically reduces the amount of regulated waste retailers generate

For this article, we applied our aquatic toxicity methodology to a larger sample of 100 test products to determine the accuracy and viability of computerized testing as an alternative to live fish testing.

In the following analysis, the terms “non-toxic” and “pass” are synonymous and indicate aquatic toxicity values greater than 500 mg/L. Conversely, the terms “toxic” and “fail” indicate aquatic toxicity values of less than 500 mg/L.

DATA

Check out the original dataset used in our analysis.

The 100 sample products were randomized and satisfied the following criteria:

Smarter Sorting has the product Safety Data Sheet (SDS) on file
At least one of the Eurofins Calscience or Aquatic Testing labs had conducted a pass or fail study on each product

Learn more about sample products and results!

METHODOLOGY

Smarter Sorting used the Acute Toxicity Estimation (ATE) to arrive at our aquatic toxicity estimates. The chemicals and associated concentrations for each product were derived from SDSs and ingredients labels. For ingredients derived from product labels in lieu of the SDS, we deduced ranges of potential concentrations based on regulatory requirements for listing chemicals on SDSs and ingredients labels. For more information on how concentrations were estimated, refer to our previous paper on calculating aquatic toxicity

Using the calculated aquatic ATE, we assigned products with ATE values greater than 500 mg/L a score of PASS. For products with ATE values of less than 500 mg/L we assigned a score of FAIL. This threshold is based on information listed on the DTSC website. (It should be noted that the published study labels several products as FAILS despite having LC50 values greater than 500 mg/L (e.g., sample numbers 180, 220, 296, 374, 389)).

When a product received both a PASS and FAIL from our calculation and the standard lab test, we considered the product a FAIL.

RESULTS

Smarter Sorting Aquatic ATE Test vs. Traditional Lab Testing

Data Analysis Metrics

To make sense of the data, let’s first unpack the results from Smarter Sorting’s aquatic ATE test as compared to traditional lab testing:

In 47 cases, we said an item was non-toxic and the labs agreed
There were zero cases where we said an item was non-toxic and the labs disagreed
In 39 cases, we said an item was toxic and the labs agreed
In 14 cases, we said an item was toxic and the labs disagreed

Smarter Sorting Aquatic ATE Test vs. Traditional Lab Testing

Let’s break down the data science jargon to understand how these metrics are applied to their respective probabilities:

Precision of 100%: Given Smarter Sorting’s classification of Non-Toxic, the likelihood that the product is actually non-toxic is 100%.
Recall of 77%: Given a laboratory classification of Non-Toxic, the likelihood of Smarter Sorting’s classification being non-toxic is 77%.
Specificity of 100%: Given a laboratory classification of Toxic, the likelihood of Smarter Sorting’s classification being Toxic is 100%.
Balanced Accuracy refers to the accuracy of Smarter Sorting’s Toxic versus Non-Toxic calculations normalized to account for the number in each class.

One important to note is that Smarter Sorting’s calculations appear to be more conservative (more likely to indicate that a product is Toxic) than laboratory testing.

CHANGING THE STATUS QUO

By working with several retailers, Smarter Sorting has observed that regulated state waste identification remains a persistent and costly problem. The status quo for many retailers is to “play it safe” and lump nearly all non-RCRA wastes together with regulated state wastes.

In other words, retailers are over-regulating their waste. They do not have the time nor resources to evaluate every state’s hazardous waste criteria to determine if some of their wastes are non-hazardous. This is precisely where computational methods of toxicology stand to help waste professionals and retailers the most.

THE SMARTER SORTING ADVANTAGE

Smarter Sorting’s aquatic toxicity calculations detect aquatic toxicity as defined by DTSC when sufficient ingredient information is available.

Using Smarter Sorting’s aquatic toxicity calculations, complicated and burdensome waste regulations can be calculated instantly, accurately, and without harming any fish. In the process, retailers may see a dramatic decrease of up to 47% in their state regulated waste.

Leveraging Smarter Sorting’s automated classifications for California State Waste could change the way regulated state waste is classified and ultimately handled in California. Most importantly, it could eliminate the need for animal testing in the evaluation of hazardous wastes.

That’s a huge win for retailers, waste professionals, and fish everywhere.