Our goal at SmarterX is to be the AI-powered engine to make any CPG decision. We’re focused on building out tooling and AI models that eclipse human performance and allow you to work faster and more meaningfully in your day-to-day. It’s critical for our partners to understand our approach to building and testing our models and rulesets. We’re going to use our ‘in scope’ rule to help explain our approach.
The Official Process
To ensure a smarter-1 decision is production-ready, we follow a stringent process:
- Ruleset Development: Rulesets are developed and refined using base LLM models, achieving an F1 score greater than 0.95 on testing and validation datasets.
- Fine-Tuning: A fine-tuned model is trained based on the developed ruleset and must also achieve an F1 score greater than 0.95.
- Independent Smoke Testing: A statistically significant set of Golden Records, not used in training or validation, is generated and tested against the fine-tuned model. The model must score an F1 greater than 0.95 to be accepted.
- Production Deployment: Once all criteria are met, the decision models are deployed to our production Smarter-1 APIs for customer use.
If you want to better understand F1 scores, read our blog post here.
Understanding IS_IN_SCOPE
IS_IN_SCOPE is a critical decision model that evaluates products to determine if they fall within specific regulatory scopes or specific requirements you may set. For regulatory classifications, it classifies products into subtypes such as battery, lightbulb, refrigerant, and formulated products. The model then produces a final IS_IN_SCOPE=true or false decision.
The Results
- High Accuracy: The fine-tuned model achieved an impressive F1 score of 0.97, demonstrating exceptional accuracy.
- Extensive Validation: We validated 854 product payloads, ensuring the predicted classifications met our rigorous expert standards.
- Robust Decision Framework: The IS_IN_SCOPE decision identifies whether a product falls into one of four subtypes—battery, lightbulb, refrigerant, and formulated—and determines an overall IS_IN_SCOPE status.
Why This Matters
- Unmatched Accuracy and Reliability: Our models undergo rigorous testing and validation to ensure they deliver highly accurate and reliable results. This accuracy translates to better regulatory compliance and reduced risk for your business.
- Expert-Validated Data: Our models are trained and validated using expert-validated data, ensuring the highest standards of quality and relevance. This means you can trust the classifications and decisions made by our system.
- Comprehensive Testing Process: The decision models are put through a robust testing process, including the use of Golden Records, which are independent test sets not used in training. This process guarantees that our models can generalize well to new, unseen data.
- Industry Leadership: By adhering to best practices in machine learning and statistical analysis, we set the standard for transparency, explainability, and reproducibility in AI-driven decision making. Our commitment to rigorous testing and validation processes ensures that we lead the industry in delivering dependable AI solutions.
This process ensures that our models are not only accurate but also transparent and explainable. Each step involves a handoff between teams and techniques, preventing any potential biases or errors. This separation of concerns is a standard best practice in AI and machine learning, ensuring reproducibility and reliability of our models.
At SmarterX, we are dedicated to delivering the highest standards of AI-driven decision-making. By choosing smarter-1 decisions, you are choosing industry-leading expertise and commitment to quality.
.png)