That AI chatbot I trained on old support tickets went totally rogue...

I was sitting in my home office in Portland last Tuesday testing a custom chatbot I built on 2 years of support logs and it started giving customers completely wrong refund policies... turns out the training data had a bug that mixed up the 2019 and 2023 return windows. How do you guys catch data rot before it messes up your models?

2 comments

2 Comments

wilson.jana8d ago

I heard someone at a meetup say they randomly sample old tickets every quarter and compare the responses to current policy docs. That would have caught your date mixup before it went live, because the model would be referencing 2019 info on a 2023 question. It's a pain to do manually but beats having customers screaming at your team.

walker.alex8d ago

Ugh that's brutal. Nothing worse than debugging a live model while customers are already seeing the wrong stuff.

-1