Thanks for the huge learning! Curious does any chatter for eval tools involve measuring financial cost as well? Considering the details of performance and model quality, not sure if there some tinkering there for output of efficiency with cost as well.
This is a great question. We track financial cost, but it's always an interesting calculus between cost, quality and how much a given improvement impacts the acquisition/retention funnel. The first two are know-able when you're doing evals, but the ultimate impact is only knowable by shipping.
Thanks for the huge learning! Curious does any chatter for eval tools involve measuring financial cost as well? Considering the details of performance and model quality, not sure if there some tinkering there for output of efficiency with cost as well.
This is a great question. We track financial cost, but it's always an interesting calculus between cost, quality and how much a given improvement impacts the acquisition/retention funnel. The first two are know-able when you're doing evals, but the ultimate impact is only knowable by shipping.