What does the infrastructure stack (routing, evals, versioning, observability) look like for running hundreds of specialized models in production?

Source topic

South Park Commons poses this as an open question for exploration. It appears within the broader discussion of How do we help every enterprise run its own fleet of models.