Machine Learning Engineer - Evals (US)

How do you choose between a simpler interpretable model and a more complex model that performs marginally better?

The decision depends on how the model's output is used downstream. If a human needs to act on and justify individual predictions, like in lending or healthcare, interpretability often outweighs a small accuracy gain. If the model feeds an automated system where explainability isn't a hard requirement, I'll take the complexity if the performance gain is meaningful and validated, not just noise.

Explain how you would set up a proper train, validation, and test split for a time series problem.

Random splitting leaks future information into training for time series, so I split chronologically instead, training on the earliest period, validating on the next, and testing on the most recent. Any cross-validation also needs to respect time order, using something like rolling-origin validation, rather than standard k-fold, which would otherwise let the model see the future during validation.

What statistical test would you use to determine if an A/B test result is significant, and what assumptions does it rely on?

For a conversion-rate comparison, a two-proportion z-test or chi-squared test is standard, and it assumes independent observations and a large enough sample size for the normal approximation to hold. If the sample is small or the metric isn't binary, I'd switch to a more appropriate test, since applying the wrong test's assumptions is a common source of false confidence in A/B results.

How do you approach hyperparameter tuning efficiently when the model is expensive to train?

I use a coarse random search first to identify the promising region of the hyperparameter space, since grid search wastes compute on combinations unlikely to matter. From there, Bayesian optimization narrows in on the best configuration with far fewer expensive training runs than an exhaustive search would require, which matters a lot when each run is costly.