AI Engineer Interview Questions for AI Training Work

Accuracy is misleading on imbalanced data because a model can score well by always predicting the majority class. I look at precision, recall, and F1 for the minority class specifically, and pick the metric that matches the real cost of false positives versus false negatives in the deployment context, since those costs are rarely symmetric in practice.

I start by checking for a train/inference mismatch: different preprocessing, batch normalization behaving differently in eval mode, or data leakage during training that inflated validation scores. Comparing the exact input pipeline used at training time against the one used at inference usually surfaces the discrepancy before I need to touch the model architecture itself.

I start with correlation and mutual information analysis to drop obviously redundant features, then use a tree-based model's feature importance as a first pass filter. From there, recursive feature elimination with cross-validation tells me where the marginal value of additional features drops off, which keeps the final feature set both smaller and more robust to overfitting.

I monitor the statistical distribution of input features and model outputs over time, using something like population stability index or KL divergence against a reference window. When drift crosses a threshold, I retrain on recent data rather than the full historical set, and I keep an alert in place so drift is caught before it shows up as a quality regression.

Overfitting shows a growing gap between training and validation performance as training continues. Data leakage shows suspiciously high validation performance from the start, often near-perfect, because information from the target has leaked into the features. I check for leakage first, since it is the more common cause of an unrealistically good result, before assuming the model is simply overfit.

I would quantify the noise level first by having a subset of labels independently reviewed, since the right strategy depends heavily on how bad the noise actually is. Light noise, models trained with robust loss functions tolerate reasonably well. Heavy noise usually means the labeling process itself needs fixing before more modeling effort is worth investing.

I would first confirm the production input distribution matches the offline evaluation set, since a mismatch there explains most gaps of this kind. Next I'd check for pipeline bugs, like a feature computed differently in production than in training. Only after ruling those out would I suspect the model itself is genuinely worse than the offline numbers suggested.

Without labels, I'd first look for a proxy signal that correlates with the real objective, or consider a weak supervision approach using heuristics to bootstrap initial labels. Unsupervised or self-supervised methods are another path if the goal is representation learning rather than a specific prediction task. The key is being explicit that any evaluation without ground truth is inherently approximate.

I laid out the analysis step by step rather than leading with the conclusion, so the stakeholder could follow the reasoning and see where their assumption broke down. Presenting the counter-evidence transparently, with the underlying data visible, made the disagreement about the data rather than about who was right, which made it easier to reach agreement on next steps.

On a real-time system, I traded a few points of accuracy for a smaller model architecture once profiling showed the larger model missed the latency budget under peak load. I validated that the accuracy drop didn't cross a threshold that mattered for the use case before making the tradeoff, since a small quality loss is only acceptable if it stays below what users actually notice.