Big Data Engineer Interview Questions for AI Training Work

The decision hinges on latency requirements and data volume patterns. Batch frameworks handle huge volumes efficiently when hourly or daily freshness is acceptable. Stream frameworks add operational complexity but are necessary when downstream systems need sub-minute freshness, like fraud detection. I avoid defaulting to streaming just because it's more capable, since it costs more to build and operate than batch.

Skew usually shows up as one or two partitions taking far longer than the rest. I address it by salting the skewed key to spread it across more partitions, or by isolating and processing the heavy keys separately from the rest of the dataset. Repartitioning based on a better key, rather than the default hash, often prevents the skew from happening at all.

True exactly-once is hard to guarantee end to end, so I aim for effectively-once by combining idempotent writes with offset tracking that only commits after a successful write. If the write itself is idempotent, like an upsert keyed on message ID, reprocessing after a failure doesn't create duplicates even though the underlying delivery is at-least-once.

I design for elastic scaling rather than provisioning for peak load at all times, since that wastes resources most of the time. Autoscaling policies based on queue depth or processing lag, combined with backpressure so the system degrades gracefully instead of falling over under a spike, handle unpredictability better than static overprovisioning.

I keep a single canonical event schema at the source and let real-time and batch consumers read from the same underlying stream, with the batch path simply materializing periodic snapshots of that stream. Maintaining two separate schemas for the same underlying data creates drift risk and doubles the maintenance burden for no real benefit.

I'd check for data skew first, since a gradual slowdown without an error usually means one partition's workload grew disproportionately. I'd also check cluster resource utilization for contention with other jobs and confirm the input data volume hasn't grown unexpectedly. Comparing execution plans between a fast run and a slow run usually narrows it down quickly.

I'd run the new pipeline in parallel against the same input stream, writing to a separate output location, and compare results against the existing pipeline before cutting traffic over. Once outputs match consistently over a meaningful window, I'd switch consumers to the new pipeline's output and only then decommission the old one.

I'd assume existing partitioning and cluster sizing won't hold and start from a capacity estimate based on the new source's actual throughput characteristics, not just scaling current numbers linearly. I would also test whether downstream consumers, not just the ingestion layer, can handle the increased volume before the new source goes live.

I initially chose a single shared queue for all event types, which worked fine in testing but became a bottleneck once volume from one high-traffic event type started delaying everything else. I redesigned it with per-event-type partitioning, and the lesson was to load-test with realistic, skewed traffic patterns rather than uniform synthetic data before committing to an architecture.

A stakeholder wanted near-real-time processing for a use case that, on closer inspection, only needed data within a few minutes rather than seconds. I pushed back on the streaming architecture in favor of frequent micro-batches, which met the actual requirement with far less operational complexity than a full streaming system would have needed.