This is one of a series of posts revolving around debugging stories from the field. The goal of this series is to help demonstrate how to...

[...] focus on best practices such as when and why you should use Oozie, and when to use bundles.

Blog post chronicling the investigation of a series of errors in a Spark Streaming job reading from Spark.

A team at a client was using Spark to read and write to a Kafka topic. [...] files that would be written that were completely empty.

... we're going to discuss [...] the common excuse I see from developers to immediately try to increase resources, [the OutOfMemory error]

This week, we're going to talk about executor cores. First [...] we'll understand how setting executor cores affects our jobs...

[This week,] we'll look at the overhead memory parameter, which is available for both driver and executors.

The most common misconception I see developers fall into with regards to the driver configuration is increasing driver memory.

Increasing the number of executors certainly works [...], but there are still very important times when it doesn't work the way you'd want.

Not only is [increasing memory] haphazard and leads to inconsistent results, but it also doesn't actually do what they think it does.

Optimizing a Spark job can be a daunting task. [...] This series is going to focus on diving into the inner works of Spark [...]

If you've worked with Hadoop for any amount of time, you've likely run into the dreaded small file problem...

[...] we'll discuss some of the ways we can write environment agnostic code, which can be run on any environment within your enterprise.

We will discuss how partitions play with YARN queues. Finally, we will return to the example given in the first part of this series.

How do we ensure that GPU jobs run on worker nodes with GPUs without buying expensive GPUs for all of our worker nodes?

I'm going to explore exactly how YARN works with queues, and the various mechanisms available to control how YARN does this.

As this case shows, it is very important to understand where to look and how to filter out noise from the signal you care about.