Blog post chronicling the investigation of a series of errors in a Spark Streaming job reading from Spark.

A team at a client was using Spark to read and write to a Kafka topic. [...] files that would be written that were completely empty.

We made the change on a Sunday, but four days later, the number of files had not appreciably changed in the YARN logs directory.

This client didn't use the activity explorer much [...] I tried to run one of the paragraphs and immediately [failed].

As this case shows, it is very important to understand where to look and how to filter out noise from the signal you care about.

A client has a medium sized kerberized HDP 2.6.0 installation. Hive Interactive is disabled and Hive is set up to use HTTP transport mode.