[...] I needed to get a Hive cluster running my code and a Confluent cluster that could output Avro messages in the proper format to test.

[...] focus on best practices such as when and why you should use Oozie, and when to use bundles.

A team at a client was using Spark to read and write to a Kafka topic. [...] files that would be written that were completely empty.

We made the change on a Sunday, but four days later, the number of files had not appreciably changed in the YARN logs directory.

The system which maintains the application logs in HDFS is called the Log Aggregation system and is flexible [...]

This client didn't use the activity explorer much [...] I tried to run one of the paragraphs and immediately [failed].

At a high level, [CGC] is merely going through the cluster, taking inventory of the data and processes that run on the cluster...

We will discuss how partitions play with YARN queues. Finally, we will return to the example given in the first part of this series.

How do we ensure that GPU jobs run on worker nodes with GPUs without buying expensive GPUs for all of our worker nodes?

I'm going to explore exactly how YARN works with queues, and the various mechanisms available to control how YARN does this.

A client has a medium sized kerberized HDP 2.6.0 installation. Hive Interactive is disabled and Hive is set up to use HTTP transport mode.