Debugging from the Field: Sudden CI Test Failures
This is one of a series of posts revolving around debugging stories from the field. The goal of this series is to help demonstrate how to...
top of page
Search
-
- Mar 10, 2020
- 4 min
A Crash Course in Proper Oozie Usage
[...] focus on best practices such as when and why you should use Oozie, and when to use bundles.
310 views0 comments
-
- Mar 3, 2020
- 5 min
Debugging From The Field: When Parallelization Goes Wrong
Blog post chronicling the investigation of a series of errors in a Spark Streaming job reading from Spark.
736 views0 comments
-
- Feb 25, 2020
- 7 min
Debugging From The Field: The Case of the Empty Files
A team at a client was using Spark to read and write to a Kafka topic. [...] files that would be written that were completely empty.
324 views1 comment
-
- Feb 18, 2020
- 4 min
Spark Job Optimization Myth #6: I'm Seeing Out of Memory Exceptions, So I Need to Increase Memory
... we're going to discuss [...] the common excuse I see from developers to immediately try to increase resources, [the OutOfMemory error]
781 views0 comments
-
- Feb 11, 2020
- 5 min
Spark Job Optimization Myth #5: Increasing Executor Cores is Always a Good Idea
This week, we're going to talk about executor cores. First [...] we'll understand how setting executor cores affects our jobs...
11,399 views1 comment
-
- Feb 4, 2020
- 5 min
Spark Job Optimization Myth #4: I Need More Overhead Memory
[This week,] we'll look at the overhead memory parameter, which is available for both driver and executors.
2,203 views2 comments
-
- Jan 28, 2020
- 5 min
Spark Job Optimization Myth #3: I Need More Driver Memory
The most common misconception I see developers fall into with regards to the driver configuration is increasing driver memory.
9,994 views0 comments
-
- Jan 21, 2020
- 5 min
Spark Job Optimization Myth #2: Increasing the Number of Executors Always Improves Performance
Increasing the number of executors certainly works [...], but there are still very important times when it doesn't work the way you'd want.
5,492 views0 comments
-
- Jan 6, 2020
- 5 min
Spark Job Optimization Myth #1: Increasing the Memory Per Executor Always Improves Performance
Not only is [increasing memory] haphazard and leads to inconsistent results, but it also doesn't actually do what they think it does.
6,556 views0 comments
-
- Nov 5, 2019
- 6 min
Spark Job Optimization: Dealing with Data Skew
Optimizing a Spark job can be a daunting task. [...] This series is going to focus on diving into the inner works of Spark [...]
5,949 views0 comments
-
- Oct 29, 2019
- 6 min
Stop Feeding the Small File Monster!
If you've worked with Hadoop for any amount of time, you've likely run into the dreaded small file problem...
201 views3 comments
-
- Oct 16, 2019
- 6 min
Writing Environment Agnostic Code
[...] we'll discuss some of the ways we can write environment agnostic code, which can be run on any environment within your enterprise.
1,109 views0 comments
-
- Oct 6, 2019
- 5 min
YARN Capacity Scheduler and Node Labels Part 3
We will discuss how partitions play with YARN queues. Finally, we will return to the example given in the first part of this series.
779 views0 comments
-
- Sep 29, 2019
- 4 min
YARN Capacity Scheduler and Node Labels Part 2
How do we ensure that GPU jobs run on worker nodes with GPUs without buying expensive GPUs for all of our worker nodes?
736 views0 comments
-
- Sep 22, 2019
- 5 min
YARN Capacity Scheduler and Node Labels Part 1
I'm going to explore exactly how YARN works with queues, and the various mechanisms available to control how YARN does this.
2,277 views0 comments
-
- Sep 15, 2019
- 4 min
Debugging from the Field: Sudden En Masse Failures in 100s of Spark Streaming Jobs
As this case shows, it is very important to understand where to look and how to filter out noise from the signal you care about.
122 views0 comments
bottom of page