BLOG | Understanding Data.

David McGinnis

Feb 2, 2022
4 min

Abstracting External Services with I/O Interfaces - Part 2

David McGinnis

Jan 26, 2022
5 min

Abstracting External Services with I/O Interfaces - Part 1

While this is a concept I've used at many clients, I've never seen this concept approached elsewhere. Because of this, all of the...

David McGinnis

Jan 19, 2022
4 min

Debugging from the Field: Sudden CI Test Failures

David McGinnis

Jan 12, 2022
2 min

A New Start

David McGinnis

Jun 23, 2020
2 min

Personal Update

David McGinnis

Mar 17, 2020
6 min

Testing a Hive Patch on a Local System

David McGinnis

Mar 10, 2020
4 min

A Crash Course in Proper Oozie Usage

David McGinnis

Mar 3, 2020
5 min

Debugging From The Field: When Parallelization Goes Wrong

David McGinnis

Feb 25, 2020
7 min

Debugging From The Field: The Case of the Empty Files

David McGinnis

Feb 18, 2020
4 min

Spark Job Optimization Myth #6: I'm Seeing Out of Memory Exceptions, So I Need to Increase Memory

David McGinnis

Feb 11, 2020
5 min

Spark Job Optimization Myth #5: Increasing Executor Cores is Always a Good Idea

David McGinnis

Feb 4, 2020
5 min

Spark Job Optimization Myth #4: I Need More Overhead Memory

David McGinnis

Jan 28, 2020
5 min

Spark Job Optimization Myth #3: I Need More Driver Memory

David McGinnis

Jan 21, 2020
5 min

Spark Job Optimization Myth #2: Increasing the Number of Executors Always Improves Performance

David McGinnis

Jan 6, 2020
5 min

Spark Job Optimization Myth #1: Increasing the Memory Per Executor Always Improves Performance

David McGinnis

Dec 3, 2019
5 min

Debugging From The Field: The Case of the Ignored Configuration Change

David McGinnis

Nov 26, 2019
4 min

A Crash Course in YARN Log Aggregation

David McGinnis

Nov 20, 2019
5 min

Tracking Down Small Files with Smartsense Dashboards

David McGinnis

Nov 12, 2019
4 min

Debugging from the Field: Smartsense Activity Explorer Stops Working

David McGinnis

Nov 5, 2019
6 min

Abstracting External Services with I/O Interfaces - Part 2

Abstracting External Services with I/O Interfaces - Part 1

Debugging from the Field: Sudden CI Test Failures

A New Start

Personal Update

Testing a Hive Patch on a Local System

A Crash Course in Proper Oozie Usage

Debugging From The Field: When Parallelization Goes Wrong

Debugging From The Field: The Case of the Empty Files

Spark Job Optimization Myth #6: I'm Seeing Out of Memory Exceptions, So I Need to Increase Memory

Spark Job Optimization Myth #5: Increasing Executor Cores is Always a Good Idea

Spark Job Optimization Myth #4: I Need More Overhead Memory

Spark Job Optimization Myth #3: I Need More Driver Memory

Spark Job Optimization Myth #2: Increasing the Number of Executors Always Improves Performance

Spark Job Optimization Myth #1: Increasing the Memory Per Executor Always Improves Performance

Debugging From The Field: The Case of the Ignored Configuration Change

A Crash Course in YARN Log Aggregation

Tracking Down Small Files with Smartsense Dashboards

Debugging from the Field: Smartsense Activity Explorer Stops Working

Spark Job Optimization: Dealing with Data Skew

SIGN UP AND STAY UPDATED!