BLOG | Understanding Data.

All Posts
Administration
Machine Learning
Security
Development
Issue Report
Tutorial

Jun 24
2 min

Personal Update

I've had some subscribers reach out wondering what happened, so I felt it was important to update everyone on the reasons for my silence.

16 viewsWrite a comment

David McGinnis

Mar 17
6 min

Testing a Hive Patch on a Local System

[...] I needed to get a Hive cluster running my code and a Confluent cluster that could output Avro messages in the proper format to test.

41 viewsWrite a comment

David McGinnis

Mar 10
4 min

A Crash Course in Proper Oozie Usage

[...] focus on best practices such as when and why you should use Oozie, and when to use bundles.

20 viewsWrite a comment

David McGinnis

Mar 3
5 min

Debugging From The Field: When Parallelization Goes Wrong

Blog post chronicling the investigation of a series of errors in a Spark Streaming job reading from Spark.

82 viewsWrite a comment

David McGinnis

Feb 25
7 min

Debugging From The Field: The Case of the Empty Files

A team at a client was using Spark to read and write to a Kafka topic. [...] files that would be written that were completely empty.

25 views1 comment

David McGinnis

Feb 18
4 min

Spark Job Optimization Myth #6: I'm Seeing Out of Memory Exceptions, So I Need to Increase Memory

... we're going to discuss [...] the common excuse I see from developers to immediately try to increase resources, [the OutOfMemory error]

73 viewsWrite a comment

David McGinnis

Feb 11
5 min

Spark Job Optimization Myth #5: Increasing Executor Cores is Always a Good Idea

This week, we're going to talk about executor cores. First [...] we'll understand how setting executor cores affects our jobs...

172 viewsWrite a comment

David McGinnis

Feb 4
5 min

Spark Job Optimization Myth #4: I Need More Overhead Memory

[This week,] we'll look at the overhead memory parameter, which is available for both driver and executors.

137 views2 comments

David McGinnis

Jan 28
5 min

Spark Job Optimization Myth #3: I Need More Driver Memory

The most common misconception I see developers fall into with regards to the driver configuration is increasing driver memory.

1,400 viewsWrite a comment

David McGinnis

Jan 21
5 min

Spark Job Optimization Myth #2: Increasing the Number of Executors Always Improves Performance

Increasing the number of executors certainly works [...], but there are still very important times when it doesn't work the way you'd want.

337 viewsWrite a comment

David McGinnis

Jan 6
5 min

Spark Job Optimization Myth #1: Increasing the Memory Per Executor Always Improves Performance

Not only is [increasing memory] haphazard and leads to inconsistent results, but it also doesn't actually do what they think it does.

530 viewsWrite a comment

David McGinnis

Dec 3, 2019
5 min

Debugging From The Field: The Case of the Ignored Configuration Change

We made the change on a Sunday, but four days later, the number of files had not appreciably changed in the YARN logs directory.

40 viewsWrite a comment

David McGinnis

Nov 26, 2019
4 min

A Crash Course in YARN Log Aggregation

The system which maintains the application logs in HDFS is called the Log Aggregation system and is flexible [...]

57 viewsWrite a comment

David McGinnis

Nov 21, 2019
5 min

Tracking Down Small Files with Smartsense Dashboards

Overview Last week I mentioned Smartsense Dashboards, including an error we were seeing with the dashboards. If you missed it, you can fi...

22 viewsWrite a comment

David McGinnis

Nov 12, 2019
4 min

Debugging from the Field: Smartsense Activity Explorer Stops Working

This client didn't use the activity explorer much [...] I tried to run one of the paragraphs and immediately [failed].

87 viewsWrite a comment

David McGinnis

Nov 6, 2019
6 min

Spark Job Optimization: Dealing with Data Skew

Optimizing a Spark job can be a daunting task. [...] This series is going to focus on diving into the inner works of Spark [...]

1,754 viewsWrite a comment

David McGinnis

Oct 29, 2019
6 min

Stop Feeding the Small File Monster!

If you've worked with Hadoop for any amount of time, you've likely run into the dreaded small file problem...

59 views3 comments

David McGinnis

Oct 22, 2019
4 min

Running Garbage Collection on Your Cluster

At a high level, [CGC] is merely going through the cluster, taking inventory of the data and processes that run on the cluster...

31 viewsWrite a comment

David McGinnis

Oct 16, 2019
6 min

Writing Environment Agnostic Code

[...] we'll discuss some of the ways we can write environment agnostic code, which can be run on any environment within your enterprise.

34 viewsWrite a comment

David McGinnis

Oct 6, 2019
5 min

YARN Capacity Scheduler and Node Labels Part 3

We will discuss how partitions play with YARN queues. Finally, we will return to the example given in the first part of this series.

82 viewsWrite a comment

{"items":["5ef2b0bf0e301e0017f61564","5e693c27b5394100171c29f7","5e652a602b14a500178f388e","5e372808d8a25a0018c6eeaa","5e2cb56bf75427001767e828","5e3725000298f80017b33907","5e35ed0ec766b700171701f0","5e35dd07daf12f0017285944","5e2e0d1a7f1d210017a29e65","5e1e80689e7cd20017f59cfa","5de2cb6919e52f0018760b00","5ddaed4bce4ab700173949dc","5ddafee4ce4ab70017397582","5dc226464b13ab001741802f","5dc3484533b95000170484d8","5dbb11312df26e0017674636","5db8a78876f5db001788a68b","5d65faf84334c8027b9ba6ad","5d695a2760df0100163e02d5","5d85192e685e48001711772f"],"styles":{"groupSize":1,"showArrows":true,"cubeImages":false,"cubeType":"fill","cubeRatio":1,"isVertical":true,"gallerySize":454,"collageAmount":0,"collageDensity":0,"groupTypes":"1","oneRow":false,"imageMargin":16,"galleryMargin":0,"floatingImages":0,"chooseBestGroup":true,"smartCrop":false,"hasThumbnails":false,"enableScroll":true,"isGrid":false,"isSlider":false,"isColumns":false,"isSlideshow":false,"cropOnlyFill":false,"fixedColumns":0,"enableInfiniteScroll":true,"isRTL":false,"minItemSize":120,"rotatingGroupTypes":"","rotatingCubeRatio":"","gallerySliderImageRatio":1.7777777777777777,"numberOfImagesPerRow":3,"numberOfImagesPerCol":1,"groupsPerStrip":0,"borderRadius":0,"boxShadow":0,"gridStyle":0,"mobilePanorama":false,"placeGroupsLtr":false,"viewMode":"preview","thumbnailSpacings":4,"galleryThumbnailsAlignment":"bottom","isMasonry":true,"isAutoSlideshow":false,"slideshowLoop":false,"autoSlideshowInterval":4,"useCustomButton":false,"bottomInfoHeight":0,"titlePlacement":"SHOW_BELOW","galleryHorizontalAlign":"center","galleryTextAlign":"center","galleryVerticalAlign":"center","scrollSnap":false,"itemClick":"expand","fullscreen":true,"allowSocial":true,"allowDownload":false,"allowTitle":true,"allowDescription":false,"loveButton":true,"loveCounter":false,"videoPlay":"hover","scrollAnimation":"NO_EFFECT","scrollDirection":0,"overlayAnimation":"FADE_IN","arrowsPosition":0,"arrowsSize":23,"watermarkOpacity":40,"watermarkSize":40,"useWatermark":true,"watermarkDock":{"top":"auto","left":"auto","right":0,"bottom":0,"transform":"translate3d(0,0,0)"},"loadMoreAmount":"all","defaultShowInfoExpand":1,"allowTitleExpand":true,"allowDescriptionExpand":true,"allowLinkExpand":true,"expandInfoPosition":0,"allowFullscreenExpand":true,"fullscreenLoop":false,"galleryAlignExpand":"left","addToCartBorderWidth":1,"addToCartButtonText":"","slideshowInfoSize":200,"playButtonForAutoSlideShow":false,"allowSlideshowCounter":false,"hoveringBehaviour":"NEVER_SHOW","thumbnailSize":120,"magicLayoutSeed":1,"imageHoverAnimation":"NO_EFFECT","calculateTextBoxHeightMode":"MANUAL","calculateTextBoxWidthMode":"PERCENT","textBoxHeight":320,"textBoxWidth":200,"textBoxWidthPercent":50,"textImageSpace":10,"textBoxBorderRadius":0,"textBoxBorderWidth":0,"textsVerticalPadding":-15,"textsHorizontalPadding":-30,"titleDescriptionSpace":6,"customButtonText":"","customButtonBorderWidth":1,"customButtonBorderRadius":0,"loadMoreButtonText":"","loadMoreButtonBorderWidth":1,"loadMoreButtonBorderRadius":0,"imageInfoType":"ATTACHED_BACKGROUND","itemBorderWidth":1,"itemBorderRadius":0,"itemEnableShadow":false,"itemShadowBlur":20,"itemShadowDirection":135,"itemShadowSize":10,"imageLoadingMode":"BLUR","expandAnimation":"NO_EFFECT","imageQuality":90,"usmToggle":false,"usm_a":0,"usm_r":0,"usm_t":0,"videoSound":false,"videoSpeed":"1","videoLoop":true,"gallerySizeType":"px","gallerySizePx":454,"itemBorderColor":{"value":"rgb(179, 179, 179)"},"galleryLayout":1,"selectedLayout":"1|bottom|1|fill|true|0|true","layoutsVersion":2,"selectedLayoutV2":1,"isSlideshowFont":true,"externalInfoHeight":320,"externalInfoWidth":0},"container":{"width":940,"galleryWidth":972,"galleryHeight":1440,"scrollBase":0,"height":null}}

SIGN UP AND STAY UPDATED!