Spark and Data Sketches: Taming count-distinct

Distinct counting is a commonly used metric in trend analysis to measure popularity or performance. Although it may seem like a simple problem, the challenge quickly grows as the amount of data grows. Counting exact distinct values is slow and resource-hungry, even on a parallel engine like Spark. Probabilistic data sketches give you a fast, accurate-enough estimate instead.