aboutsummaryrefslogtreecommitdiffstats
path: root/lib
diff options
context:
space:
mode:
authorAndriy Gapon <avg@FreeBSD.org>2017-06-09 15:04:10 +0000
committerAndriy Gapon <avg@FreeBSD.org>2017-06-09 15:04:10 +0000
commit79edb7989abbfe7000fec68b85710a1016bdf5b1 (patch)
treed0c5e2c77da7843d5a328e5a0600c9383f1034bb /lib
parent27ba1b79cab23d0dc2e28a13c5fc4bc5f6fd1c4d (diff)
downloadsrc-79edb7989abbfe7000fec68b85710a1016bdf5b1.tar.gz
src-79edb7989abbfe7000fec68b85710a1016bdf5b1.zip
8269 dtrace stddev aggregation is normalized incorrectly
illumos/illumos-gate@79809f9cf402f130667349b2d4007ecd65d63c6f https://github.com/illumos/illumos-gate/commit/79809f9cf402f130667349b2d4007ecd65d63c6f https://www.illumos.org/issues/8269 It seems that currently normalization of stddev aggregation is done incorrectly. We divide both the sum of values and the sum of their squares by the normalization factor. But we should divide the sum of squares by the normalization factor squared to scale the original values properly. Reviewed by: Bryan Cantrill <bryan@joyent.com> Approved by: Robert Mustacchi <rm@joyent.com> Author: Andriy Gapon <avg@FreeBSD.org>
Notes
Notes: svn path=/vendor/illumos/dist/; revision=319744
Diffstat (limited to 'lib')
-rw-r--r--lib/libdtrace/common/dt_consume.c2
1 files changed, 2 insertions, 0 deletions
diff --git a/lib/libdtrace/common/dt_consume.c b/lib/libdtrace/common/dt_consume.c
index 7557e98ae700..fff0616e23dd 100644
--- a/lib/libdtrace/common/dt_consume.c
+++ b/lib/libdtrace/common/dt_consume.c
@@ -381,8 +381,10 @@ dt_stddev(uint64_t *data, uint64_t normal)
* The standard approximation for standard deviation is
* sqrt(average(x**2) - average(x)**2), i.e. the square root
* of the average of the squares minus the square of the average.
+ * When normalizing, we should divide the sum of x**2 by normal**2.
*/
dt_divide_128(data + 2, normal, avg_of_squares);
+ dt_divide_128(avg_of_squares, normal, avg_of_squares);
dt_divide_128(avg_of_squares, data[0], avg_of_squares);
norm_avg = (int64_t)data[1] / (int64_t)normal / (int64_t)data[0];