criterion performance measurements

overview

want to understand this report?

unary request

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.1127952052099233e-4 1.1157779769569623e-4 1.1222578474109467e-4
Standard deviation 7.006642862776891e-7 1.6019470204849632e-6 3.0016649862884594e-6

Outlying measurements have slight (8.197243287309262e-2%) effect on estimated standard deviation.

client stream: 100 messages

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.1145656673217617e-3 1.1225264180385309e-3 1.136100231001483e-3
Standard deviation 2.1175077057984874e-5 3.660369985889931e-5 6.0245576168224164e-5

Outlying measurements have moderate (0.2100730004584609%) effect on estimated standard deviation.

client stream: 1k messages

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.0274288101958585e-2 1.0341389459833119e-2 1.0418416003070792e-2
Standard deviation 1.5974251878414234e-4 2.0913496502759373e-4 2.889361409397357e-4

Outlying measurements have slight (3.222222222222209e-2%) effect on estimated standard deviation.

client stream: 10k messages

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.1035139522385206 0.1044357553238654 0.10589733927787692
Standard deviation 8.693634002187114e-4 1.7581790983451108e-3 2.6984798923229657e-3

Outlying measurements have slight (9.876543209876533e-2%) effect on estimated standard deviation.

server stream: 100 messages

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.00500185533956e-3 1.0219297837648412e-3 1.038431407924066e-3
Standard deviation 4.534639216316358e-5 5.8708277292223296e-5 7.292687430223964e-5

Outlying measurements have moderate (0.4697581699045145%) effect on estimated standard deviation.

server stream: 1k messages

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 8.263458511284804e-3 8.29655676830788e-3 8.338747504141219e-3
Standard deviation 7.989147957982453e-5 1.1338955133128914e-4 1.6239809568186118e-4

Outlying measurements have slight (2.938475665748384e-2%) effect on estimated standard deviation.

server stream: 10k messages

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 8.139915127469478e-2 8.203977915769449e-2 8.294207157255142e-2
Standard deviation 8.712272680628149e-4 1.303323554210239e-3 1.75288845055683e-3

Outlying measurements have slight (9.000000000000001e-2%) effect on estimated standard deviation.

bidi stream: 50 messages up, 50 down

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.6837263323658102e-3 3.690185989917211e-3 3.7016024806948455e-3
Standard deviation 1.87435230068336e-5 2.7406270029965376e-5 3.868689618608243e-5

Outlying measurements have slight (2.1266540642722116e-2%) effect on estimated standard deviation.

bidi stream: 500 message up, 500 down

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.539779838485038e-2 3.5467452958929724e-2 3.5585300045038584e-2
Standard deviation 1.1602363892840695e-4 1.603301186146598e-4 2.1296560804213925e-4

Outlying measurements have slight (5.8593749999999986e-2%) effect on estimated standard deviation.

bidi stream: 5000 messages up, 5000 down

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.3519412043621479 0.35207846750875343 0.3521320771337429
Standard deviation 0.0 1.2712910344311448e-4 1.4489214967143587e-4

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.