Benchmarking: Individual Results

Processing benchmark results

Import dependencies

The CarbonPlan team put together some utilities for parsing, processing, and visualizing the benchmarking results in carbonplan_benchmarks. We’ll use those utilities along with the Holoviz suite of tools for visualization to show some individual benchmarking results.

import carbonplan_benchmarks.analysis as cba
import pandas as pd

pd.options.plotting.backend = "holoviews"

Show individual results

Define URIs for the baseline images and metadata

baseline_fp = "s3://carbonplan-benchmarks/benchmark-data/v0.2/baselines.json"
url_filter = "carbonplan-benchmarks.s3.us-west-2.amazonaws.com/data/"
metadata_fp = (
    "s3://carbonplan-benchmarks/benchmark-data/v0.2/data-2023-08-24T00-33-44.json"
)

Define some plot options

plt_opts = {"width": 500, "height": 200}

Load the data and baseline images

metadata, trace_events = cba.load_data(metadata_path=metadata_fp, run=0)
snapshots = cba.load_snapshots(snapshot_path=baseline_fp)

Process the data

data = cba.process_run(
    metadata=metadata, trace_events=trace_events, snapshots=snapshots
)

Create summary statistics

cba.create_summary(metadata=metadata, data=data, url_filter=url_filter)
action approach benchmark_version browser_name browser_version headless playwright_python_version provider run_number timeout ... filtered_requests_average_encoded_data_length filtered_requests_maximum_encoded_data_length zoom duration min_rmse fps request_duration request_percent non_request_duration actual_chunk_size
dataset
pyramids-v2-3857-True-256-1-0-0-f4-0-0-0-gzipL1-100 zoom_in dynamic-client v0.2 chromium 116.0.5845.82 True 1.37.0 aws 1 False ... 63488.00 125345.0 0.0 2313.558 0.0 59.648386 408.764 17.668198 1904.794 0.524288
pyramids-v2-3857-True-256-1-0-0-f4-0-0-0-gzipL1-100 zoom_in dynamic-client v0.2 chromium 116.0.5845.82 True 1.37.0 aws 1 False ... 111303.25 206084.0 1.0 2468.501 0.0 60.360518 1365.362 55.311381 1103.139 0.524288
pyramids-v2-3857-True-256-1-0-0-f4-0-0-0-gzipL1-100 zoom_in dynamic-client v0.2 chromium 116.0.5845.82 True 1.37.0 aws 1 False ... 157726.00 342805.0 2.0 1591.088 0.0 60.336072 987.093 62.038869 603.995 0.524288
pyramids-v2-3857-True-256-1-0-0-f4-0-0-0-gzipL1-100 zoom_in dynamic-client v0.2 chromium 116.0.5845.82 True 1.37.0 aws 1 False ... 164329.00 334586.0 3.0 1229.950 0.0 59.352006 593.159 48.226269 636.791 0.524288

4 rows × 34 columns

Plot the requests, frames, and baseline comparisons

requests_plt = cba.plot_requests(data["request_data"], url_filter=url_filter).opts(
    **plt_opts
)
frames_plt = cba.plot_frames(data["frames_data"], yl=2.5).opts(**plt_opts)
rmse_plt = cba.plot_screenshot_rmse(
    screenshot_data=data["screenshot_data"], metadata=metadata
).opts(**plt_opts)
(requests_plt + frames_plt + rmse_plt).cols(1)
/Users/max/mambaforge/envs/benchmark-maps/lib/python3.10/site-packages/holoviews/plotting/bokeh/plot.py:987: UserWarning: found multiple competing values for 'toolbar.active_drag' property; using the latest value
  layout_plot = gridplot(
/Users/max/mambaforge/envs/benchmark-maps/lib/python3.10/site-packages/holoviews/plotting/bokeh/plot.py:987: UserWarning: found multiple competing values for 'toolbar.active_scroll' property; using the latest value
  layout_plot = gridplot(