import carbonplan_benchmarks.analysis as cba
import pandas as pd
pd.options.plotting.backend = "holoviews"Benchmarking: Individual Results
Processing benchmark results
Import dependencies
The CarbonPlan team put together some utilities for parsing, processing, and visualizing the benchmarking results in carbonplan_benchmarks. We’ll use those utilities along with the Holoviz suite of tools for visualization to show some individual benchmarking results.
Show individual results
Define URIs for the baseline images and metadata
baseline_fp = "s3://carbonplan-benchmarks/benchmark-data/v0.2/baselines.json"
url_filter = "carbonplan-benchmarks.s3.us-west-2.amazonaws.com/data/"
metadata_fp = (
"s3://carbonplan-benchmarks/benchmark-data/v0.2/data-2023-08-24T00-33-44.json"
)Define some plot options
plt_opts = {"width": 500, "height": 200}Load the data and baseline images
metadata, trace_events = cba.load_data(metadata_path=metadata_fp, run=0)
snapshots = cba.load_snapshots(snapshot_path=baseline_fp)Process the data
data = cba.process_run(
metadata=metadata, trace_events=trace_events, snapshots=snapshots
)Create summary statistics
cba.create_summary(metadata=metadata, data=data, url_filter=url_filter)| action | approach | benchmark_version | browser_name | browser_version | headless | playwright_python_version | provider | run_number | timeout | ... | filtered_requests_average_encoded_data_length | filtered_requests_maximum_encoded_data_length | zoom | duration | min_rmse | fps | request_duration | request_percent | non_request_duration | actual_chunk_size | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| dataset | |||||||||||||||||||||
| pyramids-v2-3857-True-256-1-0-0-f4-0-0-0-gzipL1-100 | zoom_in | dynamic-client | v0.2 | chromium | 116.0.5845.82 | True | 1.37.0 | aws | 1 | False | ... | 63488.00 | 125345.0 | 0.0 | 2313.558 | 0.0 | 59.648386 | 408.764 | 17.668198 | 1904.794 | 0.524288 |
| pyramids-v2-3857-True-256-1-0-0-f4-0-0-0-gzipL1-100 | zoom_in | dynamic-client | v0.2 | chromium | 116.0.5845.82 | True | 1.37.0 | aws | 1 | False | ... | 111303.25 | 206084.0 | 1.0 | 2468.501 | 0.0 | 60.360518 | 1365.362 | 55.311381 | 1103.139 | 0.524288 |
| pyramids-v2-3857-True-256-1-0-0-f4-0-0-0-gzipL1-100 | zoom_in | dynamic-client | v0.2 | chromium | 116.0.5845.82 | True | 1.37.0 | aws | 1 | False | ... | 157726.00 | 342805.0 | 2.0 | 1591.088 | 0.0 | 60.336072 | 987.093 | 62.038869 | 603.995 | 0.524288 |
| pyramids-v2-3857-True-256-1-0-0-f4-0-0-0-gzipL1-100 | zoom_in | dynamic-client | v0.2 | chromium | 116.0.5845.82 | True | 1.37.0 | aws | 1 | False | ... | 164329.00 | 334586.0 | 3.0 | 1229.950 | 0.0 | 59.352006 | 593.159 | 48.226269 | 636.791 | 0.524288 |
4 rows × 34 columns
Plot the requests, frames, and baseline comparisons
requests_plt = cba.plot_requests(data["request_data"], url_filter=url_filter).opts(
**plt_opts
)
frames_plt = cba.plot_frames(data["frames_data"], yl=2.5).opts(**plt_opts)
rmse_plt = cba.plot_screenshot_rmse(
screenshot_data=data["screenshot_data"], metadata=metadata
).opts(**plt_opts)
(requests_plt + frames_plt + rmse_plt).cols(1)/Users/max/mambaforge/envs/benchmark-maps/lib/python3.10/site-packages/holoviews/plotting/bokeh/plot.py:987: UserWarning: found multiple competing values for 'toolbar.active_drag' property; using the latest value
layout_plot = gridplot(
/Users/max/mambaforge/envs/benchmark-maps/lib/python3.10/site-packages/holoviews/plotting/bokeh/plot.py:987: UserWarning: found multiple competing values for 'toolbar.active_scroll' property; using the latest value
layout_plot = gridplot(