"""
This library allows you to easily benchmark pipeline performance metrics.

# Using the PipelineBenchmarker class

The `PipelineBenchmarker` class assists in automatically tracking benchmarking values of interest in any [Azure Machine Learning (AML) pipeline](https://docs.microsoft.com/en-us/azure/machine-learning/concept-ml-pipelines). This includes compute metadata, step duration time, maximum number of nodes used, and any custom parameter values designated by the user.

This document provides an overview of the purpose of the class and examples of how to implement it. Fully functional example code is available at `mlops/example_pipeline/*` and code-specific class documentation at [`benchmark_utils.html`](./benchmark_utils.html).

## Benefits of benchmarking your pipeline

When developing an MLOps solution, the team may need to **experiment with various compute configurations and miscellaneous parameters** before settling on what works best for the particular use case.

The `PipelineBenchmarker` tool makes performing these comparisons easier by automatically tracking benchmark values of interest separate from traditional model metrics.

> *While model performance metrics may certainly be logged with the `PipelineBenchmarker` class, that is not the intended purpose.*

By tracking values such as compute configuration (e.g., compute target name, size, priority, region), number of nodes used, and the number of time to complete each step and the total pipeline, we can gain a clear understanding of (1) what steps are the most computationally heavy and (2) what parameters and compute configurations we may want to change to optimize for either speed or cost.

The class can be particularly helpful in cases where calls to Azure Cognitive Services need to be logged for cost estimation. Additional values such as "number of calls to the Custom Vision service" or "duration of audio sent to the Custom Speech service" may easily be passed into the `PipelineBenchmarker` object.

## How to use the PipelineBenchmarker class

The general structure of how to use the `PipelineBenchmarker` class is as follows:

1. For each of pipeline step's Python script:
    - Import the `PipelineBenchmarker` class
    - Instantiate a `PipelineBenchmarker` object at the start of your step
    - Save benchmark values at the end of your step by calling the `.save_step_benchmark(step_name="", experiment_output_dir="", benchmark_dict={})` method on your object with the appropriate argument values. Step duration and number of nodes used are automatically saved through this step, and additional values may be stored as a dictionary through the optional `benchmark_dict` argument.
    - For full example code demonstrating this implementation, see `prepare_data.py` and `process_data.py` in `mlops/example_pipeline/steps/`
2. Create an additional AML pipeline step to perform the pipeline benchmarking consolidation (see `mlops/example_pipeline/steps/generate_benchmark_report.py`)
3. Add this benchmark report generation step to your pipeline creation script, and ensure it is the last step to execute (see `mlops/example_pipeline/Example_Pipeline.ipynb`)

## Autogenerated benchmark reports are saved to blob

With the addition of the final benchmark report generation pipeline step, consolidated benchmarks are written out.

The two outputs are as follows:

1. A singular `benchmark_<pipeline run ID>.json` is generated and saved per pipeline run. This is stored in `pipeline_benchmarks/individual_runs/` in the mounted Azure blob storage.
2. Additional rows are added onto `pipeline_benchmarks/benchmarks.csv` in the mounted Azure blob storage, with each row containing values for each step benchmarked in the pipeline run (excluding the `generate_benchmark_report.py` step).

Example output are available in `docs/example_output/*`.

"""
