Evaluation Pipelines
Hemm evaluation pipelines for Diffusers pipelines.
BaseDiffusionModel
Bases: Model
Base weave.Model
wrapping diffusers.DiffusionPipeline
.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
diffusion_model_name_or_path |
str
|
The name or path of the diffusion model. |
required |
enable_cpu_offfload |
bool
|
Enable CPU offload for the diffusion model. |
False
|
image_height |
int
|
The height of the generated image. |
512
|
image_width |
int
|
The width of the generated image. |
512
|
disable_safety_checker |
bool
|
Disable safety checker for the diffusion model. |
True
|
Source code in hemm/eval_pipelines/model.py
EvaluationPipeline
Bases: ABC
Evaluation pipeline to evaluate the a multi-modal generative model.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
model |
BaseDiffusionModel
|
The model to evaluate. |
required |
seed |
int
|
Seed value for the random number generator. |
42
|
Source code in hemm/eval_pipelines/eval_pipeline.py
12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 |
|
__call__(dataset)
Evaluate the Stable Diffusion model on the given dataset.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
dataset |
Union[List[Dict], str]
|
Dataset to evaluate the model on. If a string is passed, it is assumed to be a Weave dataset reference. |
required |
Source code in hemm/eval_pipelines/eval_pipeline.py
add_metric(metric_fn)
Add a metric function to the evaluation pipeline.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
metric_fn |
BaseMetric
|
Metric function to evaluate the generated images. |
required |
Source code in hemm/eval_pipelines/eval_pipeline.py
infer(prompt)
Inference function to generate images for the given prompt.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
prompt |
str
|
Prompt to generate the image. |
required |
Returns:
Type | Description |
---|---|
Dict[str, str]
|
Dict[str, str]: Dictionary containing base64 encoded image to be logged as a Weave object. |
Source code in hemm/eval_pipelines/eval_pipeline.py
infer_async(prompt)
async
Async inference function to generate images for the given prompt.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
prompt |
str
|
Prompt to generate the image. |
required |
Returns:
Type | Description |
---|---|
Dict[str, str]
|
Dict[str, str]: Dictionary containing base64 encoded image to be logged as a Weave object. |
Source code in hemm/eval_pipelines/eval_pipeline.py
log_summary(summary)
Log the evaluation summary to the Weights & Biases dashboard.