Tracker¶

class smexperiments.tracker.Tracker(trial_component, metrics_writer, artifact_uploader, lineage_artifact_tracker)¶

Bases: object

A SageMaker Experiments Tracker.

Use a tracker object to record experiment information to a SageMaker trial component.

A new tracker can be created in two ways:

By loading an existing trial component with load()
By creating a tracker for a new trial component with create().

When creating a tracker within a SageMaker training or processing job, use the load method with no arguments to track artifacts to the trial component automatically created for your job. When tracking within a Jupyter notebook running in SageMaker, use the create method to automatically create a new trial component.

Trackers are Python context managers and you can use them using the Python with keyword. Exceptions thrown within the with block will cause the tracker’s trial component to be marked as failed. Start and end times are automatically set when using the with statement and the trial component is saved to SageMaker at the end of the block.

Note that parameters and input/output artifacts are saved to SageMaker directly via the UpdateTrialComponent operation. In contrast metrics (via log_metric method) are saved to a file, which is then ingested into SageMaker via a metrics agent which only runs on training job hosts. As a result any metrics logged in non-training job host environments will not be ingested into SageMaker.

Parameters:	trial_component (TrialComponent) – The trial component tracked.

trial_component = None¶

classmethod load(trial_component_name=None, artifact_bucket=None, artifact_prefix=None, boto3_session=None, sagemaker_boto_client=None, training_job_name=None, processing_job_name=None)¶

Create a new Tracker by loading an existing trial component.

Note that log_metric will only work from a training job host.

Examples

from smexperiments import tracker

# load tracker from already existing trial component
my_tracker = tracker.Tracker.load(trial_component_name='xgboost')

# load tracker from a training job name
my_tracker = tracker.Tracker.load(
    training_job_name=estimator.latest_training_job.name)

# load tracker from a processing job name
my_tracker = tracker.Tracker.load(
    processing_job_name=my_processing_job.name)

Parameters:	trial_component_name – (str, optional). The name of the trial component to track. If specified, this trial component must exist in SageMaker. If you invoke this method in a running SageMaker training or processing job, then trial_component_name can be left empty. In this case, the Tracker will resolve the trial component automatically created for your SageMaker Job. artifact_bucket – (str, optional) The name of the S3 bucket to store artifacts to. artifact_prefix – (str, optional) The prefix to write artifacts to within `artifact_bucket` boto3_session – (boto3.Session, optional) The boto3.Session to use to interact with AWS services. If not specified a new default boto3 session will be created. sagemaker_boto_client – (boto3.Client, optional) The SageMaker AWS service client to use. If not specified a new client will be created from the specified `boto3_session` or default boto3.Session. training_job_name – (str, optional). The name of the training job to track via trial processing_job_name – (str, optional). The name of the processing job to track via trial component.
Returns:	The tracker for the given trial component.
Return type:	Tracker
Raises:	`ValueError` – If the trial component failed to load.

classmethod create(base_trial_component_name='TrialComponent', display_name=None, artifact_bucket=None, artifact_prefix=None, boto3_session=None, sagemaker_boto_client=None)¶

Create a new Tracker by creating a new trial component.

Note that log_metric will _not_ work when tracker is created this way.

Examples

from smexperiments import tracker

my_tracker = tracker.Tracker.create()

Parameters:	base_trial_component_name – (str,optional). The name of the trial component resource that will be appended with a timestamp. Defaults to “TrialComponent”. display_name – (str, optional). The display name of the trial component to track. artifact_bucket – (str, optional) The name of the S3 bucket to store artifacts to. artifact_prefix – (str, optional) The prefix to write artifacts to within `artifact_bucket` boto3_session – (boto3.Session, optional) The boto3.Session to use to interact with AWS services. If not specified a new default boto3 session will be created. sagemaker_boto_client – (boto3.Client, optional) The SageMaker AWS service client to use. If not specified a new client will be created from the specified `boto3_session` or default boto3.Session.
Returns:	The tracker for the new trial component.
Return type:	Tracker

log_parameter(name, value)¶

Record a single parameter value for this trial component.

Overwrites any previous value recorded for the specified parameter name.

Examples

# log hyper parameter of learning rate
my_tracker.log_parameter('learning_rate', 0.01)

Parameters:	name (str) – The name of the parameter value (str or numbers.Number) – The value of the parameter

log_parameters(parameters)¶

Record a collection of parameter values for this trial component.

Examples

# log multiple hyper parameters used in training
my_tracker.log_parameters({"learning_rate": 1.0, "gamma": 0.9, "dropout": 0.5})

Parameters:	parameters (dict[str, str or numbers.Number]) – The parameters to record.

log_input(name, value, media_type=None)¶

Record a single input artifact for this trial component.

Overwrites any previous value recorded for the specified input name.

Examples

# log input dataset s3 location
my_tracker.log_input(name='input', value='s3://inputs/path')

Parameters:	name (str) – The name of the input value. value (str) – The value. media_type (str, optional) – The MediaType (MIME type) of the value

log_output(name, value, media_type=None)¶

Record a single output artifact for this trial component.

Overwrites any previous value recorded for the specified output name.

Examples

# log output dataset s3 location
my_tracker.log_output(name='prediction', value='s3://outputs/path')

Parameters:	name (str) – The name of the output value. value (str) – The value. media_type (str, optional) – The MediaType (MIME type) of the value.

log_artifacts(directory, media_type=None)¶

Upload all the files under the directory to s3 and store it as artifacts in this trial component. The file name is used as the artifact name

Examples

# log local artifact
my_tracker.log_artifact(directory='/local/path)

Parameters:	directory (str) – The directory of the local files to upload. media_type (str, optional) – The MediaType (MIME type) of the file. If not specified, this library will attempt to infer the media type from the file extension of `file_path`.

log_artifact(file_path, name=None, media_type=None)¶

Legacy overload method to prevent breaking existing code.

Examples

# log a local file
my_tracker.log_artifact('output/artifact_data.csv', name='prediction')

Parameters:	file_path (str) – Path to the file to log. name (str, optional) – Name of the artifact. Defaults to None. media_type (str, optional) – Media type of the artifact. Defaults to None.

log_output_artifact(file_path, name=None, media_type=None)¶

Upload a local file to s3 and store it as an output artifact in this trial component.

Examples

# log local artifact
my_tracker.log_output_artifact(file_path='/local/path/artifact.tar.gz')

Parameters:	file_path (str) – The path of the local file to upload. name (str, optional) – The name of the artifact. media_type (str, optional) – The MediaType (MIME type) of the file. If not specified, this library will attempt to infer the media type from the file extension of `file_path`.

log_input_artifact(file_path, name=None, media_type=None)¶

Upload a local file to s3 and store it as an input artifact in this trial component.

Examples

# log local artifact
my_tracker.log_input_artifact(file_path='/local/path/artifact.tar.gz')

Parameters:	file_path (str) – The path of the local file to upload. name (str, optional) – The name of the artifact. media_type (str, optional) – The MediaType (MIME type) of the file. If not specified, this library will attempt to infer the media type from the file extension of `file_path`.

log_metric(metric_name, value, timestamp=None, iteration_number=None)¶

Record a custom scalar metric value for this TrialComponent.

Note that this method is for manual custom metrics, for automatic metrics see the enable_sagemaker_metrics parameter on the estimator class in the main SageMaker SDK.

Note that metrics logged with this method will only appear in SageMaker when this method is called from a training job host.

Examples

for epoch in range(epochs):
    # your training logic and calculate accuracy and loss
    my_tracker.log_metric(metric_name='accuracy', value=0.9, iteration_number=epoch)
    my_tracker.log_metric(metric_name='loss', value=0.03, iteration_number=epoch)

Parameters:

metric_name (str) – The name of the metric.
value (number) – The value of the metric.
timestamp (datetime.datetime|number, optional) – The timestamp of the metric. If specified, should either be a datetime.datetime object or a number representing the seconds since the epoch. If not specified, the current local time will be used.
iteration_number (number, optional) – The integer iteration number of the metric value.

Raises:

AttributeError – If the metrics writer is not initialized.

log_table(title=None, values=None, data_frame=None, output_artifact=True)¶

Record a table of values to an artifact. Rendering in Studio is not currently supported.

Examples

table_data = {
    "x": [1,2,3],
    "y": [4,5,6]
}
my_tracker.log_table('SampleData',table_data)

# or log a data frame
df = pd.DataFrame(data=table_data)
my_tracker.log_table('SampleData',df)

Parameters:

title (str, optional) – Title of the table. Defaults to None.
values ([type], optional) – A dictionary of values. i.e. {“x”: [1,2,3], “y”: [1,2,3]}. Defaults to None.
data_frame (DataFrame, optional) – Pandas dataframe alternative to values. Defaults to None.
output_artifact (bool) – Determines direction of association to the trial component. Defaults to output artifact. If False will be an input artifact.

Raises:

ValueError – If values or data_frame are invalid.

log_precision_recall(y_true, predicted_probabilities, positive_label=None, title=None, output_artifact=True, no_skill=None)¶

Log a precision recall graph artifact. You can view the artifact in the charts tab of the Trial Component UI. If your job is created by a pipeline execution you can view the artifact by selecting the corresponding step in the pipelines UI. See also SageMaker Pipelines

Requires sklearn.

Examples

y_true = [0, 0, 1, 1]
y_scores = [0.1, 0.4, 0.35, 0.8]
no_skill = len(y_true[y_true==1]) / len(y_true)

my_tracker.log_precision_recall(y_true, y_scores, no_skill=no_skill)

Parameters:

y_true (array) – True labels. If labels are not binary then positive_label should be given.
predicted_probabilities (array) – Estimated/predicted probabilities.
positive_label (str or int, optional) – Label of the positive class. Defaults to None.
title (str, optional) – Title of the graph, Defaults to none.
output_artifact (boolean, optional) – Determines if the artifact is associated with the Trial Component as an output artifact. If False will be an input artifact.
no_skill (int) – The precision threshold under which the classifier cannot discriminate between the classes and would predict a random class or a constant class in all cases.

Raises:

ValueError – If length mismatch between y_true and predicted_probabilities.

log_roc_curve(y_true, y_score, title=None, output_artifact=True)¶

Log a receiver operating characteristic (ROC curve) artifact. You can view the artifact in the charts tab of the Trial Component UI. If your job is created by a pipeline execution you can view the artifact by selecting the corresponding step in the pipelines UI. See also SageMaker Pipelines <https://aws.amazon.com/sagemaker/pipelines/>.

Requires sklearn.

Examples

y_true = [0, 0, 1, 1]
y_scores = [0.1, 0.4, 0.35, 0.8]

my_tracker.log_roc_curve(y_true, y_scores)

Parameters:

y_true (array) – True labels. If labels are not binary then positive_label should be given.
y_score (array) – Estimated/predicted probabilities.
title (str, optional) – Title of the graph, Defaults to none.
output_artifact (boolean, optional) – Determines if the artifact is associated with the Trial Component as an output artifact. If False will be an input artifact.

Raises:

ValueError – If mismatch between y_true and predicted_probabilities.

log_confusion_matrix(y_true, y_pred, title=None, output_artifact=True)¶

Log a confusion matrix artifact. You can view the artifact in the charts tab of the Trial Component UI. If your job is created by a pipeline execution you can view the artifact by selecting the corresponding step in the pipelines UI. See also SageMaker Pipelines

Requires sklearn.

Examples

y_true = [2, 0, 2, 2, 0, 1]
y_pred = [0, 0, 2, 2, 0, 2]

my_tracker.log_confusion_matrix(y_true, y_pred)

Parameters:	y_true (array) – True labels. y_pred (array) – Predicted labels. title (str, optional) – Title of the graph, Defaults to none. output_artifact (boolean, optional) – Determines if the artifact is associated with the Trial Component as an output artifact. If False will be an input artifact.
Raises:	`ValueError` – If length mismatch between y_true and y_pred.

close()¶: Close this tracker and save state to SageMaker.