zenml/finetuned-snowflake-arctic-embed-m-v1.5

This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-m-v1.5 on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Snowflake/snowflake-arctic-embed-m-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("zenml/finetuned-snowflake-arctic-embed-m-v1.5")
# Run inference
sentences = [
    'How does the KubernetesSparkStepOperator utilize the PipelineDockerImageBuilder class to manage Docker images for Spark jobs on Kubernetes?',
    'nsible for cluster-manager-specific configuration._io_configuration is a critical method. Even though we have materializers, Spark might require additional packages and configuration to work with a specific filesystem. This method is used as an interface to provide this configuration.\n\n_additional_configuration takes the submit_args, converts, and appends them to the overall configuration.\n\nOnce the configuration is completed, _launch_spark_job comes into play. This takes the completed configuration and runs a Spark job on the given master URL with the specified deploy_mode. By default, this is achieved by creating and executing a spark-submit command.\n\nWarning\n\nIn its first iteration, the pre-configuration with _io_configuration method is only effective when it is paired with an S3ArtifactStore (which has an authentication secret). When used with other artifact store flavors, you might be required to provide additional configuration through the submit_args.\n\nStack Component: KubernetesSparkStepOperator\n\nThe KubernetesSparkStepOperator is implemented by subclassing the base SparkStepOperator and uses the PipelineDockerImageBuilder class to build and push the required Docker images.\n\nfrom typing import Optional\n\nfrom zenml.integrations.spark.step_operators.spark_step_operator import (\n    SparkStepOperatorConfig\n)\n\nclass KubernetesSparkStepOperatorConfig(SparkStepOperatorConfig):\n    """Config for the Kubernetes Spark step operator."""\n\nnamespace: Optional[str] = None\n    service_account: Optional[str] = None\n\nfrom pyspark.conf import SparkConf\n\nfrom zenml.utils.pipeline_docker_image_builder import PipelineDockerImageBuilder\nfrom zenml.integrations.spark.step_operators.spark_step_operator import (\n    SparkStepOperator\n)\n\nclass KubernetesSparkStepOperator(SparkStepOperator):\n    """Step operator which runs Steps with Spark on Kubernetes."""',
    "ngs/python/Dockerfile -u 0 build\n\nConfiguring RBACAdditionally, you may need to create the several resources in Kubernetes in order to give Spark access to edit/manage your driver executor pods.\n\nTo do so, create a file called rbac.yaml with the following content:\n\napiVersion: v1\nkind: Namespace\nmetadata:\n  name: spark-namespace\n---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: spark-service-account\n  namespace: spark-namespace\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n  name: spark-role\n  namespace: spark-namespace\nsubjects:\n  - kind: ServiceAccount\n    name: spark-service-account\n    namespace: spark-namespace\nroleRef:\n  kind: ClusterRole\n  name: edit\n  apiGroup: rbac.authorization.k8s.io\n---\n\nAnd then execute the following command to create the resources:\n\naws eks --region=$REGION update-kubeconfig --name=$EKS_CLUSTER_NAME\n\nkubectl create -f rbac.yaml\n\nLastly, note down the namespace and the name of the service account since you will need them when registering the stack component in the next step.\n\nHow to use it\n\nTo use the KubernetesSparkStepOperator, you need:\n\nthe ZenML spark integration. If you haven't installed it already, run\n\nzenml integration install spark\n\nDocker installed and running.\n\nA remote artifact store as part of your stack.\n\nA remote container registry as part of your stack.\n\nA Kubernetes cluster deployed.\n\nWe can then register the step operator and use it in our active stack:\n\nzenml step-operator register spark_step_operator \\\n\t--flavor=spark-kubernetes \\\n\t--master=k8s://$EKS_API_SERVER_ENDPOINT \\\n\t--namespace=<SPARK_KUBERNETES_NAMESPACE> \\\n\t--service_account=<SPARK_KUBERNETES_SERVICE_ACCOUNT>\n\n# Register the stack\nzenml stack register spark_stack \\\n    -o default \\\n    -s spark_step_operator \\\n    -a spark_artifact_store \\\n    -c spark_container_registry \\\n    -i local_builder \\\n    --set",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Datasets: dim_384, dim_256, dim_128 and dim_64
• Evaluated with InformationRetrievalEvaluator

Metric	dim_384	dim_256	dim_128	dim_64
cosine_accuracy@1	0.75	0.75	0.75	0.75
cosine_accuracy@3	1.0	0.75	0.75	1.0
cosine_accuracy@5	1.0	1.0	1.0	1.0
cosine_accuracy@10	1.0	1.0	1.0	1.0
cosine_precision@1	0.75	0.75	0.75	0.75
cosine_precision@3	0.3333	0.25	0.25	0.3333
cosine_precision@5	0.2	0.2	0.2	0.2
cosine_precision@10	0.1	0.1	0.1	0.1
cosine_recall@1	0.75	0.75	0.75	0.75
cosine_recall@3	1.0	0.75	0.75	1.0
cosine_recall@5	1.0	1.0	1.0	1.0
cosine_recall@10	1.0	1.0	1.0	1.0
cosine_ndcg@10	0.875	0.8577	0.8577	0.875
cosine_mrr@10	0.8333	0.8125	0.8125	0.8333
cosine_map@100	0.8333	0.8125	0.8125	0.8333

Training Details

Training Dataset

json

• Dataset: json
• Size: 36 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 36 samples:

  	positive	anchor
  type	string	string
  details	min: 13 tokens mean: 23.11 tokens max: 38 tokens	min: 31 tokens mean: 299.64 tokens max: 512 tokens

• Samples:

  positive	anchor
  How does the ZenML BaseService registry manage serialization and re-creation of configurations for BaseService instances as part of the remote model server setup?	e details of the deployment process from the user.It needs to act as a ZenML BaseService registry, where every BaseService instance is used as an internal representation of a remote model server (see the find_model_server abstract method). To achieve this, it must be able to re-create the configuration of a BaseService from information that is persisted externally, alongside, or even as part of the remote model server configuration itself. For example, for model servers that are implemented as Kubernetes resources, the BaseService instances can be serialized and saved as Kubernetes resource annotations. This allows the model deployer to keep track of all externally running model servers and to re-create their corresponding BaseService instance representations at any given time. The model deployer also defines methods that implement basic life-cycle management on remote model servers outside the coverage of a pipeline (see stop_model_server , start_model_server and delete_model_server)....
  How do you ensure the MyExperimentTrackerFlavor is properly registered and available in ZenML?	gister flavors.my_flavor.MyExperimentTrackerFlavorZenML resolves the flavor class by taking the path where you initialized zenml (via zenml init) as the starting point of resolution. Therefore, please ensure you follow the best practice of initializing zenml at the root of your repository. If ZenML does not find an initialized ZenML repository in any parent directory, it will default to the current working directory, but usually, it's better to not have to rely on this mechanism and initialize zenml at the root. Afterward, you should see the new flavor in the list of available flavors: zenml experiment-tracker flavor list It is important to draw attention to when and how these base abstractions are coming into play in a ZenML workflow. The CustomExperimentTrackerFlavor class is imported and utilized upon the creation of the custom flavor through the CLI. The CustomExperimentTrackerConfig class is imported when someone tries to register/update a stack component with this custom fl...
  How do you load and profile a dataset using the Whylogs data validator in ZenML?	ogsDataValidatorSettings, ) from zenml import step@step( settings={ "data_validator": WhylogsDataValidatorSettings( enable_whylabs=True, dataset_id="model-1" ) } ) def data_loader() -> Tuple[ Annotated[pd.DataFrame, "data"], Annotated[DatasetProfileView, "profile"] ]: """Load the diabetes dataset.""" X, y = datasets.load_diabetes(return_X_y=True, as_frame=True) # merge X and y together df = pd.merge(X, y, left_index=True, right_index=True) profile = why.log(pandas=df).profile().view() return df, profile How do you use it? Whylogs's profiling functions take in a pandas.DataFrame dataset generate a DatasetProfileView object containing all the relevant information extracted from the dataset. There are three ways you can use whylogs in your ZenML pipelines that allow different levels of flexibility: instantiate, configure and insert the standard WhylogsProfilerStep shipped with ZenML into your pipelines. This is the easiest ...

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          384,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 4
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• tf32: False
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 4
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• eval_use_gather_object: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	dim_384_cosine_ndcg@10	dim_256_cosine_ndcg@10	dim_128_cosine_ndcg@10	dim_64_cosine_ndcg@10
1.0	1	0.875	0.875	0.8577	0.875
2.0	3	0.875	0.8577	0.8577	0.875
3.0	4	0.875	0.8577	0.8577	0.875

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.3.1
• Transformers: 4.43.1
• PyTorch: 2.5.1+cu124
• Accelerate: 1.2.0
• Datasets: 3.2.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}