End of training

README.md

@@ -0,0 +1,87 @@
+---
+base_model: Qwen/Qwen2-0.5B-Instruct
+library_name: distily
+license: apache-2.0
+tags:
+- generated_from_trainer
+model-index:
+- name: distily_experiments_loss_mse
+  results: []
+---
+
+# distily_experiments_loss_mse
+
+This student model is distilled from the teacher model [Qwen/Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct) using the dataset (unspecified).
+
+The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
+
+It achieves the following results on the evaluation set:
+- eval_enwikippl: 42178.9805
+- eval_frwikippl: 45721.7109
+- eval_zhwikippl: 198318.0
+- eval_loss: 0.0786
+- eval_runtime: 81.5551
+- eval_samples_per_second: 12.262
+- eval_steps_per_second: 3.065
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment.
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+-->
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- distillation_strategy: logits_activations
+- loss_fn: mse
+- train_embeddings: True
+- learning_rate: 0.0001
+- train_batch_size: 4
+- eval_batch_size: 4
+- seed: 42
+- gradient_accumulation_steps: 4
+- total_train_batch_size: 16
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: constant
+- num_epochs: 1.0
+
+### Resource Usage
+Peak GPU Memory: 12.6346 GB
+
+### Eval-Phase Metrics
+| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| **teacher eval** |  | 13.0697 | 11.6518 |  |  |  |  | 21.6262 |
+| 0 | 0 | 181673.8281 | 182246.2969 | 0.4342 | 81.4623 | 12.276 | 3.069 | 181831.9062 |
+| 500 | 0.0808 | 51828.1172 | 61691.7969 | 0.0957 | 81.6929 | 12.241 | 3.06 | 207997.2812 |
+| 1000 | 0.1616 | 45041.4180 | 54485.3594 | 0.0860 | 81.5919 | 12.256 | 3.064 | 209966.75 |
+| 1500 | 0.2424 | 44320.2031 | 52669.4492 | 0.0833 | 81.6149 | 12.253 | 3.063 | 212294.125 |
+| 2000 | 0.3232 | 41822.9727 | 48903.5156 | 0.0852 | 81.4039 | 12.284 | 3.071 | 223025.1094 |
+| 2500 | 0.4040 | 41753.0312 | 49089.7227 | 0.0827 | 81.431 | 12.28 | 3.07 | 220010.75 |
+| 3000 | 0.4848 | 42048.8906 | 49907.2539 | 0.0840 | 81.49 | 12.271 | 3.068 | 210042.8594 |
+| 3500 | 0.5657 | 41577.9180 | 46600.7773 | 0.0849 | 81.4631 | 12.275 | 3.069 | 221650.6875 |
+| 4000 | 0.6465 | 41242.4766 | 46538.6875 | 0.0824 | 81.561 | 12.261 | 3.065 | 211307.8125 |
+| 4500 | 0.7273 | 41470.4414 | 46413.6094 | 0.0803 | 81.5887 | 12.257 | 3.064 | 221236.5469 |
+| 5000 | 0.8081 | 42138.4922 | 45917.2109 | 0.0799 | 81.4339 | 12.28 | 3.07 | 207382.2812 |
+| 5500 | 0.8889 | 41578.75 | 45413.4766 | 0.0818 | 81.4181 | 12.282 | 3.071 | 233708.25 |
+| 6000 | 0.9697 | 42023.4766 | 45574.0469 | 0.0790 | 81.4797 | 12.273 | 3.068 | 208369.9375 |
+| 6187 | 0.9999 | 42178.9805 | 45721.7109 | 0.0786 | 81.5551 | 12.262 | 3.065 | 198318.0 |
+
+### Framework versions
+- Distily 0.1.0
+- Transformers 4.43.3
+- Pytorch 2.3.0
+- Datasets 2.20.0

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 151643,
+  "eos_token_id": 151645,
+  "transformers_version": "4.43.3"
+}

runs/Aug06_19-32-49_232a0f8c3879/events.out.tfevents.1722987362.232a0f8c3879

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5cadaf9fc16eb9868f897a08c97a0dd97b266292664e203e148b3842da9d8721
+size 520