optimum-neuron/v0.3.0/en/quickstart.html

<meta charset="utf-8" /><meta name="hf:doc:metadata" content="{&quot;title&quot;:&quot;Quickstart&quot;,&quot;local&quot;:&quot;quickstart&quot;,&quot;sections&quot;:[{&quot;title&quot;:&quot;Training&quot;,&quot;local&quot;:&quot;training&quot;,&quot;sections&quot;:[],&quot;depth&quot;:2},{&quot;title&quot;:&quot;Inference&quot;,&quot;local&quot;:&quot;inference&quot;,&quot;sections&quot;:[],&quot;depth&quot;:2}],&quot;depth&quot;:1}">
		<link href="/docs/optimum.neuron/v0.3.0/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload">
		<link rel="modulepreload" href="/docs/optimum.neuron/v0.3.0/en/_app/immutable/entry/start.6799d766.js">
		<link rel="modulepreload" href="/docs/optimum.neuron/v0.3.0/en/_app/immutable/chunks/scheduler.85c25b89.js">
		<link rel="modulepreload" href="/docs/optimum.neuron/v0.3.0/en/_app/immutable/chunks/singletons.6d87408b.js">
		<link rel="modulepreload" href="/docs/optimum.neuron/v0.3.0/en/_app/immutable/chunks/paths.19a3cd60.js">
		<link rel="modulepreload" href="/docs/optimum.neuron/v0.3.0/en/_app/immutable/entry/app.85106e1c.js">
		<link rel="modulepreload" href="/docs/optimum.neuron/v0.3.0/en/_app/immutable/chunks/index.c9bcf812.js">
		<link rel="modulepreload" href="/docs/optimum.neuron/v0.3.0/en/_app/immutable/nodes/0.5ac349ef.js">
		<link rel="modulepreload" href="/docs/optimum.neuron/v0.3.0/en/_app/immutable/chunks/each.e59479a4.js">
		<link rel="modulepreload" href="/docs/optimum.neuron/v0.3.0/en/_app/immutable/nodes/43.16378720.js">
		<link rel="modulepreload" href="/docs/optimum.neuron/v0.3.0/en/_app/immutable/chunks/CodeBlock.c004bd26.js">
		<link rel="modulepreload" href="/docs/optimum.neuron/v0.3.0/en/_app/immutable/chunks/getInferenceSnippets.5ea0a804.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{&quot;title&quot;:&quot;Quickstart&quot;,&quot;local&quot;:&quot;quickstart&quot;,&quot;sections&quot;:[{&quot;title&quot;:&quot;Training&quot;,&quot;local&quot;:&quot;training&quot;,&quot;sections&quot;:[],&quot;depth&quot;:2},{&quot;title&quot;:&quot;Inference&quot;,&quot;local&quot;:&quot;inference&quot;,&quot;sections&quot;:[],&quot;depth&quot;:2}],&quot;depth&quot;:1}"><!-- HEAD_svelte-u9bgzb_END -->      <p></p>   <h1 class="relative group"><a id="quickstart" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#quickstart"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Quickstart</span></h1> <p data-svelte-h="svelte-10oetrf">🤗 Optimum Neuron was designed with one goal in mind: <strong>to make training and inference straightforward for any 🤗 Transformers user while leveraging the complete power of AWS Accelerators</strong>.</p>  <h2 class="relative group"><a id="training" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#training"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Training</span></h2> <p data-svelte-h="svelte-xz831e">There are three main classes one needs to know:</p> <ul data-svelte-h="svelte-ffh3c9"><li>NeuronArgumentParser: inherits the original <a href="https://huggingface.co/docs/transformers/main/en/internal/trainer_utils#transformers.HfArgumentParser" rel="nofollow">HfArgumentParser</a> in Transformers with additional checks on the argument values to make sure that they will work well with AWS Trainium instances.</li> <li><a href="https://huggingface.co/docs/optimum/neuron/package_reference/trainer#optimum.neuron.NeuronTrainingArguments" rel="nofollow">NeuronTrainingArguments</a>: extends the original TrainingArguments with Trainium-specific parameters to optimize performance on AWS Trainium hardware.</li> <li><a href="https://huggingface.co/docs/optimum/neuron/package_reference/trainer" rel="nofollow">NeuronTrainer</a>: the trainer class that takes care of compiling and distributing the model to run on Trainium Chips, and performing training and evaluation.</li></ul> <p data-svelte-h="svelte-gadnp8">The <a href="https://huggingface.co/docs/optimum/neuron/package_reference/trainer" rel="nofollow">NeuronTrainer</a> is very similar to the <a href="https://huggingface.co/docs/transformers/main_classes/trainer" rel="nofollow">🤗 Transformers Trainer</a>, and adapting a script for Trainium instances will mostly consist in:</p> <ul data-svelte-h="svelte-fj876f"><li>Swapping the <code>Trainer</code> class for the <code>NeuronTrainer</code> one</li> <li>Using <code>NeuronTrainingArguments</code> instead of <code>TrainingArguments</code></li></ul> <p data-svelte-h="svelte-14gwwhs">That’s how most of the <a href="https://github.com/huggingface/optimum-neuron/tree/main/examples" rel="nofollow">example scripts</a> were adapted from their <a href="https://github.com/huggingface/transformers/tree/main/examples/pytorch" rel="nofollow">original counterparts</a>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5   text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-deletion">-from transformers import TrainingArguments</span>
<span class="hljs-addition">+from optimum.neuron import NeuronTrainingArguments</span>
<span class="hljs-deletion">-from transformers import Trainer</span>
<span class="hljs-addition">+from optimum.neuron import NeuronTrainer as Trainer</span>
<span class="hljs-deletion">-training_args = TrainingArguments(</span>
<span class="hljs-addition">+training_args = NeuronTrainingArguments(</span>
  # training arguments...
)
# A lot of code here
# Initialize our Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset if training_args.do_train else None,
    eval_dataset=eval_dataset if training_args.do_eval else None,
    compute_metrics=compute_metrics,
    tokenizer=tokenizer,
    data_collator=data_collator,
)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1083c5c">All Trainium instances come at least with 2 Neuron Cores. To leverage those we need to launch the training with <code>torchrun</code>.
Below you see an example of how to launch a training script on a <code>trn1.2xlarge</code> instance using a <code>bert-base-uncased</code> model.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5   text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->torchrun --nproc_per_node=2 huggingface-neuron-samples/text-classification/run_glue.py \
--model_name_or_path bert-base-uncased \
--dataset_name philschmid/emotion \
--do_train \
--do_eval \
--bf16 True \
--per_device_train_batch_size 16 \
--learning_rate 5e-5 \
--num_train_epochs 3 \
--output_dir ./bert-emotion<!-- HTML_TAG_END --></pre></div>  <h2 class="relative group"><a id="inference" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#inference"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Inference</span></h2> <p data-svelte-h="svelte-xknj1b">You can compile and export your 🤗 Transformers models to a serialized format before inference on Neuron devices:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5   text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->optimum-cli <span class="hljs-built_in">export</span> neuron \
  --model distilbert-base-uncased-finetuned-sst-2-english \
  --batch_size 1 \
  --sequence_length 32 \
  --auto_cast matmul \
  --auto_cast_type bf16 \
  distilbert_base_uncased_finetuned_sst2_english_neuron/<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-ltzpcv">The command above will export <code>distilbert-base-uncased-finetuned-sst-2-english</code> with static shapes: <code>batch_size=1</code> and <code>sequence_length=32</code>, and cast all <code>matmul</code> operations from FP32 to BF16. Check out the <a href="https://huggingface.co/docs/optimum-neuron/guides/export_model#exporting-a-model-to-neuron-using-the-cli" rel="nofollow">exporter guide</a> for more compilation options.</p> <p data-svelte-h="svelte-6pc1df">Then you can run the exported Neuron model on Neuron devices with <code>NeuronModelForXXX</code> classes which are similar to <code>AutoModelForXXX</code> classes in 🤗 Transformers:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5   text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->from transformers import AutoTokenizer
<span class="hljs-deletion">-from transformers import AutoModelForSequenceClassification</span>
<span class="hljs-addition">+from optimum.neuron import NeuronModelForSequenceClassification</span>

# PyTorch checkpoint
<span class="hljs-deletion">-model = AutoModelForSequenceClassification.from_pretrained(&quot;distilbert-base-uncased-finetuned-sst-2-english&quot;)</span>
<span class="hljs-addition">+model = NeuronModelForSequenceClassification.from_pretrained(&quot;distilbert_base_uncased_finetuned_sst2_english_neuron&quot;)</span>

tokenizer = AutoTokenizer.from_pretrained(&quot;distilbert-base-uncased-finetuned-sst-2-english&quot;)
inputs = tokenizer(&quot;Hamilton is considered to be the best musical of past years.&quot;, return_tensors=&quot;pt&quot;)

logits = model(**inputs).logits
print(model.config.id2label[logits.argmax().item()])
# &#x27;POSITIVE&#x27;<!-- HTML_TAG_END --></pre></div>  <p></p> 
			
			<script>
				{
					__sveltekit_1cneu7z = {
						assets: "/docs/optimum.neuron/v0.3.0/en",
						base: "/docs/optimum.neuron/v0.3.0/en",
						env: {}
					};

					const element = document.currentScript.parentElement;

					const data = [null,null];

					Promise.all([
						import("/docs/optimum.neuron/v0.3.0/en/_app/immutable/entry/start.6799d766.js"),
						import("/docs/optimum.neuron/v0.3.0/en/_app/immutable/entry/app.85106e1c.js")
					]).then(([kit, app]) => {
						kit.start(app, element, {
							node_ids: [0, 43],
							data,
							form: null,
							error: null
						});
					});
				}
			</script>