Add model card for DAA CLIPTextModel component with metadata and sample usage

#1
by nielsr HF Staff - opened
Files changed (1) hide show
  1. README.md +54 -0
README.md ADDED
@@ -0,0 +1,54 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ pipeline_tag: text-to-image
3
+ library_name: transformers
4
+ license: apache-2.0
5
+ ---
6
+
7
+ This repository hosts a `CLIPTextModel` component, which is integral to the research presented in the paper [Dynamic Attention Analysis for Backdoor Detection in Text-to-Image Diffusion Models](https://huggingface.co/papers/2504.20518).
8
+
9
+ The study introduces a novel backdoor detection perspective named **Dynamic Attention Analysis (DAA)**, showing that the **dynamic feature in attention maps** can serve as a much better indicator for backdoor detection in text-to-image diffusion models.
10
+
11
+ For the full codebase and further details on the Dynamic Attention Analysis (DAA) method, please refer to the [GitHub Repository](https://github.com/Robin-WZQ/DAA).
12
+
13
+ ## ๐Ÿ‘€ Overview
14
+
15
+ <div align=center>
16
+ <img src='https://github.com/Robin-WZQ/DAA/blob/main/viz/Overview.png' width=800>
17
+ </div>
18
+
19
+ The overview of our Dynamic Attention Analysis (DAA). **(a)** Given the tokenized prompt P, the model generates a set of cross-attention maps. **(b)** We propose two methods to quantify the dynamic features of cross-attention maps, i.e., DAA-I and DAA-S. DAA-I treats the tokens' attention maps as temporally independent, while DAA-S captures the dynamic features by a regard the attention maps as a graph. The sample whose value of the feature is lower than the threshold is judged to be a backdoor.
20
+
21
+ <div align=center>
22
+ <img src='https://github.com/Robin-WZQ/DAA/blob/main/viz/Evolve.svg' width=450>
23
+ </div>
24
+
25
+ The average relative evolution trajectories of the `<EOS>` token in benign samples (the orange line) and backdoor samples (the blue line). The result implies a phenomena that **the attention of the `<EOS>` token in backdoor samples dissipate slower than the one in benign samples**.
26
+
27
+ ## ๐Ÿƒ๐Ÿผ Running Scripts: Sample Usage for Detection
28
+
29
+ Here are examples of how to use DAA for detecting a sample (text as input), as found in the [GitHub README](https://github.com/Robin-WZQ/DAA#running-scripts):
30
+
31
+ **For detecting a sample (text as input):**
32
+ - DAA-I
33
+ ```bash
34
+ python detect_daai_uni.py --input_text "blonde man with glasses near beach" --backdoor_model_name "Rickrolling" --backdoor_model_path "./model/train/poisoned_model"
35
+ python detect_daai_uni.py --input_text "ัด blonde man with glasses near beach" --backdoor_model_name "Rickrolling" --backdoor_model_path "./model/train/poisoned_model"
36
+ ```
37
+ - DAA-S
38
+ ```bash
39
+ python detect_daas_uni.py --input_text "blonde man with glasses near beach" --backdoor_model_name "Rickrolling" --backdoor_model_path "./model/train/poisoned_model"
40
+ python detect_daas_uni.py --input_text "ัด blonde man with glasses near beach" --backdoor_model_name "Rickrolling" --backdoor_model_path "./model/train/poisoned_model"
41
+ ```
42
+
43
+ ## ๐Ÿ“„ Citation
44
+
45
+ If you find this project useful in your research, please consider citing:
46
+
47
+ ```bibtex
48
+ @article{wang2025dynamicattentionanalysisbackdoor,
49
+ title={Dynamic Attention Analysis for Backdoor Detection in Text-to-Image Diffusion Models},
50
+ author={Zhongqi Wang and Jie Zhang and Shiguang Shan and Xilin Chen},
51
+ journal={IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)},
52
+ year={2025},
53
+ }
54
+ ```