Initial commit: Hugging Face Model Ecosystem Navigator

- Interactive latent space visualization for 1.86M models
- Plotly + Gradio implementation for Hugging Face Spaces
- React + Visx implementation for custom deployment
- Embedding generation with sentence transformers
- UMAP dimensionality reduction
- Model detail modals with Hugging Face links
- Paper: Anatomy of a Machine Learning Ecosystem (arXiv:2508.06811)

.gitignore

@@ -0,0 +1,38 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+env/
+venv/
+ENV/
+.venv
+
+# Caches
+*.pkl
+*.pickle
+*.cache
+embeddings_cache.pkl
+reduced_embeddings_cache.pkl
+*.npy
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Gradio
+flagged/
+
+# Data
+*.parquet
+*.csv
+data/
+

.nvmrc

@@ -0,0 +1,2 @@
+18
+

README.md

@@ -0,0 +1,164 @@
+# Anatomy of a Machine Learning Ecosystem: 2 Million Models on Hugging Face
+
+**Authors:** Benjamin Laufer, Hamidah Oderinwale, Jon Kleinberg
+
+**Research Paper**: [arXiv:2508.06811](https://arxiv.org/abs/2508.06811)
+
+## Abstract
+
+Many have observed that the development and deployment of generative machine learning (ML) and artificial intelligence (AI) models follow a distinctive pattern in which pre-trained models are adapted and fine-tuned for specific downstream tasks. However, there is limited empirical work that examines the structure of these interactions. This paper analyzes 1.86 million models on Hugging Face, a leading peer production platform for model development. Our study of model family trees -- networks that connect fine-tuned models to their base or parent -- reveals sprawling fine-tuning lineages that vary widely in size and structure. Using an evolutionary biology lens to study ML models, we use model metadata and model cards to measure the genetic similarity and mutation of traits over model families. We find that models tend to exhibit a family resemblance, meaning their genetic markers and traits exhibit more overlap when they belong to the same model family. However, these similarities depart in certain ways from standard models of asexual reproduction, because mutations are fast and directed, such that two `sibling' models tend to exhibit more similarity than parent/child pairs. Further analysis of the directional drifts of these mutations reveals qualitative insights about the open machine learning ecosystem: Licenses counter-intuitively drift from restrictive, commercial licenses towards permissive or copyleft licenses, often in violation of upstream license's terms; models evolve from multi-lingual compatibility towards english-only compatibility; and model cards reduce in length and standardize by turning, more often, to templates and automatically generated text. Overall, this work takes a step toward an empirically grounded understanding of model fine-tuning and suggests that ecological models and methods can yield novel scientific insights.
+
+## About This Tool
+
+This interactive latent space navigator visualizes ~1.84M models from the [modelbiome/ai_ecosystem_withmodelcards](https://huggingface.co/datasets/modelbiome/ai_ecosystem_withmodelcards) dataset in a 2D space where similar models appear closer together, allowing you to explore the relationships and family structures described in the paper.
+
+## Features
+
+- **Latent Space Visualization**: 2D embedding visualization showing model relationships
+- **Interactive Exploration**: Hover, click, and zoom to explore models
+- **Smart Filtering**: Filter by library, pipeline tag, popularity, and more
+- **Color & Size Encoding**: Visualize different attributes through color and size
+- **Caching**: Efficient caching of embeddings and reduced dimensions
+- **Performance Optimized**: Handles large datasets through smart sampling
+
+## Quick Start
+
+### Option 1: Plotly + Gradio (Hugging Face Spaces)
+
+```bash
+pip install -r requirements.txt
+python app.py
+```
+
+### Option 2: Visx + React (Netlify Deployment)
+
+For Netlify deployment, deploy the frontend to Netlify and the backend to Railway or Render. Set the `REACT_APP_API_URL` environment variable to your backend URL.
+
+## Installation
+
+```bash
+pip install -r requirements.txt
+```
+
+## Usage
+
+### Local Development
+
+```bash
+python app.py
+```
+
+Or use the test script:
+
+```bash
+python test_local.py
+```
+
+The app will:
+1. Load a sample of 10,000 models from the dataset
+2. Generate embeddings (first run takes ~2-3 minutes)
+3. Reduce dimensions using UMAP
+4. Launch a Gradio interface at `http://localhost:7860`
+
+### Using the Interface
+
+1. **Filters**: Use the left sidebar to filter models by:
+   - Search query (model ID or tags)
+   - Minimum downloads
+   - Minimum likes
+   - Color mapping (library, pipeline, popularity)
+   - Size mapping (downloads, likes, trending score)
+
+2. **Exploration**: 
+   - Hover over points to see model information
+   - Zoom and pan to explore different regions
+   - Use the legend to understand color coding
+
+3. **Understanding the Space**:
+   - Models closer together are more similar
+   - Similarity is based on tags, pipeline type, library, and model card content
+
+## Deployment
+
+### Hugging Face Spaces
+
+1. Create a new Space on Hugging Face
+2. Push this repository to the Space
+3. Ensure `requirements.txt` and `app.py` are in the root
+4. The app will automatically:
+   - Load the dataset from Hugging Face Hub
+   - Generate embeddings on first run (cached afterwards)
+   - Serve the interface via Gradio
+
+**Note**: First load may take 2-3 minutes for embedding generation. Subsequent loads will be faster due to caching.
+
+### Netlify (React Frontend)
+
+1. Deploy frontend to Netlify (set base directory to `frontend`)
+2. Deploy backend to Railway/Render (set root directory to `backend`)
+3. Set `REACT_APP_API_URL` environment variable in Netlify to your backend URL
+4. Update CORS in backend to include your Netlify URL
+
+## Architecture
+
+### Current Implementation (Plotly + Gradio)
+
+- **Data Loading** (`data_loader.py`): Loads dataset from Hugging Face Hub, handles filtering and preprocessing
+- **Embedding Generation** (`embeddings.py`): Creates embeddings from model metadata using sentence transformers
+- **Dimensionality Reduction** (`dimensionality_reduction.py`): Uses UMAP to reduce to 2D for visualization
+- **Main App** (`app.py`): Gradio interface with Plotly visualizations
+
+### Alternative Implementation (Visx + React)
+
+For better performance and customization, see the `frontend/` and `backend/` directories for a React + Visx implementation:
+
+- **Backend** (`backend/api.py`): FastAPI server serving model data
+- **Frontend** (`frontend/`): React app with Visx visualizations
+
+### Comparison with Hugging Face Dataset Viewer
+
+This project uses a different approach than Hugging Face's built-in dataset viewer:
+
+- **HF Dataset Viewer**: Tabular browser for exploring dataset rows (see [dataset-viewer](https://github.com/huggingface/dataset-viewer))
+- **This Project**: Latent space visualization showing semantic relationships between models
+
+The HF viewer is optimized for browsing data structure, while this tool focuses on understanding model relationships through embeddings and spatial visualization.
+
+## Design Decisions
+
+The application uses:
+- **2D visualization** for simplicity and accessibility
+- **UMAP** for dimensionality reduction (better global structure than t-SNE)
+- **Sentence transformers** for efficient embedding generation
+- **Smart sampling** to maintain interactivity with large datasets
+- **Caching** to avoid recomputation on filter changes
+
+## Performance Notes
+
+- **Initial Sample**: 10,000 models (configurable in `app.py`)
+- **Visualization Limit**: Maximum 5,000 points for smooth interaction
+- **Embedding Model**: `all-MiniLM-L6-v2` (good balance of quality and speed)
+- **Caching**: Embeddings and reduced dimensions are cached to disk
+
+## Requirements
+
+- Python 3.8+
+- ~2-4GB RAM for 10K models
+- Internet connection for dataset download
+- Optional: GPU for faster embedding generation (not required)
+
+## Citation
+
+If you use this tool or dataset, please cite:
+
+```bibtex
+@article{laufer2025anatomy,
+  title={Anatomy of a Machine Learning Ecosystem: 2 Million Models on Hugging Face},
+  author={Laufer, Benjamin and Oderinwale, Hamidah and Kleinberg, Jon},
+  journal={arXiv preprint arXiv:2508.06811},
+  year={2025},
+  url={https://arxiv.org/abs/2508.06811}
+}
+```
+
+**Paper**: [arXiv:2508.06811](https://arxiv.org/abs/2508.06811)

app.py

@@ -0,0 +1,451 @@
+"""
+Main Gradio application for the Hugging Face Model Ecosystem Navigator.
+"""
+import gradio as gr
+import plotly.graph_objects as go
+import plotly.express as px
+import pandas as pd
+import numpy as np
+from typing import Optional, Tuple
+import os
+
+from data_loader import ModelDataLoader
+from embeddings import ModelEmbedder
+from dimensionality_reduction import DimensionReducer
+
+
+class ModelNavigatorApp:
+    """Main application class for the model navigator."""
+    
+    def __init__(self):
+        self.data_loader = ModelDataLoader()
+        self.embedder: Optional[ModelEmbedder] = None
+        self.reducer: Optional[DimensionReducer] = None
+        self.df: Optional[pd.DataFrame] = None
+        self.embeddings: Optional[np.ndarray] = None
+        self.reduced_embeddings: Optional[np.ndarray] = None
+        self.current_filtered_df: Optional[pd.DataFrame] = None
+        
+    def load_initial_data(self, sample_size: int = 10000):
+        """Load initial sample of data."""
+        print("Loading initial data...")
+        self.df = self.data_loader.load_data(sample_size=sample_size)
+        self.df = self.data_loader.preprocess_for_embedding(self.df)
+        return f"Loaded {len(self.df)} models"
+    
+    def generate_visualization(
+        self,
+        color_by: str = "library_name",
+        size_by: str = "downloads",
+        min_downloads: int = 0,
+        min_likes: int = 0,
+        search_query: str = "",
+        selected_libraries: list = None,
+        selected_pipeline_tags: list = None,
+        use_cache: bool = True
+    ) -> Tuple[go.Figure, pd.DataFrame]:
+        """
+        Generate interactive visualization.
+        
+        Returns:
+            Plotly figure and filtered dataframe
+        """
+        if self.df is None or len(self.df) == 0:
+            return go.Figure(), pd.DataFrame()
+        
+        # Filter data
+        filtered_df = self.data_loader.filter_data(
+            df=self.df,
+            min_downloads=min_downloads,
+            min_likes=min_likes,
+            libraries=selected_libraries if selected_libraries else None,
+            pipeline_tags=selected_pipeline_tags if selected_pipeline_tags else None,
+            search_query=search_query if search_query else None
+        )
+        
+        if len(filtered_df) == 0:
+            empty_fig = go.Figure()
+            empty_fig.add_annotation(
+                text="No models match the selected filters",
+                xref="paper", yref="paper",
+                x=0.5, y=0.5, showarrow=False
+            )
+            return empty_fig, filtered_df
+        
+        # Limit to reasonable size for performance
+        max_points = 5000
+        if len(filtered_df) > max_points:
+            filtered_df = filtered_df.sample(n=max_points, random_state=42)
+            print(f"Sampled {max_points} models for visualization")
+        
+        # Get indices for filtered data
+        filtered_indices = filtered_df.index.tolist()
+        
+        # Generate or load embeddings
+        cache_file = "embeddings_cache.pkl"
+        if use_cache and os.path.exists(cache_file) and self.embeddings is None:
+            try:
+                if self.embedder is None:
+                    self.embedder = ModelEmbedder()
+                self.embeddings = self.embedder.load_embeddings(cache_file)
+            except Exception as e:
+                print(f"Could not load cached embeddings: {e}")
+                pass
+        
+        if self.embeddings is None:
+            if self.embedder is None:
+                self.embedder = ModelEmbedder()
+            
+            # Generate embeddings for all data
+            texts = self.df['combined_text'].tolist()
+            self.embeddings = self.embedder.generate_embeddings(texts)
+            
+            if use_cache:
+                self.embedder.save_embeddings(self.embeddings, cache_file)
+        
+        # Get embeddings for filtered data
+        filtered_embeddings = self.embeddings[filtered_indices]
+        
+        # Reduce dimensions
+        if self.reducer is None:
+            self.reducer = DimensionReducer(method="umap", n_components=2)
+        
+        reduced_cache_file = "reduced_embeddings_cache.npy"
+        if use_cache and os.path.exists(reduced_cache_file):
+            try:
+                self.reduced_embeddings = np.load(reduced_cache_file, allow_pickle=True)
+                if len(self.reduced_embeddings) != len(self.df):
+                    self.reduced_embeddings = None
+            except Exception as e:
+                print(f"Could not load cached reduced embeddings: {e}")
+                pass
+        
+        if self.reduced_embeddings is None or len(self.reduced_embeddings) != len(self.df):
+            self.reduced_embeddings = self.reducer.fit_transform(self.embeddings)
+            if use_cache:
+                np.save(reduced_cache_file, self.reduced_embeddings)
+        
+        filtered_reduced = self.reduced_embeddings[filtered_indices]
+        
+        # Prepare data for plotting
+        plot_df = filtered_df.copy()
+        plot_df['x'] = filtered_reduced[:, 0]
+        plot_df['y'] = filtered_reduced[:, 1]
+        
+        # Color mapping
+        if color_by in plot_df.columns:
+            color_values = plot_df[color_by].fillna('Unknown')
+        else:
+            color_values = pd.Series(['All Models'] * len(plot_df))
+        
+        # Size mapping
+        if size_by and size_by != "None" and size_by in plot_df.columns:
+            size_values = plot_df[size_by].fillna(0)
+            # Normalize sizes
+            if size_values.max() > 0:
+                size_values = 5 + 15 * (size_values / size_values.max())
+            else:
+                size_values = pd.Series([10] * len(plot_df))
+        else:
+            size_values = pd.Series([10] * len(plot_df))
+        
+        # Create hover text
+        hover_texts = []
+        for idx, row in plot_df.iterrows():
+            hover = f"<b>{row.get('model_id', 'Unknown')}</b><br>"
+            hover += f"Library: {row.get('library_name', 'N/A')}<br>"
+            hover += f"Pipeline: {row.get('pipeline_tag', 'N/A')}<br>"
+            hover += f"Downloads: {row.get('downloads', 0):,}<br>"
+            hover += f"Likes: {row.get('likes', 0):,}"
+            hover_texts.append(hover)
+        
+        # Create plotly figure
+        fig = go.Figure()
+        
+        # Store model IDs with indices for click handling
+        model_id_map = {i: row.get('model_id', 'Unknown') for i, row in plot_df.iterrows()}
+        
+        # Group by color if categorical
+        is_categorical = len(color_values) > 0 and isinstance(color_values.iloc[0], str)
+        
+        if is_categorical and color_by in plot_df.columns:
+            unique_colors = color_values.unique()
+            colors = px.colors.qualitative.Set3 + px.colors.qualitative.Pastel
+            color_map = {val: colors[i % len(colors)] for i, val in enumerate(unique_colors)}
+            
+            for color_val in unique_colors:
+                mask = color_values == color_val
+                subset_df = plot_df[mask]
+                subset_hover = [hover_texts[i] for i, m in enumerate(mask) if m]
+                subset_sizes = size_values[mask]
+                
+                # Create customdata with model IDs for click handling
+                subset_customdata = [
+                    [int(idx), str(row.get('model_id', 'Unknown'))] 
+                    for idx, row in subset_df.iterrows()
+                ]
+                
+                fig.add_trace(go.Scatter(
+                    x=subset_df['x'],
+                    y=subset_df['y'],
+                    mode='markers',
+                    name=str(color_val)[:30],  # Truncate long names
+                    marker=dict(
+                        size=subset_sizes.values,
+                        color=color_map[color_val],
+                        opacity=0.7,
+                        line=dict(width=0.5, color='white')
+                    ),
+                    text=subset_df['model_id'].tolist(),
+                    customdata=subset_customdata,
+                    hovertemplate='%{text}<br>Click for details<extra></extra>',
+                    showlegend=True
+                ))
+        else:
+            # Continuous color scale
+            customdata = [
+                [int(idx), str(row.get('model_id', 'Unknown'))] 
+                for idx, row in plot_df.iterrows()
+            ]
+            
+            fig.add_trace(go.Scatter(
+                x=plot_df['x'],
+                y=plot_df['y'],
+                mode='markers',
+                marker=dict(
+                    size=size_values.values,
+                    color=color_values.values,
+                    colorscale='Viridis',
+                    opacity=0.7,
+                    line=dict(width=0.5, color='white'),
+                    colorbar=dict(title=color_by)
+                ),
+                text=plot_df['model_id'].tolist(),
+                customdata=customdata,
+                hovertemplate='%{text}<br>Click for details<extra></extra>',
+                showlegend=False
+            ))
+        
+        # Update layout
+        fig.update_layout(
+            title={
+                'text': f'Model Latent Space Navigator ({len(plot_df)} models)',
+                'x': 0.5,
+                'xanchor': 'center'
+            },
+            xaxis_title="Dimension 1",
+            yaxis_title="Dimension 2",
+            hovermode='closest',
+            template='plotly_white',
+            height=700,
+            clickmode='event+select'
+        )
+        
+        return fig, filtered_df
+    
+    def get_model_details(self, model_id: str) -> str:
+        """Get detailed information about a model."""
+        if self.df is None:
+            return "No data loaded"
+        
+        model = self.df[self.df.get('model_id', '') == model_id]
+        if len(model) == 0:
+            return f"Model '{model_id}' not found"
+        
+        model = model.iloc[0]
+        
+        details = f"# {model.get('model_id', 'Unknown')}\n\n"
+        details += f"**Library:** {model.get('library_name', 'N/A')}\n\n"
+        details += f"**Pipeline Tag:** {model.get('pipeline_tag', 'N/A')}\n\n"
+        details += f"**Downloads:** {model.get('downloads', 0):,}\n\n"
+        details += f"**Likes:** {model.get('likes', 0):,}\n\n"
+        details += f"**Trending Score:** {model.get('trendingScore', 'N/A')}\n\n"
+        
+        if pd.notna(model.get('tags')):
+            details += f"**Tags:** {model.get('tags', '')}\n\n"
+        
+        if pd.notna(model.get('licenses')):
+            details += f"**License:** {model.get('licenses', '')}\n\n"
+        
+        if pd.notna(model.get('parent_model')):
+            details += f"**Parent Model:** {model.get('parent_model', '')}\n\n"
+        
+        return details
+
+
+def create_interface():
+    """Create and launch Gradio interface."""
+    app = ModelNavigatorApp()
+    
+    # Load initial data
+    status = app.load_initial_data(sample_size=10000)
+    print(status)
+    
+    with gr.Blocks(title="Anatomy of a Machine Learning Ecosystem", theme=gr.themes.Soft()) as demo:
+        gr.Markdown("""
+        # Anatomy of a Machine Learning Ecosystem: 2 Million Models on Hugging Face
+        
+        **Authors:** Benjamin Laufer, Hamidah Oderinwale, Jon Kleinberg
+        
+        Many have observed that the development and deployment of generative machine learning (ML) and artificial intelligence (AI) models follow a distinctive pattern in which pre-trained models are adapted and fine-tuned for specific downstream tasks. However, there is limited empirical work that examines the structure of these interactions. This paper analyzes 1.86 million models on Hugging Face, a leading peer production platform for model development. Our study of model family trees -- networks that connect fine-tuned models to their base or parent -- reveals sprawling fine-tuning lineages that vary widely in size and structure. Using an evolutionary biology lens to study ML models, we use model metadata and model cards to measure the genetic similarity and mutation of traits over model families.
+        
+        **Read the full paper**: [arXiv:2508.06811](https://arxiv.org/abs/2508.06811)
+        
+        ---
+        
+        **How to use this navigator:**
+        - Adjust filters to explore different subsets of models
+        - Hover over points to see model information
+        - Use color and size options to highlight different attributes
+        - Similar models appear closer together in the latent space
+        - Models are positioned based on their similarity (tags, pipeline, library, and model card content)
+        """)
+        
+        with gr.Row():
+            with gr.Column(scale=1):
+                gr.Markdown("### Filters")
+                
+                search_query = gr.Textbox(
+                    label="Search Models",
+                    placeholder="Search by model ID or tags...",
+                    value=""
+                )
+                
+                min_downloads = gr.Slider(
+                    label="Min Downloads",
+                    minimum=0,
+                    maximum=1000000,
+                    value=0,
+                    step=1000
+                )
+                
+                min_likes = gr.Slider(
+                    label="Min Likes",
+                    minimum=0,
+                    maximum=10000,
+                    value=0,
+                    step=10
+                )
+                
+                color_by = gr.Dropdown(
+                    label="Color By",
+                    choices=["library_name", "pipeline_tag", "downloads", "likes"],
+                    value="library_name"
+                )
+                
+                size_by = gr.Dropdown(
+                    label="Size By",
+                    choices=["downloads", "likes", "trendingScore", "None"],
+                    value="downloads"
+                )
+                
+                update_btn = gr.Button("Update Visualization", variant="primary")
+            
+            with gr.Column(scale=3):
+                plot = gr.Plot(label="Model Latent Space")
+                model_details = gr.Markdown(
+                    value="**Instructions:** Use the filters above to explore models. Hover over points to see details, **click on a point** to view full model information and link to Hugging Face.",
+                    label="Model Details"
+                )
+        
+        def handle_plot_click(evt: gr.SelectData):
+            """Handle plot click and show model details."""
+            if evt is None or app.df is None:
+                return "**Click on a model point to see details**"
+            
+            try:
+                # Get the point index from the click event
+                point_idx = evt.index
+                if point_idx is None:
+                    return "**Click on a model point to see details**"
+                
+                # Get the current filtered dataframe
+                if app.current_filtered_df is not None and len(app.current_filtered_df) > 0:
+                    filtered_df = app.current_filtered_df
+                else:
+                    # Fallback: use the full dataframe
+                    filtered_df = app.df
+                
+                # Limit to max_points if needed
+                if len(filtered_df) > 5000:
+                    filtered_df = filtered_df.sample(n=5000, random_state=42)
+                
+                if point_idx < len(filtered_df):
+                    model_row = filtered_df.iloc[point_idx]
+                    model_id = model_row.get('model_id', 'Unknown')
+                    
+                    # Get full model details from the original dataframe
+                    model = app.df[app.df.get('model_id', '') == model_id]
+                    if len(model) == 0:
+                        return f"**Model not found:** {model_id}"
+                    
+                    model = model.iloc[0]
+                    hf_url = f"https://huggingface.co/{model_id}"
+                    
+                    details = f"""# {model_id}
+
+**[View on Hugging Face]({hf_url})**
+
+## Model Information
+
+- **Library:** {model.get('library_name', 'N/A')}
+- **Pipeline Tag:** {model.get('pipeline_tag', 'N/A')}
+- **Downloads:** {model.get('downloads', 0):,}
+- **Likes:** {model.get('likes', 0):,}
+"""
+                    if pd.notna(model.get('trendingScore')):
+                        details += f"- **Trending Score:** {model.get('trendingScore', 0):.2f}\n\n"
+                    else:
+                        details += "\n"
+                    
+                    if pd.notna(model.get('tags')):
+                        details += f"**Tags:** {model.get('tags', '')}\n\n"
+                    if pd.notna(model.get('licenses')):
+                        details += f"**License:** {model.get('licenses', '')}\n\n"
+                    if pd.notna(model.get('parent_model')):
+                        details += f"**Parent Model:** {model.get('parent_model', '')}\n\n"
+                    
+                    return details
+                else:
+                    return f"**Point index out of range:** {point_idx}"
+            except Exception as e:
+                import traceback
+                return f"**Error loading model details:**\n```\n{str(e)}\n{traceback.format_exc()}\n```"
+            
+            return "**Click on a model point to see details**"
+        
+        def update_plot_and_store(color_by_val, size_by_val, min_dl, min_lk, search):
+            fig, df = app.generate_visualization(
+                color_by=color_by_val,
+                size_by=size_by_val,
+                min_downloads=int(min_dl),
+                min_likes=int(min_lk),
+                search_query=search
+            )
+            # Store the filtered dataframe for click handling
+            app.current_filtered_df = df
+            return fig
+        
+        update_btn.click(
+            fn=update_plot_and_store,
+            inputs=[color_by, size_by, min_downloads, min_likes, search_query],
+            outputs=plot
+        )
+        
+        # Handle plot clicks - Gradio's Plot component supports click events
+        plot.select(
+            fn=handle_plot_click,
+            outputs=model_details
+        )
+        
+        # Initial plot
+        initial_fig, initial_df = app.generate_visualization()
+        plot.value = initial_fig
+        app.current_filtered_df = initial_df
+    
+    return demo
+
+
+if __name__ == "__main__":
+    demo = create_interface()
+    demo.launch(share=False, server_name="0.0.0.0", server_port=7860)

backend/api.py

@@ -0,0 +1,194 @@
+"""
+FastAPI backend for serving model data to React/Visx frontend.
+"""
+from fastapi import FastAPI, HTTPException, Query
+from fastapi.middleware.cors import CORSMiddleware
+from typing import Optional, List
+import pandas as pd
+import numpy as np
+import os
+from pydantic import BaseModel
+
+from data_loader import ModelDataLoader
+from embeddings import ModelEmbedder
+from dimensionality_reduction import DimensionReducer
+
+app = FastAPI(title="HF Model Ecosystem API")
+
+# CORS middleware for React frontend
+# Update allow_origins with your Netlify URL in production
+# Note: Add your specific Netlify URL after deployment
+FRONTEND_URL = os.getenv("FRONTEND_URL", "http://localhost:3000")
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=[
+        "http://localhost:3000",  # Local development
+        FRONTEND_URL,  # Production frontend URL
+        # Add your Netlify URL here after deployment, e.g.:
+        # "https://your-app-name.netlify.app",
+    ],
+    allow_credentials=True,
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+# Global state
+data_loader = ModelDataLoader()
+embedder: Optional[ModelEmbedder] = None
+reducer: Optional[DimensionReducer] = None
+df: Optional[pd.DataFrame] = None
+embeddings: Optional[np.ndarray] = None
+reduced_embeddings: Optional[np.ndarray] = None
+
+
+class FilterParams(BaseModel):
+    min_downloads: int = 0
+    min_likes: int = 0
+    search_query: Optional[str] = None
+    libraries: Optional[List[str]] = None
+    pipeline_tags: Optional[List[str]] = None
+
+
+class ModelPoint(BaseModel):
+    model_id: str
+    x: float
+    y: float
+    library_name: Optional[str]
+    pipeline_tag: Optional[str]
+    downloads: int
+    likes: int
+    trending_score: Optional[float]
+    tags: Optional[str]
+
+
+@app.on_event("startup")
+async def startup_event():
+    """Initialize data and models on startup."""
+    global df, embedder, reducer
+    
+    print("Loading data...")
+    df = data_loader.load_data(sample_size=10000)
+    df = data_loader.preprocess_for_embedding(df)
+    
+    print("Initializing embedder...")
+    embedder = ModelEmbedder()
+    
+    print("Initializing reducer...")
+    reducer = DimensionReducer(method="umap", n_components=2)
+    
+    print("API ready!")
+
+
+@app.get("/")
+async def root():
+    return {"message": "HF Model Ecosystem API", "status": "running"}
+
+
+@app.get("/api/models", response_model=List[ModelPoint])
+async def get_models(
+    min_downloads: int = Query(0),
+    min_likes: int = Query(0),
+    search_query: Optional[str] = Query(None),
+    color_by: str = Query("library_name"),
+    size_by: str = Query("downloads"),
+    max_points: int = Query(5000)
+):
+    """
+    Get filtered models with 2D coordinates for visualization.
+    """
+    global df, embedder, reducer, embeddings, reduced_embeddings
+    
+    if df is None:
+        raise HTTPException(status_code=503, detail="Data not loaded")
+    
+    # Filter data
+    filtered_df = data_loader.filter_data(
+        df=df,
+        min_downloads=min_downloads,
+        min_likes=min_likes,
+        search_query=search_query,
+        libraries=None,  # Can be added as query params
+        pipeline_tags=None
+    )
+    
+    if len(filtered_df) == 0:
+        return []
+    
+    # Limit points
+    if len(filtered_df) > max_points:
+        filtered_df = filtered_df.sample(n=max_points, random_state=42)
+    
+    # Generate embeddings if needed
+    if embeddings is None:
+        texts = df['combined_text'].tolist()
+        embeddings = embedder.generate_embeddings(texts)
+    
+    # Reduce dimensions if needed
+    if reduced_embeddings is None:
+        reduced_embeddings = reducer.fit_transform(embeddings)
+    
+    # Get coordinates for filtered data
+    filtered_indices = filtered_df.index.tolist()
+    filtered_reduced = reduced_embeddings[filtered_indices]
+    
+    # Prepare response
+    models = []
+    for idx, (i, row) in enumerate(filtered_df.iterrows()):
+        models.append(ModelPoint(
+            model_id=row.get('model_id', 'Unknown'),
+            x=float(filtered_reduced[idx, 0]),
+            y=float(filtered_reduced[idx, 1]),
+            library_name=row.get('library_name'),
+            pipeline_tag=row.get('pipeline_tag'),
+            downloads=int(row.get('downloads', 0)),
+            likes=int(row.get('likes', 0)),
+            trending_score=float(row.get('trendingScore', 0)) if pd.notna(row.get('trendingScore')) else None,
+            tags=row.get('tags') if pd.notna(row.get('tags')) else None
+        ))
+    
+    return models
+
+
+@app.get("/api/stats")
+async def get_stats():
+    """Get dataset statistics."""
+    if df is None:
+        raise HTTPException(status_code=503, detail="Data not loaded")
+    
+    return {
+        "total_models": len(df),
+        "unique_libraries": df['library_name'].nunique() if 'library_name' in df.columns else 0,
+        "unique_pipelines": df['pipeline_tag'].nunique() if 'pipeline_tag' in df.columns else 0,
+        "avg_downloads": float(df['downloads'].mean()) if 'downloads' in df.columns else 0,
+        "avg_likes": float(df['likes'].mean()) if 'likes' in df.columns else 0
+    }
+
+
+@app.get("/api/model/{model_id}")
+async def get_model_details(model_id: str):
+    """Get detailed information about a specific model."""
+    if df is None:
+        raise HTTPException(status_code=503, detail="Data not loaded")
+    
+    model = df[df.get('model_id', '') == model_id]
+    if len(model) == 0:
+        raise HTTPException(status_code=404, detail="Model not found")
+    
+    model = model.iloc[0]
+    return {
+        "model_id": model.get('model_id'),
+        "library_name": model.get('library_name'),
+        "pipeline_tag": model.get('pipeline_tag'),
+        "downloads": int(model.get('downloads', 0)),
+        "likes": int(model.get('likes', 0)),
+        "trending_score": float(model.get('trendingScore', 0)) if pd.notna(model.get('trendingScore')) else None,
+        "tags": model.get('tags') if pd.notna(model.get('tags')) else None,
+        "licenses": model.get('licenses') if pd.notna(model.get('licenses')) else None,
+        "parent_model": model.get('parent_model') if pd.notna(model.get('parent_model')) else None
+    }
+
+
+if __name__ == "__main__":
+    import uvicorn
+    uvicorn.run(app, host="0.0.0.0", port=8000)
+

backend/requirements.txt

@@ -0,0 +1,12 @@
+fastapi>=0.104.0
+uvicorn[standard]>=0.24.0
+pydantic>=2.0.0
+pandas>=2.0.0
+numpy>=1.24.0
+sentence-transformers>=2.2.0
+umap-learn>=0.5.4
+scikit-learn>=1.3.0
+datasets>=2.14.0
+huggingface-hub>=0.17.0
+tqdm>=4.66.0
+

data_loader.py

@@ -0,0 +1,128 @@
+"""
+Data loading and preprocessing for the Hugging Face model ecosystem dataset.
+"""
+import pandas as pd
+from datasets import load_dataset
+from typing import Optional, Dict, List
+import numpy as np
+
+
+class ModelDataLoader:
+    """Load and preprocess model data from Hugging Face dataset."""
+    
+    def __init__(self, dataset_name: str = "modelbiome/ai_ecosystem_withmodelcards"):
+        self.dataset_name = dataset_name
+        self.df: Optional[pd.DataFrame] = None
+        
+    def load_data(self, sample_size: Optional[int] = None, split: str = "train") -> pd.DataFrame:
+        """
+        Load dataset from Hugging Face Hub.
+        
+        Args:
+            sample_size: If provided, randomly sample this many rows
+            split: Dataset split to load
+            
+        Returns:
+            DataFrame with model data
+        """
+        print(f"Loading dataset {self.dataset_name}...")
+        dataset = load_dataset(self.dataset_name, split=split)
+        
+        if sample_size and len(dataset) > sample_size:
+            print(f"Sampling {sample_size} models from {len(dataset)} total...")
+            dataset = dataset.shuffle(seed=42).select(range(sample_size))
+        
+        self.df = dataset.to_pandas()
+        print(f"Loaded {len(self.df)} models")
+        
+        return self.df
+    
+    def preprocess_for_embedding(self, df: Optional[pd.DataFrame] = None) -> pd.DataFrame:
+        """
+        Preprocess data for embedding generation.
+        Combines text fields into a single representation.
+        
+        Args:
+            df: DataFrame to process (uses self.df if None)
+            
+        Returns:
+            DataFrame with combined text field
+        """
+        if df is None:
+            df = self.df.copy()
+        else:
+            df = df.copy()
+        
+        # Fill NaN values
+        text_fields = ['tags', 'pipeline_tag', 'library_name', 'modelCard']
+        for field in text_fields:
+            if field in df.columns:
+                df[field] = df[field].fillna('')
+        
+        # Combine text fields for embedding
+        df['combined_text'] = (
+            df.get('tags', '').astype(str) + ' ' +
+            df.get('pipeline_tag', '').astype(str) + ' ' +
+            df.get('library_name', '').astype(str) + ' ' +
+            df['modelCard'].astype(str).str[:500]  # Limit modelCard to first 500 chars
+        )
+        
+        return df
+    
+    def filter_data(
+        self,
+        df: Optional[pd.DataFrame] = None,
+        min_downloads: Optional[int] = None,
+        min_likes: Optional[int] = None,
+        libraries: Optional[List[str]] = None,
+        pipeline_tags: Optional[List[str]] = None,
+        search_query: Optional[str] = None
+    ) -> pd.DataFrame:
+        """
+        Filter dataset based on criteria.
+        
+        Args:
+            df: DataFrame to filter (uses self.df if None)
+            min_downloads: Minimum download count
+            min_likes: Minimum like count
+            libraries: List of library names to include
+            pipeline_tags: List of pipeline tags to include
+            search_query: Text search in model_id or tags
+            
+        Returns:
+            Filtered DataFrame
+        """
+        if df is None:
+            df = self.df.copy()
+        else:
+            df = df.copy()
+        
+        if min_downloads is not None:
+            df = df[df.get('downloads', 0) >= min_downloads]
+        
+        if min_likes is not None:
+            df = df[df.get('likes', 0) >= min_likes]
+        
+        if libraries:
+            df = df[df.get('library_name', '').isin(libraries)]
+        
+        if pipeline_tags:
+            df = df[df.get('pipeline_tag', '').isin(pipeline_tags)]
+        
+        if search_query:
+            query_lower = search_query.lower()
+            mask = (
+                df.get('model_id', '').astype(str).str.lower().str.contains(query_lower) |
+                df.get('tags', '').astype(str).str.lower().str.contains(query_lower)
+            )
+            df = df[mask]
+        
+        return df
+    
+    def get_unique_values(self, column: str) -> List[str]:
+        """Get unique non-null values from a column."""
+        if self.df is None:
+            return []
+        values = self.df[column].dropna().unique().tolist()
+        return sorted([str(v) for v in values if v and str(v) != 'nan'])
+

dimensionality_reduction.py

@@ -0,0 +1,78 @@
+"""
+Dimensionality reduction for visualization (UMAP, t-SNE).
+"""
+import numpy as np
+from umap import UMAP
+from sklearn.manifold import TSNE
+from typing import Optional
+import pickle
+import os
+
+
+class DimensionReducer:
+    """Reduce high-dimensional embeddings to 2D/3D for visualization."""
+    
+    def __init__(self, method: str = "umap", n_components: int = 2):
+        """
+        Initialize reducer.
+        
+        Args:
+            method: 'umap' or 'tsne'
+            n_components: Number of dimensions (2 or 3)
+        """
+        self.method = method.lower()
+        self.n_components = n_components
+        
+        if self.method == "umap":
+            self.reducer = UMAP(
+                n_components=n_components,
+                n_neighbors=15,
+                min_dist=0.1,
+                metric='cosine',
+                random_state=42
+            )
+        elif self.method == "tsne":
+            self.reducer = TSNE(
+                n_components=n_components,
+                perplexity=30,
+                random_state=42,
+                n_iter=1000
+            )
+        else:
+            raise ValueError(f"Unknown method: {method}. Use 'umap' or 'tsne'")
+    
+    def fit_transform(self, embeddings: np.ndarray) -> np.ndarray:
+        """
+        Fit reducer and transform embeddings.
+        
+        Args:
+            embeddings: High-dimensional embeddings (n_samples, embedding_dim)
+            
+        Returns:
+            Reduced embeddings (n_samples, n_components)
+        """
+        print(f"Reducing dimensions using {self.method.upper()}...")
+        reduced = self.reducer.fit_transform(embeddings)
+        print(f"Reduced to {self.n_components}D: shape {reduced.shape}")
+        return reduced
+    
+    def transform(self, embeddings: np.ndarray) -> np.ndarray:
+        """Transform new embeddings (only for UMAP, t-SNE doesn't support this)."""
+        if self.method == "umap":
+            return self.reducer.transform(embeddings)
+        else:
+            raise ValueError("t-SNE doesn't support transform. Use fit_transform instead.")
+    
+    def save_reducer(self, filepath: str):
+        """Save fitted reducer to disk."""
+        os.makedirs(os.path.dirname(filepath) if os.path.dirname(filepath) else '.', exist_ok=True)
+        with open(filepath, 'wb') as f:
+            pickle.dump(self.reducer, f)
+        print(f"Reducer saved to {filepath}")
+    
+    def load_reducer(self, filepath: str):
+        """Load fitted reducer from disk."""
+        with open(filepath, 'rb') as f:
+            self.reducer = pickle.load(f)
+        print(f"Reducer loaded from {filepath}")
+

embeddings.py

@@ -0,0 +1,71 @@
+"""
+Generate embeddings for models using sentence transformers.
+"""
+import numpy as np
+from sentence_transformers import SentenceTransformer
+from typing import List, Optional
+import pickle
+import os
+from tqdm import tqdm
+
+
+class ModelEmbedder:
+    """Generate embeddings for model descriptions."""
+    
+    def __init__(self, model_name: str = "all-MiniLM-L6-v2", cache_dir: Optional[str] = None):
+        """
+        Initialize embedder.
+        
+        Args:
+            model_name: Sentence transformer model name
+            cache_dir: Directory to cache embeddings
+        """
+        self.model_name = model_name
+        self.cache_dir = cache_dir
+        print(f"Loading embedding model: {model_name}...")
+        self.model = SentenceTransformer(model_name)
+        print("Embedding model loaded!")
+    
+    def generate_embeddings(
+        self,
+        texts: List[str],
+        batch_size: int = 32,
+        show_progress: bool = True
+    ) -> np.ndarray:
+        """
+        Generate embeddings for a list of texts.
+        
+        Args:
+            texts: List of text strings to embed
+            batch_size: Batch size for encoding
+            show_progress: Whether to show progress bar
+            
+        Returns:
+            numpy array of embeddings (n_samples, embedding_dim)
+        """
+        if show_progress:
+            print(f"Generating embeddings for {len(texts)} models...")
+        
+        embeddings = self.model.encode(
+            texts,
+            batch_size=batch_size,
+            show_progress_bar=show_progress,
+            convert_to_numpy=True
+        )
+        
+        return embeddings
+    
+    def save_embeddings(self, embeddings: np.ndarray, filepath: str):
+        """Save embeddings to disk."""
+        os.makedirs(os.path.dirname(filepath) if os.path.dirname(filepath) else '.', exist_ok=True)
+        with open(filepath, 'wb') as f:
+            pickle.dump(embeddings, f)
+        print(f"Embeddings saved to {filepath}")
+    
+    def load_embeddings(self, filepath: str) -> np.ndarray:
+        """Load embeddings from disk."""
+        with open(filepath, 'rb') as f:
+            embeddings = pickle.load(f)
+        print(f"Embeddings loaded from {filepath}")
+        return embeddings
+

frontend/.gitignore

@@ -0,0 +1,24 @@
+# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
+
+# dependencies
+/node_modules
+/.pnp
+.pnp.js
+
+# testing
+/coverage
+
+# production
+/build
+
+# misc
+.DS_Store
+.env.local
+.env.development.local
+.env.test.local
+.env.production.local
+
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+

frontend/.nvmrc

@@ -0,0 +1,2 @@
+18
+

frontend/_redirects

@@ -0,0 +1,2 @@
+/*    /index.html   200
+

frontend/netlify.toml

@@ -0,0 +1,13 @@
+[build]
+  publish = "build"
+  command = "npm run build"
+
+[build.environment]
+  NODE_VERSION = "18"
+  REACT_APP_API_URL = "https://your-backend-url.railway.app"
+
+[[redirects]]
+  from = "/*"
+  to = "/index.html"
+  status = 200
+

frontend/package.json

@@ -0,0 +1,50 @@
+{
+  "name": "hf-model-navigator-frontend",
+  "version": "1.0.0",
+  "description": "React frontend with Visx for HF Model Ecosystem Navigator",
+  "private": true,
+  "dependencies": {
+    "@visx/axis": "^3.0.0",
+    "@visx/brush": "^3.0.0",
+    "@visx/event": "^3.0.0",
+    "@visx/gradient": "^3.0.0",
+    "@visx/group": "^3.0.0",
+    "@visx/legend": "^3.0.0",
+    "@visx/point": "^3.0.0",
+    "@visx/scale": "^3.0.0",
+    "@visx/shape": "^3.0.0",
+    "@visx/tooltip": "^3.0.0",
+    "@visx/visx": "^3.0.0",
+    "react": "^18.2.0",
+    "react-dom": "^18.2.0",
+    "react-scripts": "5.0.1",
+    "typescript": "^5.0.0",
+    "@types/react": "^18.2.0",
+    "@types/react-dom": "^18.2.0",
+    "axios": "^1.6.0"
+  },
+  "scripts": {
+    "start": "react-scripts start",
+    "build": "react-scripts build",
+    "test": "react-scripts test",
+    "eject": "react-scripts eject"
+  },
+  "eslintConfig": {
+    "extends": [
+      "react-app"
+    ]
+  },
+  "browserslist": {
+    "production": [
+      ">0.2%",
+      "not dead",
+      "not op_mini all"
+    ],
+    "development": [
+      "last 1 chrome version",
+      "last 1 firefox version",
+      "last 1 safari version"
+    ]
+  }
+}
+

frontend/public/_redirects

@@ -0,0 +1,2 @@
+/*    /index.html   200
+

frontend/public/index.html

@@ -0,0 +1,18 @@
+<!DOCTYPE html>
+<html lang="en">
+  <head>
+    <meta charset="utf-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <meta name="theme-color" content="#000000" />
+    <meta
+      name="description"
+      content="Anatomy of a Machine Learning Ecosystem: 2 Million Models on Hugging Face. Analysis of 1.86 million models on Hugging Face, revealing fine-tuning lineages and model family structures using evolutionary biology methods."
+    />
+    <title>Anatomy of a Machine Learning Ecosystem: 2 Million Models on Hugging Face</title>
+  </head>
+  <body>
+    <noscript>You need to enable JavaScript to run this app.</noscript>
+    <div id="root"></div>
+  </body>
+</html>
+

frontend/src/App.css

@@ -0,0 +1,110 @@
+.App {
+  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen',
+    'Ubuntu', 'Cantarell', 'Fira Sans', 'Droid Sans', 'Helvetica Neue',
+    sans-serif;
+  -webkit-font-smoothing: antialiased;
+  -moz-osx-font-smoothing: grayscale;
+}
+
+.App-header {
+  background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
+  color: white;
+  padding: 2rem;
+  text-align: center;
+}
+
+.App-header h1 {
+  margin: 0 0 1rem 0;
+  font-size: 2rem;
+  font-weight: 600;
+}
+
+.App-header p {
+  margin: 0;
+  opacity: 0.9;
+}
+
+.App-header a {
+  color: white;
+  text-decoration: underline;
+  opacity: 0.9;
+  transition: opacity 0.2s;
+}
+
+.App-header a:hover {
+  opacity: 1;
+  text-decoration: none;
+}
+
+.stats {
+  display: flex;
+  gap: 2rem;
+  justify-content: center;
+  margin-top: 1rem;
+  font-size: 0.9rem;
+}
+
+.main-content {
+  display: flex;
+  height: calc(100vh - 200px);
+}
+
+.sidebar {
+  width: 300px;
+  padding: 2rem;
+  background: #f5f5f5;
+  overflow-y: auto;
+  border-right: 1px solid #e0e0e0;
+}
+
+.sidebar h2 {
+  margin-top: 0;
+  font-size: 1.5rem;
+}
+
+.sidebar label {
+  display: block;
+  margin-bottom: 1.5rem;
+  font-weight: 500;
+}
+
+.sidebar input[type="text"],
+.sidebar select {
+  width: 100%;
+  padding: 0.5rem;
+  margin-top: 0.5rem;
+  border: 1px solid #ccc;
+  border-radius: 4px;
+  font-size: 0.9rem;
+}
+
+.sidebar input[type="range"] {
+  width: 100%;
+  margin-top: 0.5rem;
+}
+
+.visualization {
+  flex: 1;
+  padding: 2rem;
+  display: flex;
+  align-items: center;
+  justify-content: center;
+  background: white;
+}
+
+.loading,
+.error,
+.empty {
+  text-align: center;
+  padding: 2rem;
+  font-size: 1.2rem;
+}
+
+.error {
+  color: #d32f2f;
+}
+
+.empty {
+  color: #666;
+}
+

frontend/src/App.tsx

@@ -0,0 +1,197 @@
+/**
+ * Main React app component using Visx for visualization.
+ */
+import React, { useState, useEffect, useCallback } from 'react';
+import ScatterPlot from './components/ScatterPlot';
+import ModelModal from './components/ModelModal';
+import { ModelPoint, Stats } from './types';
+import './App.css';
+
+const API_BASE = process.env.REACT_APP_API_URL || 'http://localhost:8000';
+
+function App() {
+  const [data, setData] = useState<ModelPoint[]>([]);
+  const [stats, setStats] = useState<Stats | null>(null);
+  const [loading, setLoading] = useState(true);
+  const [error, setError] = useState<string | null>(null);
+  const [selectedModel, setSelectedModel] = useState<ModelPoint | null>(null);
+  const [isModalOpen, setIsModalOpen] = useState(false);
+
+  // Filters
+  const [minDownloads, setMinDownloads] = useState(0);
+  const [minLikes, setMinLikes] = useState(0);
+  const [searchQuery, setSearchQuery] = useState('');
+  const [colorBy, setColorBy] = useState('library_name');
+  const [sizeBy, setSizeBy] = useState('downloads');
+
+  // Dimensions
+  const [width, setWidth] = useState(window.innerWidth * 0.7);
+  const [height, setHeight] = useState(window.innerHeight * 0.7);
+
+  useEffect(() => {
+    const handleResize = () => {
+      setWidth(window.innerWidth * 0.7);
+      setHeight(window.innerHeight * 0.7);
+    };
+    window.addEventListener('resize', handleResize);
+    return () => window.removeEventListener('resize', handleResize);
+  }, []);
+
+  const fetchData = useCallback(async () => {
+    setLoading(true);
+    setError(null);
+    try {
+      const params = new URLSearchParams({
+        min_downloads: minDownloads.toString(),
+        min_likes: minLikes.toString(),
+        color_by: colorBy,
+        size_by: sizeBy,
+        max_points: '5000',
+      });
+      if (searchQuery) {
+        params.append('search_query', searchQuery);
+      }
+
+      const response = await fetch(`${API_BASE}/api/models?${params}`);
+      if (!response.ok) throw new Error('Failed to fetch models');
+      const models = await response.json();
+      setData(models);
+    } catch (err) {
+      setError(err instanceof Error ? err.message : 'Unknown error');
+    } finally {
+      setLoading(false);
+    }
+  }, [minDownloads, minLikes, searchQuery, colorBy, sizeBy]);
+
+  useEffect(() => {
+    fetchData();
+  }, [fetchData]);
+
+  useEffect(() => {
+    // Fetch stats once
+    fetch(`${API_BASE}/api/stats`)
+      .then(res => res.json())
+      .then(setStats)
+      .catch(console.error);
+  }, []);
+
+  return (
+    <div className="App">
+      <header className="App-header">
+        <h1>Anatomy of a Machine Learning Ecosystem: 2 Million Models on Hugging Face</h1>
+        <p style={{ maxWidth: '900px', margin: '0 auto', lineHeight: '1.6' }}>
+          Many have observed that the development and deployment of generative machine learning (ML) and artificial intelligence (AI) models follow a distinctive pattern in which pre-trained models are adapted and fine-tuned for specific downstream tasks. However, there is limited empirical work that examines the structure of these interactions. This paper analyzes 1.86 million models on Hugging Face, a leading peer production platform for model development. Our study of model family trees reveals sprawling fine-tuning lineages that vary widely in size and structure. Using an evolutionary biology lens, we measure genetic similarity and mutation of traits over model families.
+          {' '}
+          <a 
+            href="https://arxiv.org/abs/2508.06811" 
+            target="_blank" 
+            rel="noopener noreferrer"
+            style={{ color: 'white', textDecoration: 'underline', fontWeight: '500' }}
+          >
+            Read the full paper →
+          </a>
+        </p>
+        <p style={{ marginTop: '0.5rem', fontSize: '0.9rem', opacity: 0.9 }}>
+          <strong>Authors:</strong> Benjamin Laufer, Hamidah Oderinwale, Jon Kleinberg
+        </p>
+        {stats && (
+          <div className="stats">
+            <span>Total Models: {stats.total_models.toLocaleString()}</span>
+            <span>Libraries: {stats.unique_libraries}</span>
+            <span>Pipelines: {stats.unique_pipelines}</span>
+          </div>
+        )}
+      </header>
+
+      <div className="main-content">
+        <aside className="sidebar">
+          <h2>Filters</h2>
+          
+          <label>
+            Search:
+            <input
+              type="text"
+              value={searchQuery}
+              onChange={(e) => setSearchQuery(e.target.value)}
+              placeholder="Model ID or tags..."
+            />
+          </label>
+
+          <label>
+            Min Downloads: {minDownloads.toLocaleString()}
+            <input
+              type="range"
+              min="0"
+              max="1000000"
+              step="1000"
+              value={minDownloads}
+              onChange={(e) => setMinDownloads(Number(e.target.value))}
+            />
+          </label>
+
+          <label>
+            Min Likes: {minLikes.toLocaleString()}
+            <input
+              type="range"
+              min="0"
+              max="10000"
+              step="10"
+              value={minLikes}
+              onChange={(e) => setMinLikes(Number(e.target.value))}
+            />
+          </label>
+
+          <label>
+            Color By:
+            <select value={colorBy} onChange={(e) => setColorBy(e.target.value)}>
+              <option value="library_name">Library</option>
+              <option value="pipeline_tag">Pipeline</option>
+              <option value="downloads">Downloads</option>
+              <option value="likes">Likes</option>
+            </select>
+          </label>
+
+          <label>
+            Size By:
+            <select value={sizeBy} onChange={(e) => setSizeBy(e.target.value)}>
+              <option value="downloads">Downloads</option>
+              <option value="likes">Likes</option>
+              <option value="trendingScore">Trending Score</option>
+              <option value="none">None</option>
+            </select>
+          </label>
+        </aside>
+
+        <main className="visualization">
+          {loading && <div className="loading">Loading models...</div>}
+          {error && <div className="error">Error: {error}</div>}
+          {!loading && !error && data.length === 0 && (
+            <div className="empty">No models match the filters</div>
+          )}
+          {!loading && !error && data.length > 0 && (
+            <ScatterPlot
+              width={width}
+              height={height}
+              data={data}
+              colorBy={colorBy}
+              sizeBy={sizeBy}
+              onPointClick={(model) => {
+                setSelectedModel(model);
+                setIsModalOpen(true);
+              }}
+            />
+          )}
+        </main>
+
+        <ModelModal
+          model={selectedModel}
+          isOpen={isModalOpen}
+          onClose={() => setIsModalOpen(false)}
+        />
+      </div>
+    </div>
+  );
+}
+
+export default App;
+

frontend/src/components/ModelModal.css

@@ -0,0 +1,161 @@
+.modal-overlay {
+  position: fixed;
+  top: 0;
+  left: 0;
+  right: 0;
+  bottom: 0;
+  background: rgba(0, 0, 0, 0.7);
+  display: flex;
+  align-items: center;
+  justify-content: center;
+  z-index: 1000;
+  padding: 2rem;
+  animation: fadeIn 0.2s ease-in;
+}
+
+@keyframes fadeIn {
+  from {
+    opacity: 0;
+  }
+  to {
+    opacity: 1;
+  }
+}
+
+.modal-content {
+  background: white;
+  border-radius: 12px;
+  max-width: 600px;
+  width: 100%;
+  max-height: 90vh;
+  overflow-y: auto;
+  padding: 2rem;
+  position: relative;
+  box-shadow: 0 20px 60px rgba(0, 0, 0, 0.3);
+  animation: slideUp 0.3s ease-out;
+}
+
+@keyframes slideUp {
+  from {
+    transform: translateY(20px);
+    opacity: 0;
+  }
+  to {
+    transform: translateY(0);
+    opacity: 1;
+  }
+}
+
+.modal-close {
+  position: absolute;
+  top: 1rem;
+  right: 1rem;
+  background: none;
+  border: none;
+  font-size: 2rem;
+  line-height: 1;
+  cursor: pointer;
+  color: #666;
+  padding: 0;
+  width: 32px;
+  height: 32px;
+  display: flex;
+  align-items: center;
+  justify-content: center;
+  border-radius: 50%;
+  transition: all 0.2s;
+}
+
+.modal-close:hover {
+  background: #f0f0f0;
+  color: #000;
+}
+
+.modal-content h2 {
+  margin: 0 0 1.5rem 0;
+  font-size: 1.5rem;
+  color: #333;
+  word-break: break-word;
+}
+
+.modal-section {
+  margin-bottom: 1.5rem;
+}
+
+.modal-section:last-child {
+  margin-bottom: 0;
+}
+
+.modal-section h3 {
+  margin: 0 0 0.75rem 0;
+  font-size: 1rem;
+  font-weight: 600;
+  color: #555;
+  text-transform: uppercase;
+  letter-spacing: 0.5px;
+}
+
+.modal-info-grid {
+  display: grid;
+  grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
+  gap: 1rem;
+}
+
+.modal-info-item {
+  display: flex;
+  flex-direction: column;
+  gap: 0.25rem;
+}
+
+.modal-info-item strong {
+  font-size: 0.875rem;
+  color: #666;
+  font-weight: 500;
+}
+
+.modal-info-item span {
+  font-size: 1rem;
+  color: #333;
+  font-weight: 500;
+}
+
+.modal-tags {
+  margin: 0;
+  padding: 0.75rem;
+  background: #f5f5f5;
+  border-radius: 6px;
+  color: #333;
+  font-size: 0.9rem;
+  line-height: 1.5;
+}
+
+.modal-link {
+  display: inline-flex;
+  align-items: center;
+  gap: 0.5rem;
+  padding: 0.75rem 1.5rem;
+  background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
+  color: white;
+  text-decoration: none;
+  border-radius: 6px;
+  font-weight: 500;
+  transition: all 0.2s;
+  margin-top: 0.5rem;
+}
+
+.modal-link:hover {
+  transform: translateY(-2px);
+  box-shadow: 0 4px 12px rgba(102, 126, 234, 0.4);
+}
+
+@media (max-width: 768px) {
+  .modal-content {
+    padding: 1.5rem;
+    max-width: 100%;
+  }
+
+  .modal-info-grid {
+    grid-template-columns: 1fr;
+  }
+}
+

frontend/src/components/ModelModal.tsx

@@ -0,0 +1,90 @@
+/**
+ * Modal component for displaying detailed model information.
+ */
+import React from 'react';
+import { ModelPoint } from '../types';
+import './ModelModal.css';
+
+interface ModelModalProps {
+  model: ModelPoint | null;
+  isOpen: boolean;
+  onClose: () => void;
+}
+
+export default function ModelModal({ model, isOpen, onClose }: ModelModalProps) {
+  if (!isOpen || !model) return null;
+
+  const hfUrl = `https://huggingface.co/${model.model_id}`;
+
+  return (
+    <div className="modal-overlay" onClick={onClose}>
+      <div className="modal-content" onClick={(e) => e.stopPropagation()}>
+        <button className="modal-close" onClick={onClose}>×</button>
+        
+        <h2>{model.model_id}</h2>
+        
+        <div className="modal-section">
+          <h3>Model Information</h3>
+          <div className="modal-info-grid">
+            <div className="modal-info-item">
+              <strong>Library:</strong>
+              <span>{model.library_name || 'N/A'}</span>
+            </div>
+            <div className="modal-info-item">
+              <strong>Pipeline Tag:</strong>
+              <span>{model.pipeline_tag || 'N/A'}</span>
+            </div>
+            <div className="modal-info-item">
+              <strong>Downloads:</strong>
+              <span>{model.downloads.toLocaleString()}</span>
+            </div>
+            <div className="modal-info-item">
+              <strong>Likes:</strong>
+              <span>{model.likes.toLocaleString()}</span>
+            </div>
+            {model.trending_score !== null && (
+              <div className="modal-info-item">
+                <strong>Trending Score:</strong>
+                <span>{model.trending_score.toFixed(2)}</span>
+              </div>
+            )}
+          </div>
+        </div>
+
+        {model.tags && (
+          <div className="modal-section">
+            <h3>Tags</h3>
+            <p className="modal-tags">{model.tags}</p>
+          </div>
+        )}
+
+        <div className="modal-section">
+          <h3>Links</h3>
+          <a 
+            href={hfUrl} 
+            target="_blank" 
+            rel="noopener noreferrer"
+            className="modal-link"
+          >
+            View on Hugging Face →
+          </a>
+        </div>
+
+        <div className="modal-section">
+          <h3>Position in Latent Space</h3>
+          <div className="modal-info-grid">
+            <div className="modal-info-item">
+              <strong>Dimension 1:</strong>
+              <span>{model.x.toFixed(4)}</span>
+            </div>
+            <div className="modal-info-item">
+              <strong>Dimension 2:</strong>
+              <span>{model.y.toFixed(4)}</span>
+            </div>
+          </div>
+        </div>
+      </div>
+    </div>
+  );
+}
+

frontend/src/components/ScatterPlot.tsx

@@ -0,0 +1,235 @@
+/**
+ * Visx-based scatter plot component for model visualization.
+ * Based on visx gallery examples: https://visx.airbnb.tech/gallery
+ */
+import React, { useMemo, useCallback } from 'react';
+import { Group } from '@visx/group';
+import { scaleLinear, scaleOrdinal } from '@visx/scale';
+import { AxisBottom, AxisLeft } from '@visx/axis';
+import { GridRows, GridColumns } from '@visx/grid';
+import { Tooltip, useTooltip } from '@visx/tooltip';
+import { LegendOrdinal } from '@visx/legend';
+// Using circle elements directly instead of Point component
+// Color schemes - using a predefined palette
+const colorPalette = [
+  '#8dd3c7', '#ffffb3', '#bebada', '#fb8072', '#80b1d3',
+  '#fdb462', '#b3de69', '#fccde5', '#d9d9d9', '#bc80bd',
+  '#ccebc5', '#ffed6f'
+];
+import { ModelPoint } from '../types';
+
+interface ScatterPlotProps {
+  width: number;
+  height: number;
+  data: ModelPoint[];
+  colorBy: string;
+  sizeBy: string;
+  margin?: { top: number; right: number; bottom: number; left: number };
+  onPointClick?: (model: ModelPoint) => void;
+}
+
+const defaultMargin = { top: 40, right: 40, bottom: 60, left: 60 };
+
+export default function ScatterPlot({
+  width,
+  height,
+  data,
+  colorBy,
+  sizeBy,
+  margin = defaultMargin,
+  onPointClick,
+}: ScatterPlotProps) {
+  const {
+    tooltipData,
+    tooltipLeft,
+    tooltipTop,
+    tooltipOpen,
+    showTooltip,
+    hideTooltip,
+  } = useTooltip<ModelPoint>();
+
+  // Bounds
+  const xMax = width - margin.left - margin.right;
+  const yMax = height - margin.top - margin.bottom;
+
+  // Scales
+  const xScale = useMemo(
+    () =>
+      scaleLinear<number>({
+        domain: [Math.min(...data.map(d => d.x)), Math.max(...data.map(d => d.x))],
+        range: [0, xMax],
+        nice: true,
+      }),
+    [data, xMax]
+  );
+
+  const yScale = useMemo(
+    () =>
+      scaleLinear<number>({
+        domain: [Math.min(...data.map(d => d.y)), Math.max(...data.map(d => d.y))],
+        range: [yMax, 0],
+        nice: true,
+      }),
+    [data, yMax]
+  );
+
+  // Color scale
+  const getColorValue = (d: ModelPoint) => {
+    if (colorBy === 'library_name') return d.library_name || 'Unknown';
+    if (colorBy === 'pipeline_tag') return d.pipeline_tag || 'Unknown';
+    if (colorBy === 'downloads') return d.downloads;
+    if (colorBy === 'likes') return d.likes;
+    return 'All';
+  };
+
+  const colorValues = useMemo(() => data.map(getColorValue), [data, colorBy]);
+  const isCategorical = colorBy === 'library_name' || colorBy === 'pipeline_tag';
+
+  const colorScale = useMemo(() => {
+    if (isCategorical) {
+      const uniqueValues = Array.from(new Set(colorValues));
+      return scaleOrdinal<string, string>({
+        domain: uniqueValues,
+        range: colorPalette,
+      });
+    } else {
+      // For continuous, we'll use a linear scale with a color interpolator
+      const min = Math.min(...(colorValues as number[]));
+      const max = Math.max(...(colorValues as number[]));
+      return scaleLinear<number, string>({
+        domain: [min, max],
+        range: ['#440154', '#fde725'], // Viridis-like colors
+      });
+    }
+  }, [colorValues, isCategorical]);
+
+  // Size scale
+  const getSizeValue = (d: ModelPoint) => {
+    if (sizeBy === 'downloads') return d.downloads;
+    if (sizeBy === 'likes') return d.likes;
+    if (sizeBy === 'trendingScore' && d.trending_score) return d.trending_score;
+    return 10;
+  };
+
+  const sizeValues = useMemo(() => data.map(getSizeValue), [data, sizeBy]);
+  const minSize = Math.min(...sizeValues);
+  const maxSize = Math.max(...sizeValues);
+
+  const sizeScale = useMemo(
+    () =>
+      scaleLinear<number>({
+        domain: [minSize, maxSize],
+        range: [5, 20],
+      }),
+    [minSize, maxSize]
+  );
+
+  // Handle point hover
+  const handleMouseOver = useCallback(
+    (event: React.MouseEvent, datum: ModelPoint) => {
+      const coords = { x: event.clientX, y: event.clientY };
+      showTooltip({
+        tooltipLeft: coords.x,
+        tooltipTop: coords.y,
+        tooltipData: datum,
+      });
+    },
+    [showTooltip]
+  );
+
+  return (
+    <div style={{ position: 'relative' }}>
+      <svg width={width} height={height}>
+        <Group left={margin.left} top={margin.top}>
+          {/* Grid */}
+          <GridRows scale={yScale} width={xMax} strokeDasharray="3,3" stroke="#e0e0e0" />
+          <GridColumns scale={xScale} height={yMax} strokeDasharray="3,3" stroke="#e0e0e0" />
+
+          {/* Points */}
+          {data.map((d, i) => {
+            const x = xScale(d.x);
+            const y = yScale(d.y);
+            const color = isCategorical
+              ? colorScale(getColorValue(d) as string)
+              : colorScale(getColorValue(d) as number);
+            const size = sizeScale(getSizeValue(d));
+
+            return (
+              <circle
+                key={`point-${i}`}
+                cx={x}
+                cy={y}
+                r={size / 2}
+                fill={color}
+                opacity={0.7}
+                stroke="white"
+                strokeWidth={0.5}
+                onMouseOver={(e) => handleMouseOver(e, d)}
+                onMouseOut={hideTooltip}
+                onClick={() => onPointClick && onPointClick(d)}
+                style={{ cursor: 'pointer' }}
+              />
+            );
+          })}
+
+          {/* Axes */}
+          <AxisBottom
+            top={yMax}
+            scale={xScale}
+            numTicks={5}
+            label="Dimension 1"
+            stroke="#333"
+            tickStroke="#333"
+          />
+          <AxisLeft
+            scale={yScale}
+            numTicks={5}
+            label="Dimension 2"
+            stroke="#333"
+            tickStroke="#333"
+          />
+        </Group>
+      </svg>
+
+      {/* Tooltip */}
+      {tooltipOpen && tooltipData && (
+        <Tooltip
+          top={tooltipTop}
+          left={tooltipLeft}
+          style={{
+            backgroundColor: 'rgba(0, 0, 0, 0.9)',
+            color: 'white',
+            padding: '8px',
+            borderRadius: '4px',
+            fontSize: '12px',
+          }}
+        >
+          <div>
+            <strong>{tooltipData.model_id}</strong>
+            <br />
+            Library: {tooltipData.library_name || 'N/A'}
+            <br />
+            Pipeline: {tooltipData.pipeline_tag || 'N/A'}
+            <br />
+            Downloads: {tooltipData.downloads.toLocaleString()}
+            <br />
+            Likes: {tooltipData.likes.toLocaleString()}
+          </div>
+        </Tooltip>
+      )}
+
+      {/* Legend */}
+      {isCategorical && (
+        <div style={{ position: 'absolute', top: 10, right: 10 }}>
+          <LegendOrdinal
+            scale={colorScale as any}
+            labelFormat={(label) => label}
+            direction="column"
+            style={{ fontSize: '12px' }}
+          />
+        </div>
+      )}
+    </div>
+  );
+}
+

frontend/src/index.css

@@ -0,0 +1,18 @@
+body {
+  margin: 0;
+  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen',
+    'Ubuntu', 'Cantarell', 'Fira Sans', 'Droid Sans', 'Helvetica Neue',
+    sans-serif;
+  -webkit-font-smoothing: antialiased;
+  -moz-osx-font-smoothing: grayscale;
+}
+
+code {
+  font-family: source-code-pro, Menlo, Monaco, Consolas, 'Courier New',
+    monospace;
+}
+
+* {
+  box-sizing: border-box;
+}
+

frontend/src/index.tsx

@@ -0,0 +1,14 @@
+import React from 'react';
+import ReactDOM from 'react-dom/client';
+import './index.css';
+import App from './App';
+
+const root = ReactDOM.createRoot(
+  document.getElementById('root') as HTMLElement
+);
+root.render(
+  <React.StrictMode>
+    <App />
+  </React.StrictMode>
+);
+

frontend/src/types.ts

@@ -0,0 +1,20 @@
+export interface ModelPoint {
+  model_id: string;
+  x: number;
+  y: number;
+  library_name: string | null;
+  pipeline_tag: string | null;
+  downloads: number;
+  likes: number;
+  trending_score: number | null;
+  tags: string | null;
+}
+
+export interface Stats {
+  total_models: number;
+  unique_libraries: number;
+  unique_pipelines: number;
+  avg_downloads: number;
+  avg_likes: number;
+}
+

frontend/tsconfig.json

@@ -0,0 +1,27 @@
+{
+  "compilerOptions": {
+    "target": "es5",
+    "lib": [
+      "dom",
+      "dom.iterable",
+      "esnext"
+    ],
+    "allowJs": true,
+    "skipLibCheck": true,
+    "esModuleInterop": true,
+    "allowSyntheticDefaultImports": true,
+    "strict": true,
+    "forceConsistentCasingInFileNames": true,
+    "noFallthroughCasesInSwitch": true,
+    "module": "esnext",
+    "moduleResolution": "node",
+    "resolveJsonModule": true,
+    "isolatedModules": true,
+    "noEmit": true,
+    "jsx": "react-jsx"
+  },
+  "include": [
+    "src"
+  ]
+}
+

netlify-functions/api.py

@@ -0,0 +1,180 @@
+"""
+Netlify Serverless Function for model data API.
+This is a simplified version that works with Netlify Functions.
+"""
+import json
+import os
+import sys
+
+# Add parent directory to path to import modules
+sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
+
+from data_loader import ModelDataLoader
+from embeddings import ModelEmbedder
+from dimensionality_reduction import DimensionReducer
+import pandas as pd
+import numpy as np
+
+# Global state (persists across invocations in serverless)
+data_loader = ModelDataLoader()
+embedder = None
+reducer = None
+df = None
+embeddings = None
+reduced_embeddings = None
+
+
+def handler(event, context):
+    """
+    Netlify serverless function handler.
+    """
+    global embedder, reducer, df, embeddings, reduced_embeddings
+    
+    # Parse query parameters
+    query_params = event.get('queryStringParameters') or {}
+    path = event.get('path', '')
+    
+    # CORS headers
+    headers = {
+        'Access-Control-Allow-Origin': '*',
+        'Access-Control-Allow-Headers': 'Content-Type',
+        'Access-Control-Allow-Methods': 'GET, OPTIONS',
+        'Content-Type': 'application/json',
+    }
+    
+    # Handle OPTIONS (CORS preflight)
+    if event.get('httpMethod') == 'OPTIONS':
+        return {
+            'statusCode': 200,
+            'headers': headers,
+            'body': ''
+        }
+    
+    # Initialize data on first request
+    if df is None:
+        try:
+            print("Loading data...")
+            df = data_loader.load_data(sample_size=10000)
+            df = data_loader.preprocess_for_embedding(df)
+            print(f"Loaded {len(df)} models")
+        except Exception as e:
+            return {
+                'statusCode': 500,
+                'headers': headers,
+                'body': json.dumps({'error': f'Failed to load data: {str(e)}'})
+            }
+    
+    # Route requests
+    if path.endswith('/api/models') or '/api/models' in path:
+        return get_models(query_params, headers)
+    elif path.endswith('/api/stats') or '/api/stats' in path:
+        return get_stats(headers)
+    else:
+        return {
+            'statusCode': 404,
+            'headers': headers,
+            'body': json.dumps({'error': 'Not found'})
+        }
+
+
+def get_models(query_params, headers):
+    """Get filtered models."""
+    global df, embedder, reducer, embeddings, reduced_embeddings
+    
+    try:
+        min_downloads = int(query_params.get('min_downloads', 0))
+        min_likes = int(query_params.get('min_likes', 0))
+        search_query = query_params.get('search_query')
+        max_points = int(query_params.get('max_points', 5000))
+        
+        # Filter data
+        filtered_df = data_loader.filter_data(
+            df=df,
+            min_downloads=min_downloads,
+            min_likes=min_likes,
+            search_query=search_query
+        )
+        
+        if len(filtered_df) == 0:
+            return {
+                'statusCode': 200,
+                'headers': headers,
+                'body': json.dumps([])
+            }
+        
+        # Limit points
+        if len(filtered_df) > max_points:
+            filtered_df = filtered_df.sample(n=max_points, random_state=42)
+        
+        # Generate embeddings if needed
+        if embedder is None:
+            embedder = ModelEmbedder()
+        
+        if embeddings is None:
+            texts = df['combined_text'].tolist()
+            embeddings = embedder.generate_embeddings(texts)
+        
+        # Reduce dimensions if needed
+        if reducer is None:
+            reducer = DimensionReducer(method="umap", n_components=2)
+        
+        if reduced_embeddings is None:
+            reduced_embeddings = reducer.fit_transform(embeddings)
+        
+        # Get coordinates
+        filtered_indices = filtered_df.index.tolist()
+        filtered_reduced = reduced_embeddings[filtered_indices]
+        
+        # Prepare response
+        models = []
+        for idx, (i, row) in enumerate(filtered_df.iterrows()):
+            models.append({
+                'model_id': row.get('model_id', 'Unknown'),
+                'x': float(filtered_reduced[idx, 0]),
+                'y': float(filtered_reduced[idx, 1]),
+                'library_name': row.get('library_name'),
+                'pipeline_tag': row.get('pipeline_tag'),
+                'downloads': int(row.get('downloads', 0)),
+                'likes': int(row.get('likes', 0)),
+                'trending_score': float(row.get('trendingScore', 0)) if pd.notna(row.get('trendingScore')) else None,
+                'tags': row.get('tags') if pd.notna(row.get('tags')) else None
+            })
+        
+        return {
+            'statusCode': 200,
+            'headers': headers,
+            'body': json.dumps(models)
+        }
+    except Exception as e:
+        return {
+            'statusCode': 500,
+            'headers': headers,
+            'body': json.dumps({'error': str(e)})
+        }
+
+
+def get_stats(headers):
+    """Get dataset statistics."""
+    global df
+    
+    if df is None:
+        return {
+            'statusCode': 503,
+            'headers': headers,
+            'body': json.dumps({'error': 'Data not loaded'})
+        }
+    
+    stats = {
+        'total_models': len(df),
+        'unique_libraries': df['library_name'].nunique() if 'library_name' in df.columns else 0,
+        'unique_pipelines': df['pipeline_tag'].nunique() if 'pipeline_tag' in df.columns else 0,
+        'avg_downloads': float(df['downloads'].mean()) if 'downloads' in df.columns else 0,
+        'avg_likes': float(df['likes'].mean()) if 'likes' in df.columns else 0
+    }
+    
+    return {
+        'statusCode': 200,
+        'headers': headers,
+        'body': json.dumps(stats)
+    }
+

netlify-functions/models.py

@@ -0,0 +1,23 @@
+"""
+Netlify serverless function wrapper for models API.
+This file is the entry point for Netlify Functions.
+"""
+import sys
+import os
+
+# Add parent directories to path
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..'))
+
+from api import handler
+
+# Netlify Functions expects a handler function
+def lambda_handler(event, context):
+    """
+    AWS Lambda/Netlify Functions handler.
+    Converts Netlify event format to our handler format.
+    """
+    # Convert Netlify event to our format
+    # Netlify passes path in event['path']
+    # Query params are in event['queryStringParameters']
+    return handler(event, context)
+

netlify-functions/requirements.txt

@@ -0,0 +1,9 @@
+pandas>=2.0.0
+numpy>=1.24.0
+sentence-transformers>=2.2.0
+umap-learn>=0.5.4
+scikit-learn>=1.3.0
+datasets>=2.14.0
+huggingface-hub>=0.17.0
+tqdm>=4.66.0
+

netlify.toml

@@ -0,0 +1,27 @@
+[build]
+  base = "frontend"
+  publish = "frontend/build"
+  command = "npm install && npm run build"
+
+[build.environment]
+  NODE_VERSION = "18"
+
+# Redirect all routes to index.html for React Router
+[[redirects]]
+  from = "/*"
+  to = "/index.html"
+  status = 200
+
+# Netlify Functions (if using serverless backend)
+[functions]
+  directory = "netlify-functions"
+  node_bundler = "esbuild"
+
+# Headers for API routes
+[[headers]]
+  for = "/.netlify/functions/*"
+  [headers.values]
+    Access-Control-Allow-Origin = "*"
+    Access-Control-Allow-Headers = "Content-Type"
+    Access-Control-Allow-Methods = "GET, POST, OPTIONS"
+

requirements.txt

@@ -0,0 +1,12 @@
+gradio>=4.0.0
+plotly>=5.18.0
+pandas>=2.0.0
+numpy>=1.24.0
+sentence-transformers>=2.2.0
+umap-learn>=0.5.4
+scikit-learn>=1.3.0
+datasets>=2.14.0
+huggingface-hub>=0.17.0
+tqdm>=4.66.0
+python-dotenv>=1.0.0
+

test_local.py

@@ -0,0 +1,13 @@
+"""
+Quick test script to verify the application works locally.
+Run this before deploying to Hugging Face Spaces.
+"""
+import sys
+from app import create_interface
+
+if __name__ == "__main__":
+    print("Creating interface...")
+    demo = create_interface()
+    print("Launching demo...")
+    demo.launch(share=False, server_name="127.0.0.1", server_port=7860)
+