Upload 107 files

markdown_files/1._Development_Tools.md

@@ -0,0 +1,22 @@
+---
+title: "1. Development Tools"
+original_url: "https://tds.s-anand.net/#/development-tools?id=development-tools"
+downloaded_at: "2025-06-08T23:21:33.929318"
+---
+
+[Development Tools](#/development-tools?id=development-tools)
+=============================================================
+
+**NOTE**: The tools in this module are **PRE-REQUISITES** for the course. You would have used most of these before. If most of this is new to you, please take this course later.
+
+Some tools are fundamental to data science because they are industry standards and widely used by data science professionals. Mastering these tools will align you with current best practices and making you more adaptable in a fast-evolving industry.
+
+The tools we cover here are not just popular, they’re the core technology behind most of today’s data science and software development.
+
+[Previous
+
+Tools in Data Science](#/README)
+
+[Next
+
+Editor: VS Code](#/vscode)

markdown_files/2._Deployment_Tools.md

@@ -0,0 +1,18 @@
+---
+title: "2. Deployment Tools"
+original_url: "https://tds.s-anand.net/#/deployment-tools?id=deployment-tools"
+downloaded_at: "2025-06-08T23:26:43.558808"
+---
+
+[Deployment Tools](#/deployment-tools?id=deployment-tools)
+==========================================================
+
+Any application you build is likely to be deployed somewhere. This section covers the most popular tools involved in deploying an application.
+
+[Previous
+
+Version Control: Git, GitHub](#/git)
+
+[Next
+
+Markdown](#/markdown)

markdown_files/3._Large_Language_Models.md

@@ -0,0 +1,68 @@
+---
+title: "3. Large Language Models"
+original_url: "https://tds.s-anand.net/#/large-language-models?id=large-language-models"
+downloaded_at: "2025-06-08T23:23:17.306109"
+---
+
+[Large Language Models](#/large-language-models?id=large-language-models)
+=========================================================================
+
+This module covers the practical usage of large language models (LLMs).
+
+**LLMs incur a cost.** For the May 2025 batch, use [aipipe.org](https://aipipe.org/) as a proxy.
+Emails with `@ds.study.iitm.ac.in` get a **$1 per calendar month** allowance. (Don’t exceed that.)
+
+Read the [AI Pipe documentation](https://github.com/sanand0/aipipe) to learn how to use it. But in short:
+
+1. Replace `OPENAI_BASE_URL`, i.e. `https://api.openai.com/v1` with `https://aipipe.org/openrouter/v1...` or `https://aipipe.org/openai/v1...`
+2. Replace `OPENAI_API_KEY` with the [`AIPIPE_TOKEN`](https://aipipe.org/login)
+3. Replace model names, e.g. `gpt-4.1-nano`, with `openai/gpt-4.1-nano`
+
+For example, let’s use [Gemini 2.0 Flash Lite](https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-0-flash-lite) via [OpenRouter](https://openrouter.ai/google/gemini-2.0-flash-lite-001) for chat completions and [Text Embedding 3 Small](https://platform.openai.com/docs/models/text-embedding-3-small) via [OpenAI](https://platform.openai.com/docs/) for embeddings:
+
+```
+curl https://aipipe.org/openrouter/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer $AIPIPE_TOKEN" \
+  -d '{
+    "model": "google/gemini-2.0-flash-lite-001",
+    "messages": [{ "role": "user", "content": "What is 2 + 2?"} }]
+  }'
+
+curl https://aipipe.org/openai/v1/embeddings \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer $AIPIPE_TOKEN" \
+  -d '{ "model": "text-embedding-3-small", "input": "What is 2 + 2?" }'Copy to clipboardErrorCopied
+```
+
+Or using [`llm`](https://llm.datasette.io/):
+
+```
+llm keys set openai --value $AIPIPE_TOKEN
+
+export OPENAI_BASE_URL=https://aipipe.org/openrouter/v1
+llm 'What is 2 + 2?' -m openrouter/google/gemini-2.0-flash-lite-001
+
+export OPENAI_BASE_URL=https://aipipe.org/openai/v1
+llm embed -c 'What is 2 + 2' -m 3-smallCopy to clipboardErrorCopied
+```
+
+**For a 50% discount** (but slower speed), use [Flex processing](https://platform.openai.com/docs/guides/flex-processing) by adding `service_tier: "flex"` to your JSON request.
+
+[AI Proxy - Jan 2025](#/large-language-models?id=ai-proxy-jan-2025)
+-------------------------------------------------------------------
+
+For the Jan 2025 batch, we had created API keys for everyone with an `iitm.ac.in` email to use `gpt-4o-mini` and `text-embedding-3-small`. Your usage is limited to **$1 per calendar month** for this course. Don’t exceed that.
+
+**Use [AI Proxy](https://github.com/sanand0/aiproxy)** instead of OpenAI. Specifically:
+
+1. Replace your API to `https://api.openai.com/...` with `https://aiproxy.sanand.workers.dev/openai/...`
+2. Replace the `OPENAI_API_KEY` with the `AIPROXY_TOKEN` that someone will give you.
+
+[Previous
+
+Local LLMs: Ollama](#/ollama)
+
+[Next
+
+Prompt engineering](#/prompt-engineering)

markdown_files/4._Data_Sourcing.md

@@ -0,0 +1,39 @@
+---
+title: "4. Data Sourcing"
+original_url: "https://tds.s-anand.net/#/data-sourcing?id=data-sourcing"
+downloaded_at: "2025-06-08T23:24:22.670487"
+---
+
+[Data Sourcing](#/data-sourcing?id=data-sourcing)
+=================================================
+
+Before you do any kind of data science, you obviously have to get the data to be able to analyze it, visualize it, narrate it, and deploy it.
+And what we are going to cover in this module is how you get the data.
+
+There are three ways you can get the data.
+
+1. The first is you can **download** the data. Either somebody gives you the data and says download it from here, or you are asked to download it from the internet because it’s a public data source. But that’s the first way—you download the data.
+2. The second way is you can **query it** from somewhere. It may be on a database. It may be available through an API. It may be available through a library. But these are ways in which you can selectively query parts of the data and stitch it together.
+3. The third way is you have to **scrape it**. It’s not directly available in a convenient form that you can query or download. But it is, in fact, on a web page. It’s available on a PDF file. It’s available in a Word document. It’s available on an Excel file. It’s kind of structured, but you will have to figure out that structure and extract it from there.
+
+In this module, we will be looking at the tools that will help you either download from a data source or query from an API or from a database or from a library. And finally, how you can scrape from different sources.
+
+[![Data Sourcing - Introduction](https://i.ytimg.com/vi_webp/1LyblMkJzOo/sddefault.webp)](https://youtu.be/1LyblMkJzOo)
+
+Here are links used in the video:
+
+* [The Movies Dataset](https://www.kaggle.com/rounakbanik/the-movies-dataset)
+* [IMDb Datasets](https://imdb.com/interfaces/)
+* [Download the IMDb Datasets](https://datasets.imdbws.com/)
+* [Explore the Internet Movie Database](https://gramener.com/imdb/)
+* [What does the world search for?](https://gramener.com/search/)
+* [HowStat - Cricket statistics](https://howstat.com/cricket/home.asp)
+* [Cricket Strike Rates](https://gramener.com/cricket/)
+
+[Previous
+
+Project 1](#/project-tds-virtual-ta)
+
+[Next
+
+Scraping with Excel](#/scraping-with-excel)

markdown_files/5._Data_Preparation.md

@@ -0,0 +1,34 @@
+---
+title: "5. Data Preparation"
+original_url: "https://tds.s-anand.net/#/data-preparation?id=data-preparation"
+downloaded_at: "2025-06-08T23:22:16.649843"
+---
+
+[Data Preparation](#/data-preparation?id=data-preparation)
+==========================================================
+
+Data preparation is crucial because raw data is rarely perfect.
+
+It often contains errors, inconsistencies, or missing values. For example, marks data may have ‘NA’ or ‘absent’ for non-attendees, which you need to handle.
+
+This section teaches you how to clean up data, convert it to different formats, aggregate it if required, and get a feel for the data before you analyze.
+
+Here are links used in the video:
+
+* [Presentation used in the video](https://docs.google.com/presentation/d/1Gb0QnPUN1YOwM_O5EqDdXUdL-5Azp1Tf/view)
+* [Scraping assembly elections - Notebook](https://colab.research.google.com/drive/1SP8yVxzmofQO48-yXF3rujqWk2iM0KSl)
+* [Assembly election results (CSV)](https://github.com/datameet/india-election-data/blob/master/assembly-elections/assembly.csv)
+* [`pdftotext` software](https://www.xpdfreader.com/pdftotext-man.html)
+* [OpenRefine software](https://openrefine.org)
+* [The most persistent party](https://gramener.com/election/parliament#story.ddp)
+* [TN assembly election cartogram](https://gramener.com/election/cartogram?ST_NAME=Tamil%20Nadu)
+
+[![Data Preparation - Introduction](https://i.ytimg.com/vi_webp/dF3zchJJKqk/sddefault.webp)](https://youtu.be/dF3zchJJKqk)
+
+[Previous
+
+Scraping: Live Sessions](#/scraping-live-sessions)
+
+[Next
+
+Data Cleansing in Excel](#/data-cleansing-in-excel)

markdown_files/6._Data_Analysis.md

@@ -0,0 +1,31 @@
+---
+title: "6. Data Analysis"
+original_url: "https://tds.s-anand.net/#/data-analysis?id=data-analysis"
+downloaded_at: "2025-06-08T23:26:37.046522"
+---
+
+[Data analysis](#/data-analysis?id=data-analysis)
+=================================================
+
+[Data Analysis: Introduction Podcast](https://drive.google.com/file/d/1isjtxFa43CLIFlLpo8mwwQfBog9VlXYl/view) by [NotebookLM](https://notebooklm.google.com/)
+
+Once you’ve prepared the data, your next task is to analyze it to get insights that are not immediately obvious.
+
+In this module, you’ll learn:
+
+* **Statistical analysis**: Calculate correlations, regressions, forecasts, and outliers using **spreadsheets**
+* **Data summarization**: Aggregate and pivot data using **Python** and **databases**.
+* **Geo-data Collection & Processing**: Gather and process geospatial data using tools like Python (GeoPandas) and QGIS.
+* **Geo-visualization**: Create and visualize geospatial data on maps using Excel, QGIS, and Python libraries such as Folium.
+* **Network & Proximity Analysis**: Analyze geospatial relationships and perform network analysis to understand data distribution and clustering.
+* **Storytelling & Decision Making**: Develop narratives and make informed decisions based on geospatial data insights.
+
+[![Data Analysis - Introduction](https://i.ytimg.com/vi_webp/CRSljunxjnk/sddefault.webp)](https://youtu.be/CRSljunxjnk)
+
+[Previous
+
+Extracting Audio and Transcripts](#/extracting-audio-and-transcripts)
+
+[Next
+
+Correlation with Excel](#/correlation-with-excel)

markdown_files/7._Data_Visualization.md

@@ -0,0 +1,18 @@
+---
+title: "7. Data Visualization"
+original_url: "https://tds.s-anand.net/#/data-visualization?id=data-visualization"
+downloaded_at: "2025-06-08T23:27:12.693601"
+---
+
+[Data visualization](#/data-visualization?id=data-visualization)
+================================================================
+
+[![Data visualization](https://i.ytimg.com/vi_webp/XkxRDql00UU/sddefault.webp)](https://youtu.be/XkxRDql00UU)
+
+[Previous
+
+Network Analysis in Python](#/network-analysis-in-python)
+
+[Next
+
+Visualizing Forecasts with Excel](#/visualizing-forecasts-with-excel)

markdown_files/AI_Code_Editors__GitHub_Copilot.md

@@ -0,0 +1,31 @@
+---
+title: "AI Code Editors: GitHub Copilot"
+original_url: "https://tds.s-anand.net/#/github-copilot?id=ai-editor-github-copilot"
+downloaded_at: "2025-06-08T23:26:20.399680"
+---
+
+[AI Editor: GitHub Copilot](#/github-copilot?id=ai-editor-github-copilot)
+-------------------------------------------------------------------------
+
+AI Code Editors like [GitHub Copilot](https://github.com/features/copilot), [Cursor](https://www.cursor.com/), [Windsurf](http://windsurf.com/), [Roo Code](https://roocode.com/), [Cline](https://cline.bot/), [Continue.dev](https://www.continue.dev/), etc. use LLMs to help you write code faster.
+
+Most are built on top of [VS Code](#/vscode). These are now a standard tool in every developer’s toolkit.
+
+[GitHub Copilot](https://github.com/features/copilot) is [free](https://github.com/features/copilot/plans) (as of May 2025) for 2,000 completions and 50 chats.
+
+[![Getting started with GitHub Copilot | Tutorial (11 min)](https://i.ytimg.com/vi_webp/n0NlxUyA7FI/sddefault.webp)](https://youtu.be/n0NlxUyA7FI)
+
+You should learn about:
+
+* [Code Suggestions](https://docs.github.com/en/enterprise-cloud@latest/copilot/using-github-copilot/using-github-copilot-code-suggestions-in-your-editor), which is a basic feature.
+* [Using Chat](https://docs.github.com/en/copilot/github-copilot-chat/using-github-copilot-chat-in-your-ide), which lets you code in natural language.
+* [Changing the chat model](https://docs.github.com/en/copilot/using-github-copilot/ai-models/changing-the-ai-model-for-copilot-chat). The free version includes Claude 3.5 Sonnet, a good coding model.
+* [Prompts](https://docs.github.com/en/copilot/copilot-chat-cookbook) to understand how people use AI code editors.
+
+[Previous
+
+Editor: VS Code](#/vscode)
+
+[Next
+
+Python tools: uv](#/uv)

markdown_files/AI_Terminal_Tools__llm.md

@@ -0,0 +1,76 @@
+---
+title: "AI Terminal Tools: llm"
+original_url: "https://tds.s-anand.net/#/llm?id=llm-cli-llm"
+downloaded_at: "2025-06-08T23:25:09.715323"
+---
+
+[LLM CLI: llm](#/llm?id=llm-cli-llm)
+------------------------------------
+
+[`llm`](https://pypi.org/project/llm) is a command-line utility for interacting with large language models—simplifying prompts, managing models and plugins, logging every conversation, and extracting structured data for pipelines.
+
+[![Language models on the command-line w/ Simon Willison](https://i.ytimg.com/vi_webp/QUXQNi6jQ30/sddefault.webp)](https://youtu.be/QUXQNi6jQ30?t=100)
+
+### [Basic Usage](#/llm?id=basic-usage)
+
+[Install llm](https://github.com/simonw/llm#installation). Then set up your [`OPENAI_API_KEY`](https://platform.openai.com/api-keys) environment variable. See [Getting started](https://github.com/simonw/llm?tab=readme-ov-file#getting-started).
+
+**TDS Students**: See [Large Language Models](#/large-language-models) for instructions on how to get and use `OPENAI_API_KEY`.
+
+```
+# Run a simple prompt
+llm 'five great names for a pet pelican'
+
+# Continue a conversation
+llm -c 'now do walruses'
+
+# Start a memory-aware chat session
+llm chat
+
+# Specify a model
+llm -m gpt-4.1-nano 'Summarize tomorrow’s meeting agenda'
+
+# Extract JSON output
+llm 'List the top 5 Python viz libraries with descriptions' \
+  --schema-multi 'name,description'Copy to clipboardErrorCopied
+```
+
+Or use llm without installation using [`uvx`](#/uv):
+
+```
+# Run llm via uvx without any prior installation
+uvx llm 'Translate "Hello, world" into Japanese'
+
+# Specify a model
+uvx llm -m gpt-4.1-nano 'Draft a 200-word blog post on data ethics'
+
+# Use structured JSON output
+uvx llm 'List the top 5 programming languages in 2025 with their release years' \
+  --schema-multi 'rank,language,release_year'Copy to clipboardErrorCopied
+```
+
+### [Key Features](#/llm?id=key-features)
+
+* **Interactive prompts**: `llm '…'` — Fast shell access to any LLM.
+* **Conversational flow**: `-c '…'` — Continue context across prompts.
+* **Model switching**: `-m MODEL` — Use OpenAI, Anthropic, local models, and more.
+* **Structured output**: `llm json` — Produce JSON for automation.
+* **Logging & history**: `llm logs path` — Persist every prompt/response in SQLite.
+* **Web UI**: `datasette "$(llm logs path)"` — Browse your entire history with Datasette.
+* **Persistent chat**: `llm chat` — Keep the model in memory across multiple interactions.
+* **Plugin ecosystem**: `llm install PLUGIN` — Add support for new models, data sources, or workflows. ([Language models on the command-line - Simon Willison’s Weblog](https://simonwillison.net/2024/Jun/17/cli-language-models/?utm_source=chatgpt.com))
+
+### [Practical Uses](#/llm?id=practical-uses)
+
+* **Automated coding**. Generate code scaffolding, review helpers, or utilities on demand. For example, after running`llm install llm-cmd`, run `llm cmd 'Undo the last git commit'`. Inspired by [Simon’s post on using LLMs for rapid tool building](https://simonwillison.net/2025/Mar/11/using-llms-for-code/).
+* **Transcript processing**. Summarize YouTube or podcast transcripts using Gemini. See [Putting Gemini 2.5 Pro through its paces](https://www.macstories.net/mac/llm-youtube-transcripts-with-claude-and-gemini-in-shortcuts/).
+* **Commit messages**. Turn diffs into descriptive commit messages, e.g. `git diff | llm 'Write a concise git commit message explaining these changes'`. \
+* **Data extraction**. Convert free-text into structured JSON for automation. [Structured data extraction from unstructured content using LLM schemas](https://simonwillison.net/2025/Feb/28/llm-schemas/).
+
+[Previous
+
+Terminal: Bash](#/bash)
+
+[Next
+
+Spreadsheet: Excel, Google Sheets](#/spreadsheets)

markdown_files/Actor_Network_Visualization.md

@@ -0,0 +1,26 @@
+---
+title: "Actor Network Visualization"
+original_url: "https://tds.s-anand.net/#/actor-network-visualization?id=actor-network-visualization"
+downloaded_at: "2025-06-08T23:23:12.679629"
+---
+
+[Actor Network Visualization](#/actor-network-visualization?id=actor-network-visualization)
+-------------------------------------------------------------------------------------------
+
+Find the shortest path between Govinda & Angelina Jolie using IMDb data using Python: [networkx](https://pypi.org/project/networkx/) or [scikit-network](https://pypi.org/project/scikit-network).
+
+[![Jolie No. 1](https://i.ytimg.com/vi_webp/lcwMsPxPIjc/sddefault.webp)](https://youtu.be/lcwMsPxPIjc)
+
+* [Notebook: How this video was created](https://github.com/sanand0/jolie-no-1/blob/master/jolie-no-1.ipynb)
+* [The data used to visualize the network](https://github.com/sanand0/jolie-no-1/blob/master/imdb-actor-pairing.ipynb)
+* [The shortest path between actors](https://github.com/sanand0/jolie-no-1/blob/master/shortest-path.ipynb)
+* [IMDB data](https://developer.imdb.com/non-commercial-datasets/)
+* [Codebase](https://github.com/sanand0/jolie-no-1)
+
+[Previous
+
+Data Visualization with ChatGPT](#/data-visualization-with-chatgpt)
+
+[Next
+
+RAWgraphs](#/rawgraphs)

markdown_files/Authentication__Google_Auth.md

@@ -0,0 +1,93 @@
+---
+title: "Authentication: Google Auth"
+original_url: "https://tds.s-anand.net/#/google-auth?id=google-authentication-with-fastapi"
+downloaded_at: "2025-06-08T23:25:42.202598"
+---
+
+[Google Authentication with FastAPI](#/google-auth?id=google-authentication-with-fastapi)
+-----------------------------------------------------------------------------------------
+
+Secure your API endpoints using Google ID tokens to restrict access to specific email addresses.
+
+[![🔥 Python FastAPI Google Login Tutorial | OAuth2 Authentication (19 min)](https://i.ytimg.com/vi_webp/4ExQYRCwbzw/sddefault.webp)](https://youtu.be/4ExQYRCwbzw)
+
+Google Auth is the most commonly implemented single sign-on mechanism because:
+
+* It’s popular and user-friendly. Users can log in with their existing Google accounts.
+* It’s secure: Google supports OAuth2 and OpenID Connect to handle authentication.
+
+Here’s how you build a FastAPI app that identifies the user.
+
+1. Go to the [Google Cloud Console – Credentials](https://console.developers.google.com/apis/credentials) and click **Create Credentials > OAuth client ID**.
+2. Choose **Web application**, set your authorized redirect URIs (e.g., `http://localhost:8000/`).
+3. Copy the **Client ID** and **Client Secret** into a `.env` file:
+
+   ```
+   GOOGLE_CLIENT_ID=your-client-id.apps.googleusercontent.com
+   GOOGLE_CLIENT_SECRET=your-client-secretCopy to clipboardErrorCopied
+   ```
+4. Create your FastAPI `app.py`:
+
+```
+# /// script
+# dependencies = ["python-dotenv", "fastapi", "uvicorn", "itsdangerous", "httpx", "authlib"]
+# ///
+
+import os
+from dotenv import load_dotenv
+from fastapi import FastAPI, Request
+from fastapi.responses import RedirectResponse
+from starlette.middleware.sessions import SessionMiddleware
+from authlib.integrations.starlette_client import OAuth
+
+load_dotenv()
+app = FastAPI()
+app.add_middleware(SessionMiddleware, secret_key="create-a-random-secret-key")
+
+oauth = OAuth()
+oauth.register(
+    name="google",
+    client_id=os.getenv("GOOGLE_CLIENT_ID"),
+    client_secret=os.getenv("GOOGLE_CLIENT_SECRET"),
+    server_metadata_url="https://accounts.google.com/.well-known/openid-configuration",
+    client_kwargs={"scope": "openid email profile"},
+)
+
+@app.get("/")
+async def application(request: Request):
+    user = request.session.get("user")
+    # 3. For authenticated users: say hello
+    if user:
+        return f"Hello {user['email']}"
+    # 2. For users who have just logged in, save their details in the session
+    if "code" in request.query_params:
+        token = await oauth.google.authorize_access_token(request)
+        request.session["user"] = token["userinfo"]
+        return RedirectResponse("/")
+    # 1. For users who are logging in for the first time, redirect to Google login
+    return await oauth.google.authorize_redirect(request, request.url)
+
+if __name__ == "__main__":
+    import uvicorn
+    uvicorn.run(app, port=8000)Copy to clipboardErrorCopied
+```
+
+Now, run `uv run app.py`.
+
+1. When you visit <http://localhost:8000/> you’ll be redirected to a Google login page.
+2. When you log in, you’ll be redirected back to <http://localhost:8000/>
+3. Now you’ll see the email ID you logged in with.
+
+Instead of displaying the email, you can show different content based on the user. For example:
+
+* Allow access to specfic users and not others
+* Fetch the user’s personalized information
+* Display different content based on the user
+
+[Previous
+
+Web Framework: FastAPI](#/fastapi)
+
+[Next
+
+Local LLMs: Ollama](#/ollama)

markdown_files/BBC_Weather_API_with_Python.md

@@ -0,0 +1,74 @@
+---
+title: "BBC Weather API with Python"
+original_url: "https://tds.s-anand.net/#/bbc-weather-api-with-python?id=bbc-weather-location-id-with-python"
+downloaded_at: "2025-06-08T23:24:13.538036"
+---
+
+[BBC Weather location ID with Python](#/bbc-weather-api-with-python?id=bbc-weather-location-id-with-python)
+-----------------------------------------------------------------------------------------------------------
+
+[![BBC Weather location API with Python](https://i.ytimg.com/vi_webp/IafLrvnamAw/sddefault.webp)](https://youtu.be/IafLrvnamAw)
+
+You’ll learn how to get the location ID of any city from the BBC Weather API – as a precursor to scraping weather data – covering:
+
+* **Understanding API Calls**: Learn how backend API calls work when searching for a city on the BBC weather website.
+* **Inspecting Web Interactions**: Use the browser’s inspect element feature to track API calls and understand the network activity.
+* **Extracting Location IDs**: Identify and extract the location ID from the API response using Python.
+* **Using Python Libraries**: Import and use requests, json, and urlencode libraries to make API calls and process responses.
+* **Constructing API URLs**: Create structured API URLs dynamically with constant prefixes and query parameters using urlencode.
+* **Building Functions**: Develop a Python function that accepts a city name, constructs the API call, and returns the location ID.
+
+To open the browser Developer Tools on Chrome, Edge, or Firefox, you can:
+
+* Right-click on the page and select “Inspect” to open the developer tools
+* OR: Press `F12`
+* OR: Press `Ctrl+Shift+I` on Windows
+* OR: Press `Cmd+Opt+I` on Mac
+
+Here are links and references:
+
+* [BBC Location ID scraping - Notebook](https://colab.research.google.com/drive/1-iV-tbtRicKR_HXWeu4Hi5aXJCV3QdQp)
+* [BBC Weather - Palo Alto (location ID: 5380748)](https://www.bbc.com/weather/5380748)
+* [BBC Locator Service - Los Angeles](https://locator-service.api.bbci.co.uk/locations?api_key=AGbFAKx58hyjQScCXIYrxuEwJh2W2cmv&stack=aws&locale=en&filter=international&place-types=settlement%2Cairport%2Cdistrict&order=importance&s=los%20angeles&a=true&format=json)
+* Learn about the [`requests` package](https://docs.python-requests.org/en/latest/user/quickstart/). Watch [Python Requests Tutorial: Request Web Pages, Download Images, POST Data, Read JSON, and More](https://youtu.be/tb8gHvYlCFs)
+
+[BBC Weather data with Python](#/bbc-weather-api-with-python?id=bbc-weather-data-with-python)
+---------------------------------------------------------------------------------------------
+
+[![Scrape BBC weather with Python](https://i.ytimg.com/vi_webp/Uc4DgQJDRoI/sddefault.webp)](https://youtu.be/Uc4DgQJDRoI)
+
+You’ll learn how to scrape the live weather data of a city from the BBC Weather API, covering:
+
+* **Introduction to Web Scraping**: Understand the basics of web scraping and its legality.
+* **Libraries Overview**: Learn the importance of [`requests`](https://docs.python-requests.org/en/latest/user/quickstart/) and [`BeautifulSoup`](https://beautiful-soup-4.readthedocs.io/).
+* **Fetching HTML**: Use [`requests`](https://docs.python-requests.org/en/latest/user/quickstart/) to fetch HTML content from a web page.
+* **Parsing HTML**: Utilize [`BeautifulSoup`](https://beautiful-soup-4.readthedocs.io/) to parse and navigate the HTML content.
+* **Identifying Data**: Inspect HTML elements to locate specific data (e.g., high and low temperatures).
+* **Extracting Data**: Extract relevant data using [`BeautifulSoup`](https://beautiful-soup-4.readthedocs.io/)‘s `find_all()` function.
+* **Data Cleanup**: Clean extracted data to remove unwanted elements.
+* **Post-Processing**: Use regular expressions to split large strings into meaningful parts.
+* **Data Structuring**: Combine extracted data into a structured pandas DataFrame.
+* **Handling Special Characters**: Replace unwanted characters for better data manipulation.
+* **Saving Data**: Save the cleaned data into CSV and Excel formats.
+
+Here are links and references:
+
+* [BBC Weather scraping - Notebook](https://colab.research.google.com/drive/1-gkMzE-TKe3U_yh1v0NPn4TM687H2Hcf)
+* [BBC Locator Service - Mumbai](https://locator-service.api.bbci.co.uk/locations?api_key=AGbFAKx58hyjQScCXIYrxuEwJh2W2cmv&stack=aws&locale=en&filter=international&place-types=settlement%2Cairport%2Cdistrict&order=importance&s=mumbai&a=true&format=json)
+* [BBC Weather - Mumbai (location ID: 1275339)](https://www.bbc.com/weather/1275339)
+* [BBC Weather API - Mumbai (location ID: 1275339)](https://weather-broker-cdn.api.bbci.co.uk/en/forecast/aggregated/1275339)
+* Learn about the [`json` package](https://docs.python.org/3/library/json.html). Watch [Python Tutorial: Working with JSON Data using the json Module](https://youtu.be/9N6a-VLBa2I)
+* Learn about the [`BeautifulSoup` package](https://beautiful-soup-4.readthedocs.io/). Watch [Python Tutorial: Web Scraping with BeautifulSoup and Requests](https://youtu.be/ng2o98k983k)
+* Learn about the [`pandas` package](https://pandas.pydata.org/pandas-docs/stable/user_guide/10min.html). Watch
+  + [Python Pandas Tutorial (Part 1): Getting Started with Data Analysis - Installation and Loading Data](https://youtu.be/ZyhVh-qRZPA)
+  + [Python Pandas Tutorial (Part 2): DataFrame and Series Basics - Selecting Rows and Columns](https://youtu.be/zmdjNSmRXF4)
+* Learn about the [`re` package](https://docs.python.org/3/library/re.html). Watch [Python Tutorial: re Module - How to Write and Match Regular Expressions (Regex)](https://youtu.be/K8L6KVGG-7o)
+* Learn about the [`datetime` package](https://docs.python.org/3/library/datetime.html). Watch [Python Tutorial: Datetime Module - How to work with Dates, Times, Timedeltas, and Timezones](https://youtu.be/eirjjyP2qcQ)
+
+[Previous
+
+Crawling with the CLI](#/crawling-cli)
+
+[Next
+
+Scraping IMDb with JavaScript](#/scraping-imdb-with-javascript)

markdown_files/Base_64_Encoding.md

@@ -0,0 +1,77 @@
+---
+title: "Base 64 Encoding"
+original_url: "https://tds.s-anand.net/#/base64-encoding?id=base-64-encoding"
+downloaded_at: "2025-06-08T23:27:25.543180"
+---
+
+[Base 64 Encoding](#/base64-encoding?id=base-64-encoding)
+=========================================================
+
+Base64 is a method to convert binary data into ASCII text. It’s essential when you need to transmit binary data through text-only channels or embed binary content in text formats.
+
+Watch this quick explanation of how Base64 works (3 min):
+
+[![What is Base64? (3 min)](https://i.ytimg.com/vi_webp/8qkxeZmKmOY/sddefault.webp)](https://youtu.be/8qkxeZmKmOY)
+
+Here’s how it works:
+
+* It takes 3 bytes (24 bits) and converts them into 4 ASCII characters
+* … using 64 characters: A-Z, a-z, 0-9, + and / (padding with `=` to make the length a multiple of 4)
+* There’s a URL-safe variant of Base64 that replaces + and / with - and \_ to avoid issues in URLs
+* Base64 adds ~33% overhead (since every 3 bytes becomes 4 characters)
+
+Common Python operations with Base64:
+
+```
+import base64
+
+# Basic encoding/decoding
+text = "Hello, World!"
+# Convert text to base64
+encoded = base64.b64encode(text.encode()).decode()  # SGVsbG8sIFdvcmxkIQ==
+# Convert base64 back to text
+decoded = base64.b64decode(encoded).decode()        # Hello, World!
+# Convert to URL-safe base64
+url_safe = base64.urlsafe_b64encode(text.encode()).decode()  # SGVsbG8sIFdvcmxkIQ==
+
+# Working with binary files (e.g., images)
+with open('image.png', 'rb') as f:
+    binary_data = f.read()
+    image_b64 = base64.b64encode(binary_data).decode()
+
+# Data URI example (embed images in HTML/CSS)
+data_uri = f"data:image/png;base64,{image_b64}"Copy to clipboardErrorCopied
+```
+
+Data URIs allow embedding binary data directly in HTML/CSS. This reduces the number of HTTP requests and also works offline. But it increases the file size.
+
+For example, here’s an SVG image embedded as a data URI:
+
+```
+<img
+  src="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAzMiAzMiI+PGNpcmNsZSBjeD0iMTYiIGN5PSIxNiIgcj0iMTUiIGZpbGw9IiMyNTYzZWIiLz48cGF0aCBmaWxsPSIjZmZmIiBkPSJtMTYgNyAyIDcgNyAyLTcgMi0yIDctMi03LTctMiA3LTJaIi8+PC9zdmc+"
+/>Copy to clipboardErrorCopied
+```
+
+Base64 is used in many places:
+
+* JSON: Encoding binary data in JSON payloads
+* Email: MIME attachments encoding
+* Auth: HTTP Basic Authentication headers
+* JWT: Encoding tokens in web authentication
+* SSL/TLS: PEM certificate format
+* SAML: Encoding assertions in SSO
+* Git: Encoding binary files in patches
+
+Tools for working with Base64:
+
+* [Base64 Decoder/Encoder](https://www.base64decode.org/) for online encoding/decoding
+* [data: URI Generator](https://dopiaza.org/tools/datauri/index.php) converts files to Data URIs
+
+[Previous
+
+LLM Text Extraction](#/llm-text-extraction)
+
+[Next
+
+Vision Models](#/vision-models)

markdown_files/Browser__DevTools.md

@@ -0,0 +1,69 @@
+---
+title: "Browser: DevTools"
+original_url: "https://tds.s-anand.net/#/devtools?id=browser-devtools"
+downloaded_at: "2025-06-08T23:21:14.785028"
+---
+
+[Browser: DevTools](#/devtools?id=browser-devtools)
+---------------------------------------------------
+
+[Chrome DevTools](https://developer.chrome.com/docs/devtools/overview/) is the de facto standard for web development and data analysis in the browser.
+You’ll use this a lot when debugging and inspecting web pages.
+
+Here are the key features you’ll use most:
+
+1. **Elements Panel**
+
+   * Inspect and modify HTML/CSS in real-time
+   * Copy CSS selectors for web scraping
+   * Debug layout issues with the Box Model
+
+   ```
+   // Copy selector in Console
+   copy($0); // Copies selector of selected elementCopy to clipboardErrorCopied
+   ```
+2. **Console Panel**
+
+   * JavaScript REPL environment
+   * Log and debug data
+   * Common console methods:
+
+   ```
+   console.table(data); // Display data in table format
+   console.group("Name"); // Group related logs
+   console.time("Label"); // Measure execution timeCopy to clipboardErrorCopied
+   ```
+3. **Network Panel**
+
+   * Monitor API requests and responses
+   * Simulate slow connections
+   * Right-click on a request and select “Copy as fetch” to get the request.
+4. **Essential Keyboard Shortcuts**
+
+   * `Ctrl+Shift+I` (Windows) / `Cmd+Opt+I` (Mac): Open DevTools
+   * `Ctrl+Shift+C`: Select element to inspect
+   * `Ctrl+L`: Clear console
+   * `$0`: Reference currently selected element
+   * `$$('selector')`: Query selector all (returns array)
+
+Videos from Chrome Developers (37 min total):
+
+* [Fun & powerful: Intro to Chrome DevTools](https://youtu.be/t1c5tNPpXjs) (5 min)
+* [Different ways to open Chrome DevTools](https://youtu.be/X65TAP8a530) (5 min)
+* [Faster DevTools navigation with shortcuts and settings](https://youtu.be/xHusjrb_34A) (3 min)
+* [How to log messages in the Console](https://youtu.be/76U0gtuV9AY) (6 min)
+* [How to speed up your workflow with Console shortcuts](https://youtu.be/hdRDTj6ObiE) (6 min)
+* [HTML vs DOM? Let’s debug them](https://youtu.be/J-02VNxE7lE) (5 min)
+* [Caching demystified: Inspect, clear, and disable caches](https://youtu.be/mSMb-aH6sUw) (7 min)
+* [Console message logging](https://youtu.be/76U0gtuV9AY) (6 min)
+* [Console workflow shortcuts](https://youtu.be/hdRDTj6ObiE) (6 min)
+* [HTML vs DOM debugging](https://youtu.be/J-02VNxE7lE) (5 min)
+* [Cache inspection and management](https://youtu.be/mSMb-aH6sUw) (7 min)
+
+[Previous
+
+Unicode](#/unicode)
+
+[Next
+
+CSS Selectors](#/css-selectors)

markdown_files/CI_CD__GitHub_Actions.md

@@ -0,0 +1,79 @@
+---
+title: "CI/CD: GitHub Actions"
+original_url: "https://tds.s-anand.net/#/github-actions?id=cicd-github-actions"
+downloaded_at: "2025-06-08T23:24:27.252899"
+---
+
+[CI/CD: GitHub Actions](#/github-actions?id=cicd-github-actions)
+----------------------------------------------------------------
+
+[GitHub Actions](https://github.com/features/actions) is a powerful automation platform built into GitHub. It helps automate your development workflow - running tests, deploying applications, updating datasets, retraining models, etc.
+
+* Understand the basics of [YAML configuration files](https://docs.github.com/en/actions/writing-workflows/quickstart)
+* Explore the [pre-built actions from the marketplace](https://github.com/marketplace?type=actions)
+* How to [handle secrets securely](https://docs.github.com/en/actions/security-for-github-actions/security-guides/using-secrets-in-github-actions)
+* [Triggering a workflow](https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/triggering-a-workflow)
+* Staying within the [free tier limits](https://docs.github.com/en/billing/managing-billing-for-your-products/managing-billing-for-github-actions/about-billing-for-github-actions)
+* [Caching dependencies to speed up workflows](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows)
+
+Here is a sample `.github/workflows/iss-location.yml` that runs daily, appends the International Space Station location data into `iss-location.json`, and commits it to the repository.
+
+```
+name: Log ISS Location Data Daily
+
+on:
+  schedule:
+    # Runs at 12:00 UTC (noon) every day
+    - cron: "0 12 * * *"
+  workflow_dispatch: # Allows manual triggering
+
+jobs:
+  collect-iss-data:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+
+      - name: Fetch ISS location data
+        run: | # python
+          uv run --with requests python << 'EOF'
+          import requests
+
+          data = requests.get('http://api.open-notify.org/iss-now.json').text
+          with open('iss-location.jsonl', 'a') as f:
+              f.write(data + '\n')
+          'EOF'
+
+      - name: Commit and push changes
+        run: | # shell
+          git config --local user.email "github-actions[bot]@users.noreply.github.com"
+          git config --local user.name "github-actions[bot]"
+          git add iss-location.jsonl
+          git commit -m "Update ISS position data [skip ci]" || exit 0
+          git pushCopy to clipboardErrorCopied
+```
+
+Tools:
+
+* [GitHub CLI](https://cli.github.com/): Manage workflows from terminal
+* [Super-Linter](https://github.com/github/super-linter): Validate code style
+* [Release Drafter](https://github.com/release-drafter/release-drafter): Automate releases
+* [act](https://github.com/nektos/act): Run actions locally
+
+[![Github Actions CI/CD - Everything you need to know to get started](https://i.ytimg.com/vi_webp/mFFXuXjVgkU/sddefault.webp)](https://youtu.be/mFFXuXjVgkU)
+
+* [How to handle secrets in GitHub Actions](https://youtu.be/1tD7km5jK70)
+
+[Previous
+
+Serverless hosting: Vercel](#/vercel)
+
+[Next
+
+Containers: Docker, Podman](#/docker)

markdown_files/CORS.md

@@ -0,0 +1,88 @@
+---
+title: "CORS"
+original_url: "https://tds.s-anand.net/#/cors?id=cors-cross-origin-resource-sharing"
+downloaded_at: "2025-06-08T23:26:45.742238"
+---
+
+[CORS: Cross-Origin Resource Sharing](#/cors?id=cors-cross-origin-resource-sharing)
+-----------------------------------------------------------------------------------
+
+CORS (Cross-Origin Resource Sharing) is a security mechanism that controls how web browsers handle requests between different origins (domains, protocols, or ports). Data scientists need CORS for APIs serving data or analysis to a browser on a different domain.
+
+Watch this practical explanation of CORS (3 min):
+
+[![CORS in 100 Seconds](https://i.ytimg.com/vi_webp/4KHiSt0oLJ0/sddefault.webp)](https://youtu.be/4KHiSt0oLJ0)
+
+Key CORS concepts:
+
+* **Same-Origin Policy**: Browsers block requests between different origins by default
+* **CORS Headers**: Server responses must include specific headers to allow cross-origin requests
+* **Preflight Requests**: Browsers send OPTIONS requests to check if the actual request is allowed
+* **Credentials**: Special handling required for requests with cookies or authentication
+
+If you’re exposing your API with a GET request publicly, the only thing you need to do is set the HTTP header `Access-Control-Allow-Origin: *`.
+
+Here are other common CORS headers:
+
+```
+Access-Control-Allow-Origin: https://example.com
+Access-Control-Allow-Methods: GET, POST, PUT, DELETE
+Access-Control-Allow-Headers: Content-Type, Authorization
+Access-Control-Allow-Credentials: trueCopy to clipboardErrorCopied
+```
+
+To implement CORS in FastAPI, use the [`CORSMiddleware` middleware](https://fastapi.tiangolo.com/tutorial/cors/):
+
+```
+from fastapi import FastAPI
+from fastapi.middleware.cors import CORSMiddleware
+
+app = FastAPI()
+
+app.add_middleware(CORSMiddleware, allow_origins=["*"]) # Allow GET requests from all origins
+# Or, provide more granular control:
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["https://example.com"],  # Allow a specific domain
+    allow_credentials=True,  # Allow cookies
+    allow_methods=["GET", "POST", "PUT", "DELETE"],  # Allow specific methods
+    allow_headers=["*"],  # Allow all headers
+)Copy to clipboardErrorCopied
+```
+
+Testing CORS with JavaScript:
+
+```
+// Simple request
+const response = await fetch("https://api.example.com/data", {
+  method: "GET",
+  headers: { "Content-Type": "application/json" },
+});
+
+// Request with credentials
+const response = await fetch("https://api.example.com/data", {
+  credentials: "include",
+  headers: { "Content-Type": "application/json" },
+});Copy to clipboardErrorCopied
+```
+
+Useful CORS debugging tools:
+
+* [CORS Checker](https://cors-test.codehappy.dev/): Test CORS configurations
+* Browser DevTools Network tab: Inspect CORS headers and preflight requests
+* [cors-anywhere](https://github.com/Rob--W/cors-anywhere): CORS proxy for development
+
+Common CORS errors and solutions:
+
+* `No 'Access-Control-Allow-Origin' header`: Configure server to send proper CORS headers
+* `Request header field not allowed`: Add required headers to `Access-Control-Allow-Headers`
+* `Credentials flag`: Set both `credentials: 'include'` and `Access-Control-Allow-Credentials: true`
+* `Wild card error`: Cannot use `*` with credentials; specify exact origins
+
+[Previous
+
+Tunneling: ngrok](#/ngrok)
+
+[Next
+
+REST APIs](#/rest-apis)

markdown_files/CSS_Selectors.md

@@ -0,0 +1,39 @@
+---
+title: "CSS Selectors"
+original_url: "https://tds.s-anand.net/#/css-selectors?id=css-selectors"
+downloaded_at: "2025-06-08T23:24:42.527184"
+---
+
+[CSS Selectors](#/css-selectors?id=css-selectors)
+-------------------------------------------------
+
+CSS selectors are patterns used to select and style HTML elements on a web page. They are fundamental to web development and data scraping, allowing you to precisely target elements for styling or extraction.
+
+For data scientists, understanding CSS selectors is crucial when:
+
+* Web scraping with tools like Beautiful Soup or Scrapy
+* Selecting elements for browser automation with Selenium
+* Styling data visualizations and web applications
+* Debugging website issues using browser DevTools
+
+Watch this comprehensive introduction to CSS selectors (20 min):
+
+[![Learn Every CSS Selector In 20 Minutes (20 min)](https://i.ytimg.com/vi_webp/l1mER1bV0N0/sddefault.webp)](https://youtu.be/l1mER1bV0N0)
+
+The Mozilla Developer Network (MDN) provides detailed documentation on the three main types of selectors:
+
+* [Basic CSS selectors](https://developer.mozilla.org/en-US/docs/Learn_web_development/Core/Styling_basics/Basic_selectors): Learn about element (`div`), class (`.container`), ID (`#header`), and universal (`*`) selectors
+* [Attribute selectors](https://developer.mozilla.org/en-US/docs/Learn_web_development/Core/Styling_basics/Attribute_selectors): Target elements based on their attributes or attribute values (`[type="text"]`)
+* [Combinators](https://developer.mozilla.org/en-US/docs/Learn_web_development/Core/Styling_basics/Combinators): Use relationships between elements (`div > p`, `div + p`, `div ~ p`)
+
+Practice your CSS selector skills with this interactive tool:
+
+* [CSS Diner](https://flukeout.github.io/): A fun game that teaches CSS selectors through increasingly challenging levels
+
+[Previous
+
+Browser: DevTools](#/devtools)
+
+[Next
+
+JSON](#/json)

markdown_files/Cleaning_Data_with_OpenRefine.md

@@ -0,0 +1,31 @@
+---
+title: "Cleaning Data with OpenRefine"
+original_url: "https://tds.s-anand.net/#/cleaning-data-with-openrefine?id=cleaning-data-with-openrefine"
+downloaded_at: "2025-06-08T23:26:48.911609"
+---
+
+[Cleaning Data with OpenRefine](#/cleaning-data-with-openrefine?id=cleaning-data-with-openrefine)
+-------------------------------------------------------------------------------------------------
+
+[![Cleaning data with OpenRefine](https://i.ytimg.com/vi_webp/zxEtfHseE84/sddefault.webp)](https://youtu.be/zxEtfHseE84)
+
+This session covers the use of OpenRefine for data cleaning, focusing on resolving entity discrepancies:
+
+* **Data Upload and Project Creation**: Import data into OpenRefine and create a new project for analysis.
+* **Faceting Data**: Use text facets to group similar entries and identify frequency of address crumbs.
+* **Clustering Methodology**: Apply clustering algorithms to merge similar entries with minor differences, such as punctuation.
+* **Manual and Automated Clustering**: Learn to merge clusters manually or in one go, trusting the system’s clustering accuracy.
+* **Entity Resolution**: Clean and save the data by resolving multiple versions of the same entity using Open Refine.
+
+Here are links used in the video:
+
+* [OpenRefine software](https://openrefine.org)
+* [Dataset for OpenRefine](https://drive.google.com/file/d/1ccu0Xxk8UJUa2Dz4lihmvzhLjvPy42Ai/view)
+
+[Previous
+
+Data Preparation in the Editor](#/data-preparation-in-the-editor)
+
+[Next
+
+Profiling Data with Python](#/profiling-data-with-python)

markdown_files/Containers__Docker,_Podman.md

@@ -0,0 +1,94 @@
+---
+title: "Containers: Docker, Podman"
+original_url: "https://tds.s-anand.net/#/docker?id=containers-docker-podman"
+downloaded_at: "2025-06-08T23:26:01.579602"
+---
+
+[Containers: Docker, Podman](#/docker?id=containers-docker-podman)
+------------------------------------------------------------------
+
+[Docker](https://www.docker.com/) and [Podman](https://podman.io/) are containerization tools that package your application and its dependencies into a standardized unit for software development and deployment.
+
+Docker is the industry standard. Podman is compatible with Docker and has better security (and a slightly more open license). In this course, we recommend Podman but Docker works in the same way.
+
+Initialize the container engine:
+
+```
+podman machine init
+podman machine startCopy to clipboardErrorCopied
+```
+
+Common Operations. (You can use `docker` instead of `podman` in the same way.)
+
+```
+# Pull an image
+podman pull python:3.11-slim
+
+# Run a container
+podman run -it python:3.11-slim
+
+# List containers
+podman ps -a
+
+# Stop container
+podman stop container_id
+
+# Scan image for vulnerabilities
+podman scan myapp:latest
+
+# Remove container
+podman rm container_id
+
+# Remove all stopped containers
+podman container pruneCopy to clipboardErrorCopied
+```
+
+You can create a `Dockerfile` to build a container image. Here’s a sample `Dockerfile` that converts a Python script into a container image.
+
+```
+FROM python:3.11-slim
+# Set working directory
+WORKDIR /app
+# Typically, you would use `COPY . .` to copy files from the host machine,
+# but here we're just using a simple script.
+RUN echo 'print("Hello, world!")' > app.py
+# Run the script
+CMD ["python", "app.py"]Copy to clipboardErrorCopied
+```
+
+To build, run, and deploy the container, run these commands:
+
+```
+# Create an account on https://hub.docker.com/ and then login
+podman login docker.io
+
+# Build and run the container
+podman build -t py-hello .
+podman run -it py-hello
+
+# Push the container to Docker Hub. Replace $DOCKER_HUB_USERNAME with your Docker Hub username.
+podman push py-hello:latest docker.io/$DOCKER_HUB_USERNAME/py-hello
+
+# Push adding a specific tag, e.g. dev
+TAG=dev podman push py-hello docker.io/$DOCKER_HUB_USERNAME/py-hello:$TAGCopy to clipboardErrorCopied
+```
+
+Tools:
+
+* [Dive](https://github.com/wagoodman/dive): Explore image layers
+* [Skopeo](https://github.com/containers/skopeo): Work with container images
+* [Trivy](https://github.com/aquasecurity/trivy): Security scanner
+
+[![Podman Tutorial Zero to Hero | Full 1 Hour Course](https://i.ytimg.com/vi_webp/YXfA5O5Mr18/sddefault.webp)](https://youtu.be/YXfA5O5Mr18)
+
+[![Learn Docker in 7 Easy Steps - Full Beginner's Tutorial](https://i.ytimg.com/vi_webp/gAkwW2tuIqE/sddefault.webp)](https://youtu.be/gAkwW2tuIqE)
+
+* Optional: For Windows, see [WSL 2 with Docker getting started](https://youtu.be/5RQbdMn04Oc)
+
+[Previous
+
+CI/CD: GitHub Actions](#/github-actions)
+
+[Next
+
+DevContainers: GitHub Codespaces](#/github-codespaces)

markdown_files/Convert_HTML_to_Markdown.md

@@ -0,0 +1,183 @@
+---
+title: "Convert HTML to Markdown"
+original_url: "https://tds.s-anand.net/#/convert-html-to-markdown?id=markdown-crawler"
+downloaded_at: "2025-06-08T23:25:38.805247"
+---
+
+[Converting HTML to Markdown](#/convert-html-to-markdown?id=converting-html-to-markdown)
+----------------------------------------------------------------------------------------
+
+When working with web content, converting HTML files to plain text or Markdown is a common requirement for content extraction, analysis, and preservation. For example:
+
+* **Content analysis**: Extract clean text from HTML for natural language processing
+* **Data mining**: Strip formatting to focus on the actual content
+* **Offline reading**: Convert web pages to readable formats for e-readers or offline consumption
+* **Content migration**: Move content between different CMS platforms
+* **SEO analysis**: Extract headings, content structure, and text for optimization
+* **Archive creation**: Store web content in more compact, preservation-friendly formats
+* **Accessibility**: Convert content to formats that work better with screen readers
+
+This tutorial covers both converting existing HTML files and combining web crawling with HTML-to-text conversion in a single workflow – all using the command line.
+
+### [defuddle-cli](#/convert-html-to-markdown?id=defuddle-cli)
+
+[defuddle-cli](https://github.com/defuddle/defuddle) specializes in HTML - Markdown conversion. It’s a bit slow and not very customizable but produces clean Markdown that preserves structure, links, and basic formatting. Best for content where preserving the document structure is important.
+
+```
+find . -name '*.html' -exec npx --package defuddle-cli -y defuddle parse {} --md -o {}.md \;Copy to clipboardErrorCopied
+```
+
+* `find . -name '*.html'`: Finds all HTML files in the current directory and subdirectories
+* `-exec ... \;`: Executes the following command for each file found
+* `npx --package defuddle-cli -y`: Installs and runs defuddle-cli without prompting
+* `defuddle parse {} --md`: Parses the HTML file (represented by `{}`) and converts to markdown
+* `-o {}.md`: Outputs to a file with the original name plus .md extension
+
+### [Pandoc](#/convert-html-to-markdown?id=pandoc)
+
+[Pandoc](https://pandoc.org/) is a bit slow and highly customizable, preserving almost all formatting elements, leading to verbose markdown. Best for academic or documentation conversion where precision matters.
+
+Pandoc can convert from many other formats (such as Word, PDF, LaTeX, etc.) to Markdown and vice versa, making it one of most popular and versatele document convertors.
+
+[![How to Convert a Word Document to Markdown for Free using Pandoc (12 min)](https://i.ytimg.com/vi/HPSK7q13-40/sddefault.jpg)](https://youtu.be/HPSK7q13-40)
+
+```
+find . -name '*.html' -exec pandoc -f html -t markdown_strict -o {}.md {} \;Copy to clipboardErrorCopied
+```
+
+* `find . -name '*.html'`: Finds all HTML files in the current directory and subdirectories
+* `-exec ... \;`: Executes the following command for each file found
+* `pandoc`: The Swiss Army knife of document conversion
+* `-f html -t markdown_strict`: Convert from HTML format to strict markdown
+* `-o {}.md {}`: Output to a markdown file, with the input file as the last argument
+
+### [Lynx](#/convert-html-to-markdown?id=lynx)
+
+[Lynx](https://lynx.invisible-island.net/) is fast and generates text (not Markdown) with minimal formatting. Lynx renders the HTML as it would appear in a text browser, preserving basic structure but losing complex formatting. Best for quick content extraction or when processing large numbers of files.
+
+```
+find . -type f -name '*.html' -exec sh -c 'for f; do lynx -dump -nolist "$f" > "${f%.html}.txt"; done' _ {} +Copy to clipboardErrorCopied
+```
+
+* `find . -type f -name '*.html'`: Finds all HTML files in the current directory and subdirectories
+* `-exec sh -c '...' _ {} +`: Executes a shell command with batched files for efficiency
+* `for f; do ... done`: Loops through each file in the batch
+* `lynx -dump -nolist "$f"`: Uses the lynx text browser to render HTML as plain text
+  + `-dump`: Output the rendered page to stdout
+  + `-nolist`: Don’t include the list of links at the end
+* `> "${f%.html}.txt"`: Save output to a .txt file with the same base name
+
+### [w3m](#/convert-html-to-markdown?id=w3m)
+
+[w3m](https://w3m.sourceforge.net/) is very slow processing with minimal formatting. w3m tends to be more thorough in its rendering than lynx but takes considerably longer. It supports basic JavaScript processing, making it better at handling modern websites with dynamic content. Best for cases where you need slightly better rendering than lynx, particularly for complex layouts and tables, and when some JavaScript processing is beneficial.
+
+```
+find . -type f -name '*.html' \
+  -exec sh -c 'for f; do \
+      w3m -dump -T text/html -cols 80 -no-graph "$f" > "${f%.html}.md"; \
+    done' _ {} +Copy to clipboardErrorCopied
+```
+
+* `find . -type f -name '*.html'`: Finds all HTML files in the current directory and subdirectories
+* `-exec sh -c '...' _ {} +`: Executes a shell command with batched files for efficiency
+* `for f; do ... done`: Loops through each file in the batch
+* `w3m -dump -T text/html -cols 80 -no-graph "$f"`: Uses the w3m text browser to render HTML
+  + `-dump`: Output the rendered page to stdout
+  + `-T text/html`: Specify input format as HTML
+  + `-cols 80`: Set output width to 80 columns
+  + `-no-graph`: Don’t show graphic characters for tables and frames
+* `> "${f%.html}.md"`: Save output to a .md file with the same base name
+
+### [Comparison](#/convert-html-to-markdown?id=comparison)
+
+| Approach | Speed | Format Quality | Preservation | Best For |
+| --- | --- | --- | --- | --- |
+| defuddle-cli | Slow | High | Good structure and links | Content migration, publishing |
+| pandoc | Slow | Very High | Almost everything | Academic papers, documentation |
+| lynx | Fast | Low | Basic structure only | Quick extraction, large batches |
+| w3m | Very Slow | Medium-Low | Basic structure with better tables | Improved readability over lynx |
+
+### [Optimize Batch Processing](#/convert-html-to-markdown?id=optimize-batch-processing)
+
+1. **Process in parallel**: Use GNU Parallel for multi-core processing:
+
+   ```
+   find . -name "*.html" | parallel "pandoc -f html -t markdown_strict -o {}.md {}"Copy to clipboardErrorCopied
+   ```
+2. **Filter files before processing**:
+
+   ```
+   find . -name "*.html" -type f -size -1M -exec pandoc -f html -t markdown {} -o {}.md \;Copy to clipboardErrorCopied
+   ```
+3. **Customize output format** with additional parameters:
+
+   ```
+   # For pandoc, preserve line breaks but simplify other formatting
+   find . -name "*.html" -exec pandoc -f html -t markdown --wrap=preserve --atx-headers {} -o {}.md \;Copy to clipboardErrorCopied
+   ```
+4. **Handle errors gracefully**:
+
+   ```
+   find . -name "*.html" -exec sh -c 'for f; do pandoc -f html -t markdown "$f" -o "${f%.html}.md" 2>/dev/null || echo "Failed: $f" >> conversion_errors.log; done' _ {} +Copy to clipboardErrorCopied
+   ```
+
+### [Choosing the Right Tool](#/convert-html-to-markdown?id=choosing-the-right-tool)
+
+* **Need speed with minimal formatting?** Use the lynx approach
+* **Need precise, complete conversion?** Use pandoc
+* **Need a balance of structure and cleanliness?** Try defuddle-cli
+* **Working with complex tables?** w3m might render them better
+
+Remember that the best approach depends on your specific use case, volume of files, and how you intend to use the converted text.
+
+### [Combined Crawling and Conversion](#/convert-html-to-markdown?id=combined-crawling-and-conversion)
+
+Sometimes you need to both crawl a website and convert its content to markdown or text in a single workflow, like [Crawl4AI](#/convert-html-to-markdown?id=crawl4ai) or [markdown-crawler](#/convert-html-to-markdown?id=markdown-crawler).
+
+1. **For research/data collection**: Use a specialized crawler (like Crawl4AI) with post-processing conversion
+2. **For simple website archiving**: Markdown-crawler provides a convenient all-in-one solution
+3. **For high-quality conversion**: Use wget/wget2 for crawling followed by pandoc for conversion
+4. **For maximum speed**: Combine wget with lynx in a pipeline
+
+### [Crawl4AI](#/convert-html-to-markdown?id=crawl4ai)
+
+[Crawl4AI](https://github.com/crawl4ai/crawl4ai) is designed for single-page extraction with high-quality content processing. Crawl4AI is optimized for AI training data extraction, focusing on clean, structured content rather than complete site preservation. It excels at removing boilerplate content and preserving the main article text.
+
+```
+uv venv
+source .venv/bin/activate.fish
+uv pip install crawl4ai
+crawl4ai-setupCopy to clipboardErrorCopied
+```
+
+* `uv venv`: Creates a Python virtual environment using uv (a faster alternative to virtualenv)
+* `source .venv/bin/activate.fish`: Activates the virtual environment (fish shell syntax)
+* `uv pip install crawl4ai`: Installs the crawl4ai package
+* `crawl4ai-setup`: Initializes crawl4ai’s required dependencies
+
+### [markdown-crawler](#/convert-html-to-markdown?id=markdown-crawler)
+
+[markdown-crawler](https://pypi.org/project/markdown-crawler/) combines web crawling with markdown conversion in one tool. It’s efficient for bulk processing but tends to produce lower-quality markdown conversion compared to specialized converters like pandoc or defuddle. Best for projects where quantity and integration are more important than perfect formatting.
+
+```
+uv venv
+source .venv/bin/activate.fish
+uv pip install markdown-crawler
+markdown-crawler -t 5 -d 3 -b ./markdown https://study.iitm.ac.in/ds/Copy to clipboardErrorCopied
+```
+
+* `uv venv` and activation: Same as above
+* `uv pip install markdown-crawler`: Installs the markdown-crawler package
+* `markdown-crawler`: Runs the crawler with these options:
+  + `-t 5`: Sets 5 threads for parallel crawling
+  + `-d 3`: Limits crawl depth to 3 levels
+  + `-b ./markdown`: Sets the base output directory
+  + Final argument is the starting URL
+
+[Previous
+
+Convert PDFs to Markdown](#/convert-pdfs-to-markdown)
+
+[Next
+
+LLM Website Scraping](#/llm-website-scraping)

markdown_files/Convert_PDFs_to_Markdown.md

@@ -0,0 +1,139 @@
+---
+title: "Convert PDFs to Markdown"
+original_url: "https://tds.s-anand.net/#/convert-pdfs-to-markdown?id=tips-for-optimal-pdf-conversion"
+downloaded_at: "2025-06-08T23:25:59.398450"
+---
+
+[Converting PDFs to Markdown](#/convert-pdfs-to-markdown?id=converting-pdfs-to-markdown)
+----------------------------------------------------------------------------------------
+
+PDF documents are ubiquitous in academic, business, and technical contexts, but extracting and repurposing their content can be challenging. This tutorial explores various command-line tools for converting PDFs to Markdown format, with a focus on preserving structure and formatting suitable for different use cases, including preparation for Large Language Models (LLMs).
+
+Use Cases:
+
+* **LLM training and fine-tuning**: Create clean text data from PDFs for AI model training
+* **Knowledge base creation**: Transform PDFs into searchable, editable markdown documents
+* **Content repurposing**: Convert academic papers and reports for web publication
+* **Data extraction**: Pull structured content from PDF documents for analysis
+* **Accessibility**: Convert PDFs to more accessible formats for screen readers
+* **Citation and reference management**: Extract bibliographic information from academic papers
+* **Documentation conversion**: Transform technical PDFs into maintainable documentation
+
+### [PyMuPDF4LLM](#/convert-pdfs-to-markdown?id=pymupdf4llm)
+
+[PyMuPDF4LLM](https://pymupdf.readthedocs.io/en/latest/pymupdf4llm/) is a specialized component of the PyMuPDF library that generates Markdown specifically formatted for Large Language Models. It produces high-quality markdown with good preservation of document structure. It’s specifically optimized for producing text that works well with LLMs, removing irrelevant formatting while preserving semantic structure. Requires PyTorch, which adds dependencies but enables more advanced processing capabilities.
+
+PyMuPDF4LLM uses [MuPDF](https://mupdf.com/) as its PDF parsing engine. [PyMuPDF](https://pymupdf.readthedocs.io/) is emerging as a strong default for PDF text extraction due to its accuracy and performance in handling complex PDF structures.
+
+```
+PYTHONUTF8=1 uv run --with pymupdf4llm python -c 'import pymupdf4llm; h = open("pymupdf4llm.md", "w"); h.write(pymupdf4llm.to_markdown("$FILE.pdf"))'Copy to clipboardErrorCopied
+```
+
+* `PYTHONUTF8=1`: Forces Python to use UTF-8 encoding regardless of system locale
+* `uv run --with pymupdf4llm`: Uses uv package manager to run Python with the pymupdf4llm package
+* `python -c '...'`: Executes Python code directly from the command line
+* `import pymupdf4llm`: Imports the PDF-to-Markdown module
+* `h = open("pymupdf4llm.md", "w")`: Creates a file to write the markdown output
+* `h.write(pymupdf4llm.to_markdown("$FILE.pdf"))`: Converts the PDF to markdown and writes to file
+
+[Markitdown](#/convert-pdfs-to-markdown?id=markitdown)
+------------------------------------------------------
+
+[![Microsoft MarkItDown - Convert Files and Office Documents to Markdown - Install Locally (9 min)](https://i.ytimg.com/vi/v65Oyddfxeg/sddefault.jpg)](https://youtu.be/v65Oyddfxeg)
+
+[Markitdown](https://github.com/microsoft/markitdown) is Microsoft’s tool for converting various document formats to Markdown, including PDFs, DOCX, XLSX, PPTX, and ZIP files. It’s a versatile multi-format converter that handles PDFs via PDFMiner, DOCX via Mammoth, XLSX via Pandas, and PPTX via Python-PPTX. Good for batch processing of mixed document types. The quality of PDF conversion is generally good but may struggle with complex layouts or heavily formatted documents.
+
+```
+PYTHONUTF8=1 uvx markitdown $FILE.pdf > markitdown.mdCopy to clipboardErrorCopied
+```
+
+* `PYTHONUTF8=1`: Forces Python to use UTF-8 encoding
+* `uvx markitdown`: Runs the markitdown tool via the uv package manager
+* `$FILE.pdf`: The input PDF file
+* `> markitdown.md`: Redirects output to a markdown file
+
+### [Unstructured](#/convert-pdfs-to-markdown?id=unstructured)
+
+[Unstructured](https://unstructured.io/) is rapidly becoming the de facto library for parsing over 40 different file types. It is excellent for extracting text and tables from diverse document formats. Particularly useful for generating clean content to pass to LLMs. Strong community support and actively maintained.
+
+[GROBID](#/convert-pdfs-to-markdown?id=grobid)
+----------------------------------------------
+
+If you specifically need to parse references from text-native PDFs or reliably OCR’ed ones, [GROBID](https://github.com/kermitt2/grobid) remains the de facto choice. It excels at extracting structured bibliographic information with high accuracy.
+
+```
+# Start GROBID service
+docker run -t --rm -p 8070:8070 lfoppiano/grobid:0.7.2
+
+# Process PDF with curl
+curl -X POST -F "input=@paper.pdf" localhost:8070/api/processFulltextDocument > references.tei.xmlCopy to clipboardErrorCopied
+```
+
+### [Mistral OCR API](#/convert-pdfs-to-markdown?id=mistral-ocr-api)
+
+[Mistral OCR](https://mistral.ai/products/ocr/) offers an end-to-end cloud API that preserves both text and layout, making it easier to isolate specific sections like References. It shows the most promise currently, though it requires post-processing.
+
+[Azure Document Intelligence API](#/convert-pdfs-to-markdown?id=azure-document-intelligence-api)
+------------------------------------------------------------------------------------------------
+
+For enterprise users already in the Microsoft ecosystem, [Azure Document Intelligence](https://azure.microsoft.com/en-us/products/ai-services/document-intelligence) provides excellent raw OCR with enterprise SLAs. May require custom model training or post-processing to match GROBID’s reference extraction capabilities.
+
+### [Other libraries](#/convert-pdfs-to-markdown?id=other-libraries)
+
+[Docling](https://github.com/DS4SD/docling) is IBM’s document understanding library that supports PDF conversion. It can be challenging to install, particularly on Windows and some Linux distributions. Offers advanced document understanding capabilities beyond simple text extraction.
+
+[MegaParse](https://github.com/QuivrHQ/MegaParse) takes a comprehensive approach using LibreOffice, Pandoc, Tesseract OCR, and other tools. It has Robust handling of different document types but requires an OpenAI API key for some features. Good for complex documents but has significant dependencies.
+
+[Comparison of PDF-to-Markdown Tools](#/convert-pdfs-to-markdown?id=comparison-of-pdf-to-markdown-tools)
+--------------------------------------------------------------------------------------------------------
+
+| Tool | Strengths | Weaknesses | Best For |
+| --- | --- | --- | --- |
+| PyMuPDF4LLM | Structure preservation, LLM optimization | Requires PyTorch | AI training data, semantic structure |
+| Markitdown | Multi-format support, simple usage | Less precise layout handling | Batch processing, mixed documents |
+| Unstructured | Wide format support, active development | Can be resource-intensive | Production pipelines, integration |
+| GROBID | Reference extraction excellence | Narrower use case | Academic papers, citations |
+| Docling | Advanced document understanding | Installation difficulties | Research applications |
+| MegaParse | Comprehensive approach | Requires OpenAI API | Complex documents, OCR needs |
+
+How to pick:
+
+* **Need LLM-ready content?** PyMuPDF4LLM is specifically designed for this
+* **Working with multiple document formats?** Markitdown handles diverse inputs
+* **Extracting academic references?** GROBID remains the standard
+* **Building a production pipeline?** Unstructured offers the best integration options
+* **Handling complex layouts?** Consider commercial OCR like Mistral or Azure Document Intelligence
+
+The optimal approach depends on your specific requirements regarding accuracy, structure preservation, and the intended use of the extracted content.
+
+[Tips for Optimal PDF Conversion](#/convert-pdfs-to-markdown?id=tips-for-optimal-pdf-conversion)
+------------------------------------------------------------------------------------------------
+
+1. **Pre-process PDFs** when possible:
+
+   ```
+   # Optimize a PDF for text extraction first
+   ocrmypdf --optimize 3 --skip-text input.pdf optimized.pdfCopy to clipboardErrorCopied
+   ```
+2. **Try multiple tools** on the same document to compare results:
+3. **Handle scanned PDFs** appropriately:
+
+   ```
+   # For scanned documents, run OCR first
+   ocrmypdf --force-ocr input.pdf ocr_ready.pdf
+   PYTHONUTF8=1 uvx markitdown ocr_ready.pdf > markitdown.mdCopy to clipboardErrorCopied
+   ```
+4. **Consider post-processing** for better results:
+
+   ```
+   # Simple post-processing example
+   sed -i 's/\([A-Z]\)\./\1\.\n/g' output.md  # Add line breaks after sentencesCopy to clipboardErrorCopied
+   ```
+
+[Previous
+
+Scraping PDFs with Tabula](#/scraping-pdfs-with-tabula)
+
+[Next
+
+Convert HTML to Markdown](#/convert-html-to-markdown)

markdown_files/Correlation_with_Excel.md

@@ -0,0 +1,33 @@
+---
+title: "Correlation with Excel"
+original_url: "https://tds.s-anand.net/#/correlation-with-excel?id=correlation-with-excel"
+downloaded_at: "2025-06-08T23:24:31.921246"
+---
+
+[Correlation with Excel](#/correlation-with-excel?id=correlation-with-excel)
+----------------------------------------------------------------------------
+
+[![Correlation with Excel](https://i.ytimg.com/vi_webp/lXHCyhO7DmY/sddefault.webp)](https://youtu.be/lXHCyhO7DmY)
+
+You’ll learn to calculate and interpret correlations using Excel, covering:
+
+* **Enabling the Data Analysis Tool Pack**: Steps to enable the Excel data analysis tool pack.
+* **Correlation Analysis**: Understanding statistical association between variables.
+* **Creating a Correlation Matrix**: Steps to generate and interpret a correlation matrix.
+* **Scatterplots and Trendlines**: Plotting data and adding trend lines to visualize correlations.
+* **Analyzing Results**: Comparing correlation coefficients and understanding their implications.
+* **Insights and Further Analysis**: Interpreting scatterplots and planning further analysis for deeper insights.
+
+Here are the links used in the video:
+
+* [Understand correlation](https://www.khanacademy.org/math/ap-statistics/bivariate-data-ap/correlation-coefficient-r/v/correlation-coefficient-intuition-examples)
+* [COVID-19 vaccinations data explorer - Website](https://ourworldindata.org/covid-vaccinations?country=OWID_WRL)
+* [COVID-19 vaccinations - Correlations Excel file](https://docs.google.com/spreadsheets/d/1_vQF2i5ubKmHQMBqoTwsu6AlevWsQtTD/view#gid=790744269)
+
+[Previous
+
+6. Data Analysis](#/data-analysis)
+
+[Next
+
+Regression with Excel](#/regression-with-excel)

markdown_files/Crawling_with_the_CLI.md

@@ -0,0 +1,137 @@
+---
+title: "Crawling with the CLI"
+original_url: "https://tds.s-anand.net/#/crawling-cli?id=wpull"
+downloaded_at: "2025-06-08T23:26:52.185904"
+---
+
+[Crawling with the CLI](#/crawling-cli?id=crawling-with-the-cli)
+----------------------------------------------------------------
+
+Since websites are a common source of data, we often download entire websites (crawling) and then process them offline.
+
+Web crawling is essential in many data-driven scenarios:
+
+* **Data mining and analysis**: Gathering structured data from multiple pages for market research, competitive analysis, or academic research
+* **Content archiving**: Creating offline copies of websites for preservation or backup purposes
+* **SEO analysis**: Analyzing site structure, metadata, and content to improve search rankings
+* **Legal compliance**: Capturing website content for regulatory or compliance documentation
+* **Website migration**: Creating a complete copy before moving to a new platform or design
+* **Offline access**: Downloading educational resources, documentation, or reference materials for use without internet connection
+
+The most commonly used tool for fetching websites is [`wget`](https://www.gnu.org/software/wget/). It is pre-installed in many UNIX distributions and easy to install.
+
+[![Scraping Websites using Wget (8 min)](https://i.ytimg.com/vi/pLfH5TZBGXo/sddefault.jpg)](https://youtu.be/pLfH5TZBGXo)
+
+To crawl the [IIT Madras Data Science Program website](https://study.iitm.ac.in/ds/) for example, you could run:
+
+```
+wget \
+  --recursive \
+  --level=3 \
+  --no-parent \
+  --convert-links \
+  --adjust-extension \
+  --compression=auto \
+  --accept html,htm \
+  --directory-prefix=./ds \
+  https://study.iitm.ac.in/ds/Copy to clipboardErrorCopied
+```
+
+Here’s what each option does:
+
+* `--recursive`: Enables recursive downloading (following links)
+* `--level=3`: Limits recursion depth to 3 levels from the initial URL
+* `--no-parent`: Restricts crawling to only URLs below the initial directory
+* `--convert-links`: Converts all links in downloaded documents to work locally
+* `--adjust-extension`: Adds proper extensions to files (.html, .jpg, etc.) based on MIME types
+* `--compression=auto`: Automatically handles compressed content (gzip, deflate)
+* `--accept html,htm`: Only downloads files with these extensions
+* `--directory-prefix=./ds`: Saves all downloaded files to the specified directory
+
+[wget2](https://gitlab.com/gnuwget/wget2) is a better version of `wget` and supports HTTP2, parallel connections, and only updates modified sites. The syntax is (mostly) the same.
+
+```
+wget2 \
+  --recursive \
+  --level=3 \
+  --no-parent \
+  --convert-links \
+  --adjust-extension \
+  --compression=auto \
+  --accept html,htm \
+  --directory-prefix=./ds \
+  https://study.iitm.ac.in/ds/Copy to clipboardErrorCopied
+```
+
+There are popular free and open-source alternatives to Wget:
+
+### [Wpull](#/crawling-cli?id=wpull)
+
+[Wpull](https://github.com/ArchiveTeam/wpull) is a wget‐compatible Python crawler that supports on-disk resumption, WARC output, and PhantomJS integration.
+
+```
+uvx wpull \
+  --recursive \
+  --level=3 \
+  --no-parent \
+  --convert-links \
+  --adjust-extension \
+  --compression=auto \
+  --accept html,htm \
+  --directory-prefix=./ds \
+  https://study.iitm.ac.in/ds/Copy to clipboardErrorCopied
+```
+
+### [HTTrack](#/crawling-cli?id=httrack)
+
+[HTTrack](https://www.httrack.com/html/fcguide.html) is dedicated website‐mirroring tool with rich filtering and link‐conversion options.
+
+```
+httrack "https://study.iitm.ac.in/ds/" \
+  -O "./ds" \
+  "+*.study.iitm.ac.in/ds/*" \
+  -r3Copy to clipboardErrorCopied
+```
+
+### [Robots.txt](#/crawling-cli?id=robotstxt)
+
+`robots.txt` is a standard file found in a website’s root directory that specifies which parts of the site should not be accessed by web crawlers. It’s part of the Robots Exclusion Protocol, an ethical standard for web crawling.
+
+**Why it’s important**:
+
+* **Server load protection**: Prevents excessive traffic that could overload servers
+* **Privacy protection**: Keeps sensitive or private content from being indexed
+* **Legal compliance**: Respects website owners’ rights to control access to their content
+* **Ethical web citizenship**: Shows respect for website administrators’ wishes
+
+**How to override robots.txt restrictions**:
+
+* **wget, wget2**: Use `-e robots=off`
+* **httrack**: Use `-s0`
+* **wpull**: Use `--no-robots`
+
+**When to override robots.txt (use with discretion)**:
+
+Only bypass `robots.txt` when:
+
+* You have explicit permission from the website owner
+* You’re crawling your own website
+* The content is publicly accessible and your crawling won’t cause server issues
+* You’re conducting authorized security testing
+
+Remember that bypassing `robots.txt` without legitimate reason may:
+
+* Violate terms of service
+* Lead to IP banning
+* Result in legal consequences in some jurisdictions
+* Cause reputation damage to your organization
+
+Always use the minimum necessary crawling speed and scope, and consider contacting website administrators for permission when in doubt.
+
+[Previous
+
+Scraping with Google Sheets](#/scraping-with-google-sheets)
+
+[Next
+
+BBC Weather API with Python](#/bbc-weather-api-with-python)

markdown_files/Data_Aggregation_in_Excel.md

@@ -0,0 +1,32 @@
+---
+title: "Data Aggregation in Excel"
+original_url: "https://tds.s-anand.net/#/data-aggregation-in-excel?id=data-aggregation-in-excel"
+downloaded_at: "2025-06-08T23:26:38.121838"
+---
+
+[Data Aggregation in Excel](#/data-aggregation-in-excel?id=data-aggregation-in-excel)
+-------------------------------------------------------------------------------------
+
+[![Data aggregation in Excel](https://i.ytimg.com/vi_webp/NkpT0dDU8Y4/sddefault.webp)](https://youtu.be/NkpT0dDU8Y4)
+
+You’ll learn data aggregation and visualization techniques in Excel, covering:
+
+* **Data Cleanup**: Remove empty columns and rows with missing values.
+* **Creating Excel Tables**: Convert raw data into tables for easier manipulation and formula application.
+* **Date Manipulation**: Extract week, month, and year from date columns using Excel functions (WEEKNUM, TEXT).
+* **Color Scales**: Apply color scales to visualize clusters and trends in data over time.
+* **Pivot Tables**: Create pivot tables to aggregate data by location and date, summarizing values weekly and monthly.
+* **Sparklines**: Use sparklines to visualize trends within pivot tables, making data patterns more apparent.
+* **Data Bars**: Implement data bars for graphical illustrations of numerical columns, showing trends and waves.
+
+Here are links used in the video:
+
+* [COVID-19 data Excel file - raw data](https://docs.google.com/spreadsheets/d/14HLgSmME95q--6lcBv9pUstqHL183wTd/view)
+
+[Previous
+
+Splitting Text in Excel](#/splitting-text-in-excel)
+
+[Next
+
+Data Preparation in the Shell](#/data-preparation-in-the-shell)

markdown_files/Data_Analysis_with_DuckDB.md

@@ -0,0 +1,37 @@
+---
+title: "Data Analysis with DuckDB"
+original_url: "https://tds.s-anand.net/#/data-analysis-with-duckdb?id=data-analysis-with-duckdb"
+downloaded_at: "2025-06-08T23:26:27.065997"
+---
+
+[Data Analysis with DuckDB](#/data-analysis-with-duckdb?id=data-analysis-with-duckdb)
+-------------------------------------------------------------------------------------
+
+[![Data Analysis with DuckDB](https://i.ytimg.com/vi_webp/4U0GqYrET5s/sddefault.webp)](https://youtu.be/4U0GqYrET5s)
+
+You’ll learn how to perform data analysis using DuckDB and Pandas, covering:
+
+* **Parquet for Data Storage**: Understand why Parquet is a faster, more compact, and better-typed storage format compared to CSV, JSON, and SQLite.
+* **DuckDB Setup**: Learn how to install and set up DuckDB, along with integrating it into a Jupyter notebook environment.
+* **File Format Comparisons**: Compare file formats by speed and size, observing the performance difference between saving and loading data in CSV, JSON, SQLite, and Parquet.
+* **Faster Queries with DuckDB**: Learn how DuckDB uses parallel processing, columnar storage, and on-disk operations to outperform Pandas in speed and memory efficiency.
+* **SQL Query Execution in DuckDB**: Run SQL queries directly on Parquet files and Pandas DataFrames to compute metrics such as the number of unique flight routes delayed by certain time intervals.
+* **Memory Efficiency**: Understand how DuckDB performs analytics without loading entire datasets into memory, making it highly efficient for large-scale data analysis.
+* **Mixing DuckDB and Pandas**: Learn to interleave DuckDB and Pandas operations, leveraging the strengths of both tools to perform complex queries like correlations and aggregations.
+* **Ranking and Filtering Data**: Use SQL and Pandas to rank arrival delays by distance and extract key insights, such as the earliest flight arrival for each route.
+* **Joining Data**: Create a cost analysis by joining datasets and calculating total costs of flight delays, demonstrating DuckDB’s speed in joining and aggregating large datasets.
+
+Here are the links used in the video:
+
+* [Data analysis with DuckDB - Notebook](https://drive.google.com/file/d/1Y9XSs-LeSz-ZmnQj4OGP-Q4yDkPJrmsZ/view)
+* [Parquet file format](https://parquet.apache.org/) - a fast columnar storage format that’s becoming a de facto standard for big data
+* [DuckDB](https://duckdb.org/) - a fast in-memory database that’s very good with large-scale analysis
+* [Plotly Datasets](https://github.com/plotly/datasets/) - a collection of sample datasets for analysis. This includes the [Kaggle Flights Dataset](https://www.kaggle.com/datasets/usdot/flight-delays) that the notebook downloads as [2015\_flights.parquet](https://github.com/plotly/datasets/raw/master/2015_flights.parquet)
+
+[Previous
+
+Data Analysis with Datasette](#/data-analysis-with-datasette)
+
+[Next
+
+Data Analysis with ChatGPT](#/data-analysis-with-chatgpt)

markdown_files/Data_Analysis_with_Python.md

@@ -0,0 +1,37 @@
+---
+title: "Data Analysis with Python"
+original_url: "https://tds.s-anand.net/#/data-analysis-with-python?id=data-analysis-with-python"
+downloaded_at: "2025-06-08T23:24:24.926726"
+---
+
+[Data Analysis with Python](#/data-analysis-with-python?id=data-analysis-with-python)
+-------------------------------------------------------------------------------------
+
+[![Data Analysis with Python](https://i.ytimg.com/vi_webp/ZPfZH14FK90/sddefault.webp)](https://youtu.be/ZPfZH14FK90)
+
+You’ll learn practical data analysis techniques in Python using Pandas, covering:
+
+* **Reading Parquet Files**: Utilize Pandas to read Parquet file formats for efficient data handling.
+* **Dataframe Inspection**: Methods to preview and understand the structure of a dataset.
+* **Pivot Tables**: Creating and interpreting pivot tables to summarize data.
+* **Percentage Calculations**: Normalize pivot table values to percentages for better insights.
+* **Correlation Analysis**: Calculate and interpret correlation between variables, including significance testing.
+* **Statistical Significance**: Use statistical tests to determine the significance of observed correlations.
+* **Datetime Handling**: Extract and manipulate date and time information from datetime columns.
+* **Data Visualization**: Generate and customize heat maps to visualize data patterns effectively.
+* **Leveraging AI**: Use ChatGPT to generate and refine analytical code, enhancing productivity and accuracy.
+
+Here are the links used in the video:
+
+* [Data analysis with Python - Notebook](https://colab.research.google.com/drive/1wEUEeF_e2SSmS9uf2-3fZJQ2kEFRnxah)
+* [Card transactions dataset (Parquet)](https://drive.google.com/file/u/3/d/1XGvuFjoTwlybkw0cc9u34horMF9vMhrB/view)
+* [10 minutes to Pandas](https://pandas.pydata.org/pandas-docs/stable/user_guide/10min.html)
+* [Python Pandas tutorials](https://www.youtube.com/playlist?list=PL-osiE80TeTsWmV9i9c58mdDCSskIFdDS)
+
+[Previous
+
+Outlier Detection with Excel](#/outlier-detection-with-excel)
+
+[Next
+
+Data Analysis with SQL](#/data-analysis-with-sql)

markdown_files/Data_Analysis_with_SQL.md

@@ -0,0 +1,39 @@
+---
+title: "Data Analysis with SQL"
+original_url: "https://tds.s-anand.net/#/data-analysis-with-sql?id=data-analysis-with-sql"
+downloaded_at: "2025-06-08T23:22:33.461136"
+---
+
+[Data Analysis with SQL](#/data-analysis-with-sql?id=data-analysis-with-sql)
+----------------------------------------------------------------------------
+
+[![Data Analysis with Databases](https://i.ytimg.com/vi_webp/Xn3QkYrThbI/sddefault.webp)](https://youtu.be/Xn3QkYrThbI)
+
+You’ll learn how to perform data analysis using SQL (via Python), covering:
+
+* **Database Connection**: How to connect to a MySQL database using SQLAlchemy and Pandas.
+* **SQL Queries**: Execute SQL queries directly from a Python environment to retrieve and analyze data.
+* **Counting Rows**: Use SQL to count the number of rows in a table.
+* **User Activity Analysis**: Query and identify top users by post count.
+* **Post Concentration**: Determine if a small percentage of users contribute the majority of posts using SQL aggregation.
+* **Correlation Calculation**: Calculate the Pearson correlation coefficient between user attributes such as age and reputation.
+* **Regression Analysis**: Compute the regression slope to understand the relationship between views and reputation.
+* **Handling Large Data**: Perform calculations on large datasets by fetching aggregated values from the database rather than entire datasets.
+* **Statistical Analysis in SQL**: Use SQL as a tool for statistical analysis, demonstrating its power beyond simple data retrieval.
+* **Leveraging AI**: Use ChatGPT to generate SQL queries and Python code, enhancing productivity and accuracy.
+
+Here are the links used in the video:
+
+* [Data analysis with databases - Notebook](https://colab.research.google.com/drive/1j_5AsWdf0SwVHVgfbEAcg7vYguKUN41o)
+* [SQLZoo](https://www.sqlzoo.net/wiki/SQL_Tutorial) has simple interactive tutorials to learn SQL
+* [Stats database](https://relational-data.org/dataset/Stats) that has an anonymized dump of [stats.stackexchange.com](https://stats.stackexchange.com/)
+* [Pandas `read_sql`](https://pandas.pydata.org/docs/reference/api/pandas.read_sql.html)
+* [SQLAlchemy docs](https://docs.sqlalchemy.org/)
+
+[Previous
+
+Data Analysis with Python](#/data-analysis-with-python)
+
+[Next
+
+Data Analysis with Datasette](#/data-analysis-with-datasette)

markdown_files/Data_Cleansing_in_Excel.md

@@ -0,0 +1,30 @@
+---
+title: "Data Cleansing in Excel"
+original_url: "https://tds.s-anand.net/#/data-cleansing-in-excel?id=data-cleansing-in-excel"
+downloaded_at: "2025-06-08T23:27:03.007571"
+---
+
+[Data Cleansing in Excel](#/data-cleansing-in-excel?id=data-cleansing-in-excel)
+-------------------------------------------------------------------------------
+
+[![Clean up data in Excel](https://i.ytimg.com/vi_webp/7du7xkqeu4s/sddefault.webp)](https://youtu.be/7du7xkqeu4s)
+
+You’ll learn basic but essential data cleaning techniques in Excel, covering:
+
+* **Find and Replace**: Use Ctrl+H to replace or remove specific terms (e.g., removing “[more]” from country names).
+* **Changing Data Formats**: Convert columns from general to numerical format.
+* **Removing Extra Spaces**: Use the TRIM function to clean up unnecessary spaces in text.
+* **Identifying and Removing Blank Cells**: Highlight and delete entire rows with blank cells using the “Go To Special” function.
+* **Removing Duplicates**: Use the “Remove Duplicates” feature to eliminate duplicate entries, demonstrated with country names.
+
+Here are links used in the video:
+
+* [List of Largest Cities Excel file](https://docs.google.com/spreadsheets/d/1jl8tHGoxmIba4J78aJVfT9jtZv7lfCbV/view)
+
+[Previous
+
+5. Data Preparation](#/data-preparation)
+
+[Next
+
+Data Transformation in Excel](#/data-transformation-in-excel)

markdown_files/Data_Preparation_in_the_Editor.md

@@ -0,0 +1,30 @@
+---
+title: "Data Preparation in the Editor"
+original_url: "https://tds.s-anand.net/#/data-preparation-in-the-editor?id=data-preparation-in-the-editor"
+downloaded_at: "2025-06-08T23:22:43.469063"
+---
+
+[Data Preparation in the Editor](#/data-preparation-in-the-editor?id=data-preparation-in-the-editor)
+----------------------------------------------------------------------------------------------------
+
+[![Data preparation in the editor](https://i.ytimg.com/vi_webp/99lYu43L9uM/sddefault.webp)](https://youtu.be/99lYu43L9uM)
+
+You’ll learn how to use a text editor [Visual Studio Code](https://code.visualstudio.com/) to process and clean data, covering:
+
+* **Format** JSON files
+* **Find all** and multiple cursors to extract specific fields
+* **Sort** lines
+* **Delete duplicate** lines
+* **Replace** text with multiple cursors
+
+Here are the links used in the video:
+
+* [City-wise product sales JSON](https://drive.google.com/file/d/1VEnKChf4i04iKsQfw0MwoJlfkOBGQ65B/view?usp=drive_link)
+
+[Previous
+
+Data Preparation in the Shell](#/data-preparation-in-the-shell)
+
+[Next
+
+Cleaning Data with OpenRefine](#/cleaning-data-with-openrefine)

markdown_files/Data_Preparation_in_the_Shell.md

@@ -0,0 +1,36 @@
+---
+title: "Data Preparation in the Shell"
+original_url: "https://tds.s-anand.net/#/data-preparation-in-the-shell?id=data-preparation-in-the-shell"
+downloaded_at: "2025-06-08T23:26:41.381829"
+---
+
+[Data Preparation in the Shell](#/data-preparation-in-the-shell?id=data-preparation-in-the-shell)
+-------------------------------------------------------------------------------------------------
+
+[![Data preparation in the shell](https://i.ytimg.com/vi_webp/XEdy4WK70vU/sddefault.webp)](https://youtu.be/XEdy4WK70vU)
+
+You’ll learn how to use UNIX tools to process and clean data, covering:
+
+* `curl` (or `wget`) to fetch data from websites.
+* `gzip` (or `xz`) to compress and decompress files.
+* `wc` to count lines, words, and characters in text.
+* `head` and `tail` to get the start and end of files.
+* `cut` to extract specific columns from text.
+* `uniq` to de-duplicate lines.
+* `sort` to sort lines.
+* `grep` to filter lines containing specific text.
+* `sed` to search and replace text.
+* `awk` for more complex text processing.
+
+Here are the links used in the video:
+
+* [Data preparation in the shell - Notebook](https://colab.research.google.com/drive/1KSFkQDK0v__XWaAaHKeQuIAwYV0dkTe8)
+* [Data Science at the Command Line](https://jeroenjanssens.com/dsatcl/)
+
+[Previous
+
+Data Aggregation in Excel](#/data-aggregation-in-excel)
+
+[Next
+
+Data Preparation in the Editor](#/data-preparation-in-the-editor)

markdown_files/Data_Storytelling.md

@@ -0,0 +1,18 @@
+---
+title: "Data Storytelling"
+original_url: "https://tds.s-anand.net/#/data-storytelling?id=data-storytelling"
+downloaded_at: "2025-06-08T23:21:12.671499"
+---
+
+[Data Storytelling](#/data-storytelling?id=data-storytelling)
+=============================================================
+
+[![Narrate a story](https://i.ytimg.com/vi_webp/aF93i6zVVQg/sddefault.webp)](https://youtu.be/aF93i6zVVQg)
+
+[Previous
+
+RAWgraphs](#/rawgraphs)
+
+[Next
+
+Narratives with LLMs](#/narratives-with-llms)

markdown_files/Data_Transformation_in_Excel.md

@@ -0,0 +1,30 @@
+---
+title: "Data Transformation in Excel"
+original_url: "https://tds.s-anand.net/#/data-transformation-in-excel?id=data-transformation-in-excel"
+downloaded_at: "2025-06-08T23:26:40.285938"
+---
+
+[Data Transformation in Excel](#/data-transformation-in-excel?id=data-transformation-in-excel)
+----------------------------------------------------------------------------------------------
+
+[![Data transformation in Excel](https://i.ytimg.com/vi_webp/gR2IY5Naja0/sddefault.webp)](https://youtu.be/gR2IY5Naja0)
+
+You’ll learn data transformation techniques in Excel, covering:
+
+* **Calculating Ratios**: Compute metro area to city area and metro population to city population ratios.
+* **Using Pivot Tables**: Create pivot tables to aggregate data and identify outliers.
+* **Filtering Data**: Apply filters in pivot tables to analyze specific subsets of data.
+* **Counting Data Occurrences**: Use pivot tables to count the frequency of specific entries.
+* **Creating Charts**: Generate charts from pivot table data to visualize distributions and outliers.
+
+Here are links used in the video:
+
+* [List of Largest Cities Excel file](https://docs.google.com/spreadsheets/d/1jl8tHGoxmIba4J78aJVfT9jtZv7lfCbV/view)
+
+[Previous
+
+Data Cleansing in Excel](#/data-cleansing-in-excel)
+
+[Next
+
+Splitting Text in Excel](#/splitting-text-in-excel)

markdown_files/Data_Transformation_with_dbt.md

@@ -0,0 +1,64 @@
+---
+title: "Data Transformation with dbt"
+original_url: "https://tds.s-anand.net/#/dbt?id=data-transformation-with-dbt"
+downloaded_at: "2025-06-08T23:26:28.166999"
+---
+
+[Data Transformation with dbt](#/dbt?id=data-transformation-with-dbt)
+---------------------------------------------------------------------
+
+[![Data Transformation with dbt](https://i.ytimg.com/vi_webp/5rNquRnNb4E/sddefault.webp)](https://youtu.be/5rNquRnNb4E)
+
+You’ll learn how to transform data using dbt (data build tool), covering:
+
+* **dbt Fundamentals**: Understand what dbt is and how it brings software engineering practices to data transformation
+* **Project Setup**: Learn how to initialize a dbt project, configure your warehouse connection, and structure your models
+* **Models and Materialization**: Create your first dbt models and understand different materialization strategies (view, table, incremental)
+* **Testing and Documentation**: Implement data quality tests and auto-generate documentation for your data models
+* **Jinja Templating**: Use Jinja for dynamic SQL generation, making your transformations more maintainable and reusable
+* **References and Dependencies**: Learn how to reference other models and manage model dependencies
+* **Sources and Seeds**: Configure source data connections and manage static reference data
+* **Macros and Packages**: Create reusable macros and leverage community packages to extend functionality
+* **Incremental Models**: Optimize performance by only processing new or changed data
+* **Deployment and Orchestration**: Set up dbt Cloud or integrate with Airflow for production deployment
+
+Here’s a minimal dbt model example, `models/staging/stg_customers.sql`:
+
+```
+with source as (
+    select * from {{ source('raw', 'customers') }}
+),
+
+renamed as (
+    select
+        id as customer_id,
+        first_name,
+        last_name,
+        email,
+        created_at
+    from source
+)
+
+select * from renamedCopy to clipboardErrorCopied
+```
+
+Tools and Resources:
+
+* [dbt Core](https://github.com/dbt-labs/dbt-core) - The open-source transformation tool
+* [dbt Cloud](https://www.getdbt.com/product/dbt-cloud) - Hosted platform for running dbt
+* [dbt Packages](https://hub.getdbt.com/) - Reusable modules from the community
+* [dbt Documentation](https://docs.getdbt.com/) - Comprehensive guides and references
+* [Jaffle Shop](https://github.com/dbt-labs/jaffle_shop) - Example dbt project for learning
+* [dbt Slack Community](https://www.getdbt.com/community/) - Active community for support and discussions
+
+Watch this dbt Fundamentals Course (90 min):
+
+[![dbt Fundamentals Course](https://i.ytimg.com/vi_webp/5rNquRnNb4E/sddefault.webp)](https://youtu.be/5rNquRnNb4E)
+
+[Previous
+
+Parsing JSON](#/parsing-json)
+
+[Next
+
+Transforming Images](#/transforming-images)

markdown_files/Data_Visualization_with_Seaborn.md

@@ -0,0 +1,20 @@
+---
+title: "Data Visualization with Seaborn"
+original_url: "https://tds.s-anand.net/#/data-visualization-with-seaborn?id=data-visualization-with-seaborn"
+downloaded_at: "2025-06-08T23:24:55.808928"
+---
+
+[Data Visualization with Seaborn](#/data-visualization-with-seaborn?id=data-visualization-with-seaborn)
+-------------------------------------------------------------------------------------------------------
+
+[Seaborn](https://seaborn.pydata.org/) is a data visualization library for Python. It’s based on Matplotlib but a bit easier to use, and a bit prettier.
+
+[![Seaborn Tutorial : Seaborn Full Course](https://i.ytimg.com/vi_webp/6GUZXDef2U0/sddefault.webp)](https://youtu.be/6GUZXDef2U0)
+
+[Previous
+
+Visualizing Charts with Excel](#/visualizing-charts-with-excel)
+
+[Next
+
+Data Visualization with ChatGPT](#/data-visualization-with-chatgpt)

markdown_files/Database__SQLite.md

@@ -0,0 +1,148 @@
+---
+title: "Database: SQLite"
+original_url: "https://tds.s-anand.net/#/sqlite?id=database-sqlite"
+downloaded_at: "2025-06-08T23:26:00.500923"
+---
+
+[Database: SQLite](#/sqlite?id=database-sqlite)
+-----------------------------------------------
+
+Relational databases are used to store data in a structured way. You’ll often access databases created by others for analysis.
+
+PostgreSQL, MySQL, MS SQL, Oracle, etc. are popular databases. But the most installed database is [SQLite](https://www.sqlite.org/index.html). It’s embedded into many devices and apps (e.g. your phone, browser, etc.). It’s lightweight but very scalable and powerful.
+
+Watch these introductory videos to understand SQLite and how it’s used in Python (34 min):
+
+[![SQLite Introduction - Beginners Guide to SQL and Databases (22 min)](https://i.ytimg.com/vi_webp/8Xyn8R9eKB8/sddefault.webp)](https://youtu.be/8Xyn8R9eKB8)
+
+[![SQLite Backend for Beginners - Create Quick Databases with Python and SQL (13 min)](https://i.ytimg.com/vi_webp/Ohj-CqALrwk/sddefault.webp)](https://youtu.be/Ohj-CqALrwk)
+
+There are many non-relational databases (NoSQL) like [ElasticSearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html), [MongoDB](https://www.mongodb.com/docs/manual/), [Redis](https://redis.io/docs/latest/), etc. that you should know about and we may cover later.
+
+Core Concepts:
+
+```
+-- Create a table
+CREATE TABLE users (
+    id INTEGER PRIMARY KEY,
+    name TEXT NOT NULL,
+    email TEXT UNIQUE,
+    created_at DATETIME DEFAULT CURRENT_TIMESTAMP
+);
+
+-- Insert data
+INSERT INTO users (name, email) VALUES
+    ('Alice', 'alice@example.com'),
+    ('Bob', 'bob@example.com');
+
+-- Query data
+SELECT name, COUNT(*) as count
+FROM users
+GROUP BY name
+HAVING count > 1;
+
+-- Join tables
+SELECT u.name, o.product
+FROM users u
+LEFT JOIN orders o ON u.id = o.user_id
+WHERE o.status = 'pending';Copy to clipboardErrorCopied
+```
+
+Python Integration:
+
+```
+import sqlite3
+from pathlib import Path
+import pandas as pd
+
+async def query_database(db_path: Path, query: str) -> pd.DataFrame:
+    """Execute SQL query and return results as DataFrame.
+
+    Args:
+        db_path: Path to SQLite database
+        query: SQL query to execute
+
+    Returns:
+        DataFrame with query results
+    """
+    try:
+        conn = sqlite3.connect(db_path)
+        return pd.read_sql_query(query, conn)
+    finally:
+        conn.close()
+
+# Example usage
+db = Path('data.db')
+df = await query_database(db, '''
+    SELECT date, COUNT(*) as count
+    FROM events
+    GROUP BY date
+''')Copy to clipboardErrorCopied
+```
+
+Common Operations:
+
+1. **Database Management**
+
+   ```
+   -- Backup database
+   .backup 'backup.db'
+
+   -- Import CSV
+   .mode csv
+   .import data.csv table_name
+
+   -- Export results
+   .headers on
+   .mode csv
+   .output results.csv
+   SELECT * FROM table;Copy to clipboardErrorCopied
+   ```
+2. **Performance Optimization**
+
+   ```
+   -- Create index
+   CREATE INDEX idx_user_email ON users(email);
+
+   -- Analyze query
+   EXPLAIN QUERY PLAN
+   SELECT * FROM users WHERE email LIKE '%@example.com';
+
+   -- Show indexes
+   SELECT * FROM sqlite_master WHERE type='index';Copy to clipboardErrorCopied
+   ```
+3. **Data Analysis**
+
+   ```
+   -- Time series aggregation
+   SELECT
+       date(timestamp),
+       COUNT(*) as events,
+       AVG(duration) as avg_duration
+   FROM events
+   GROUP BY date(timestamp);
+
+   -- Window functions
+   SELECT *,
+       AVG(amount) OVER (
+           PARTITION BY user_id
+           ORDER BY date
+           ROWS BETWEEN 3 PRECEDING AND CURRENT ROW
+       ) as moving_avg
+   FROM transactions;Copy to clipboardErrorCopied
+   ```
+
+Tools to work with SQLite:
+
+* [SQLiteStudio](https://sqlitestudio.pl/): Lightweight GUI
+* [DBeaver](https://dbeaver.io/): Full-featured GUI
+* [sqlite-utils](https://sqlite-utils.datasette.io/): CLI tool
+* [Datasette](https://datasette.io/): Web interface
+
+[Previous
+
+Spreadsheet: Excel, Google Sheets](#/spreadsheets)
+
+[Next
+
+Version Control: Git, GitHub](#/git)

markdown_files/DevContainers__GitHub_Codespaces.md

@@ -0,0 +1,57 @@
+---
+title: "DevContainers: GitHub Codespaces"
+original_url: "https://tds.s-anand.net/#/github-codespaces?id=features-to-explore"
+downloaded_at: "2025-06-08T23:27:28.688679"
+---
+
+[IDE: GitHub Codespaces](#/github-codespaces?id=ide-github-codespaces)
+----------------------------------------------------------------------
+
+[GitHub Codespaces](https://github.com/features/codespaces) is a cloud-hosted development environment built right into GitHub that gets you coding faster with pre-configured containers, adjustable compute power, and seamless integration with workflows like Actions and Copilot.
+
+**Why Codespaces helps**
+
+* **Reproducible onboarding**: Say goodbye to “works on my machine” woes—everyone uses the same setup for assignments or demos.
+* **Anywhere access**: Jump back into your project from a laptop, tablet, or phone without having to reinstall anything.
+* **Rapid experimentation & debugging**: Spin up short-lived environments on any branch, commit, or PR to isolate bugs or test features, or keep longer-lived codespaces for big projects.
+
+[![Introduction to GitHub Codespaces (5 min)](https://i.ytimg.com/vi_webp/-tQ2nxjqP6o/sddefault.webp)](https://www.youtube.com/watch?v=-tQ2nxjqP6o)
+
+### [Quick Setup](#/github-codespaces?id=quick-setup)
+
+1. [**From the GitHub UI**](https://github.com/codespaces)
+
+   * Go to your repo and click **Code → Codespaces → New codespace**.
+   * Pick the branch and machine specs (2–32 cores, 8–64 GB RAM), then click **Create codespace**.
+2. [**In Visual Studio Code**](https://code.visualstudio.com/docs/remote/codespaces)
+
+   * Press `Ctrl+Shift+P` (or `Cmd+Shift+P` on Mac), choose **Codespaces: Create New Codespace**, and follow the prompts.
+3. [**Via GitHub CLI**](https://docs.github.com/en/codespaces/developing-in-a-codespace/using-github-codespaces-with-github-cli)
+
+   ```
+   gh auth login
+   gh codespace create --repo OWNER/REPO
+   gh codespace list    # List all codespaces
+   gh codespace code    # opens in your local VS Code
+   gh codespace ssh     # SSH into the codepsaceCopy to clipboardErrorCopied
+   ```
+
+### [Features To Explore](#/github-codespaces?id=features-to-explore)
+
+* **Dev Containers**: Set up your environment the same way every time using a `devcontainer.json` or your own Dockerfile. [Introduction to dev containers](https://docs.github.com/en/codespaces/setting-up-your-project-for-codespaces/adding-a-dev-container-configuration/introduction-to-dev-containers)
+* **Prebuilds**: Build bigger or more complex repos in advance so codespaces start up in a flash. [About prebuilds](https://docs.github.com/en/codespaces/prebuilding-your-codespaces/about-github-codespaces-prebuilds)
+* **Port Forwarding**: Let Codespaces spot and forward the ports your web apps use automatically. [Forward ports in Codespaces](https://docs.github.com/en/codespaces/developing-in-a-codespace/forwarding-ports-in-your-codespace)
+* **Secrets & Variables**: Keep your environment variables safe in the Codespaces settings for your repo. [Manage Codespaces secrets](https://docs.github.com/en/enterprise-cloud@latest/codespaces/managing-codespaces-for-your-organization/managing-development-environment-secrets-for-your-repository-or-organization)
+* **Dotfiles Integration**: Bring in your dotfiles repo to customize shell settings, aliases, and tools in every codespace. [Personalizing your codespaces](https://docs.github.com/en/codespaces/setting-your-user-preferences/personalizing-github-codespaces-for-your-account)
+* **Machine Types & Cost Control**: Pick from VMs with 2 to 32 cores and track your usage in the billing dashboard. [Managing Codespaces costs](https://docs.github.com/en/billing/managing-billing-for-github-codespaces/about-billing-for-github-codespaces)
+* **VS Code & CLI Integration**: Flip between browser VS Code and your desktop editor, and script everything with the CLI. [VS Code Remote: Codespaces](https://code.visualstudio.com/docs/remote/codespaces)
+* **GitHub Actions**: Power up prebuilds and your CI/CD right inside codespaces using Actions workflows. [Prebuilding your codespaces](https://docs.github.com/en/codespaces/prebuilding-your-codespaces)
+* **Copilot in Codespaces**: Let Copilot help you write code with in-editor AI suggestions. [Copilot in Codespaces](https://docs.github.com/en/codespaces/reference/using-github-copilot-in-github-codespaces)
+
+[Previous
+
+Containers: Docker, Podman](#/docker)
+
+[Next
+
+Tunneling: ngrok](#/ngrok)

markdown_files/Editor__VS_Code.md

@@ -0,0 +1,31 @@
+---
+title: "Editor: VS Code"
+original_url: "https://tds.s-anand.net/#/vscode?id=editor-vs-code"
+downloaded_at: "2025-06-08T23:26:42.473263"
+---
+
+[Editor: VS Code](#/vscode?id=editor-vs-code)
+---------------------------------------------
+
+Your editor is the most important tool in your arsenal. That’s where you’ll spend most of your time. Make sure you’re comfortable with it.
+
+[**Visual Studio Code**](https://code.visualstudio.com/) is, *by far*, the most popular code editor today. According to the [2024 StackOverflow Survey](https://survey.stackoverflow.co/2024/technology/#1-integrated-development-environment) almost 75% of developers use it. We recommend you learn it well. Even if you use another editor, you’ll be working with others who use it, and it’s a good idea to have some exposure.
+
+Watch these introductory videos (35 min) from the [Visual Studio Docs](https://code.visualstudio.com/docs) to get started:
+
+* [Getting Started](https://code.visualstudio.com/docs/introvideos/basics): Set up and learn the basics of Visual Studio Code. (7 min)
+* [Code Editing](https://code.visualstudio.com/docs/introvideos/codeediting): Learn how to edit and run code in VS Code. (3 min)
+* [Productivity Tips](https://code.visualstudio.com/docs/introvideos/productivity): Become a VS Code power user with these productivity tips. (4 min)
+* [Personalize](https://code.visualstudio.com/docs/introvideos/configure): Personalize VS Code to make it yours with themes. (2 min)
+* [Extensions](https://code.visualstudio.com/docs/introvideos/extend): Add features, themes, and more to VS Code with extensions! (4 min)
+* [Debugging](https://code.visualstudio.com/docs/introvideos/debugging): Get started with debugging in VS Code. (6 min)
+* [Version Control](https://code.visualstudio.com/docs/introvideos/versioncontrol): Learn how to use Git version control in VS Code. (3 min)
+* [Customize](https://code.visualstudio.com/docs/introvideos/customize): Learn how to customize your settings and keyboard shortcuts in VS Code. (6 min)
+
+[Previous
+
+1. Development Tools](#/development-tools)
+
+[Next
+
+AI Code Editors: GitHub Copilot](#/github-copilot)

markdown_files/Embeddings.md

@@ -0,0 +1,106 @@
+---
+title: "Embeddings"
+original_url: "https://tds.s-anand.net/#/embeddings?id=openai-embeddings"
+downloaded_at: "2025-06-08T23:27:24.487774"
+---
+
+[Embeddings: OpenAI and Local Models](#/embeddings?id=embeddings-openai-and-local-models)
+-----------------------------------------------------------------------------------------
+
+Embedding models convert text into a list of numbers. These are like a map of text in numerical form. Each number represents a feature, and similar texts will have numbers close to each other. So, if the numbers are similar, the text they represent mean something similar.
+
+This is useful because text similarity is important in many common problems:
+
+1. **Search**. Find similar documents to a query.
+2. **Classification**. Classify text into categories.
+3. **Clustering**. Group similar items into clusters.
+4. **Anomaly Detection**. Find an unusual piece of text.
+
+You can run embedding models locally or using an API. Local models are better for privacy and cost. APIs are better for scale and quality.
+
+| Feature | Local Models | API |
+| --- | --- | --- |
+| **Privacy** | High | Dependent on provider |
+| **Cost** | High setup, low after that | Pay-as-you-go |
+| **Scale** | Limited by local resources | Easily scales with demand |
+| **Quality** | Varies by model | Typically high |
+
+The [Massive Text Embedding Benchmark (MTEB)](https://huggingface.co/spaces/mteb/leaderboard) provides comprehensive comparisons of embedding models. These models are compared on several parameters, but here are some key ones to look at:
+
+1. **Rank**. Higher ranked models have higher quality.
+2. **Memory Usage**. Lower is better (for similar ranks). It costs less and is faster to run.
+3. **Embedding Dimensions**. Lower is better. This is the number of numbers in the array. Smaller dimensions are cheaper to store.
+4. **Max Tokens**. Higher is better. This is the number of input tokens (words) the model can take in a *single* input.
+5. Look for higher scores in the columns for Classification, Clustering, Summarization, etc. based on your needs.
+
+### [Local Embeddings](#/embeddings?id=local-embeddings)
+
+[![Guide to Local Embeddings with Sentence Transformers](https://i.ytimg.com/vi/OATCgQtNX2o/sddefault.jpg)](https://youtu.be/OATCgQtNX2o)
+
+Here’s a minimal example using a local embedding model:
+
+```
+# /// script
+# requires-python = "==3.12"
+# dependencies = [
+#   "sentence-transformers",
+#   "httpx",
+#   "numpy",
+# ]
+# ///
+
+from sentence_transformers import SentenceTransformer
+import numpy as np
+
+model = SentenceTransformer('BAAI/bge-base-en-v1.5')  # A small, high quality model
+
+async def embed(text: str) -> list[float]:
+    """Get embedding vector for text using local model."""
+    return model.encode(text).tolist()
+
+async def get_similarity(text1: str, text2: str) -> float:
+    """Calculate cosine similarity between two texts."""
+    emb1 = np.array(await embed(text1))
+    emb2 = np.array(await embed(text2))
+    return float(np.dot(emb1, emb2) / (np.linalg.norm(emb1) * np.linalg.norm(emb2)))
+
+async def main():
+    print(await get_similarity("Apple", "Orange"))
+    print(await get_similarity("Apple", "Lightning"))
+
+
+if __name__ == "__main__":
+    import asyncio
+    asyncio.run(main())Copy to clipboardErrorCopied
+```
+
+Note the `get_similarity` function. It uses a [Cosine Similarity](https://en.wikipedia.org/wiki/Cosine_similarity) to calculate the similarity between two embeddings.
+
+### [OpenAI Embeddings](#/embeddings?id=openai-embeddings)
+
+For comparison, here’s how to use OpenAI’s API with direct HTTP calls. Replace the `embed` function in the earlier script:
+
+```
+import os
+import httpx
+
+async def embed(text: str) -> list[float]:
+    """Get embedding vector for text using OpenAI's API."""
+    async with httpx.AsyncClient() as client:
+        response = await client.post(
+            "https://api.openai.com/v1/embeddings",
+            headers={"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}"},
+            json={"model": "text-embedding-3-small", "input": text}
+        )
+        return response.json()["data"][0]["embedding"]Copy to clipboardErrorCopied
+```
+
+**NOTE**: You need to set the [`OPENAI_API_KEY`](https://platform.openai.com/api-keys) environment variable for this to work.
+
+[Previous
+
+Vision Models](#/vision-models)
+
+[Next
+
+Multimodal Embeddings](#/multimodal-embeddings)

markdown_files/Extracting_Audio_and_Transcripts.md

@@ -0,0 +1,298 @@
+---
+title: "Extracting Audio and Transcripts"
+original_url: "https://tds.s-anand.net/#/extracting-audio-and-transcripts?id=media-tools-yt-dlp"
+downloaded_at: "2025-06-08T23:25:44.497461"
+---
+
+[Extracting Audio and Transcripts](#/extracting-audio-and-transcripts?id=extracting-audio-and-transcripts)
+----------------------------------------------------------------------------------------------------------
+
+[Media Processing: FFmpeg](#/extracting-audio-and-transcripts?id=media-processing-ffmpeg)
+-----------------------------------------------------------------------------------------
+
+[FFmpeg](https://ffmpeg.org/) is the standard command-line tool for processing video and audio files. It’s essential for data scientists working with media files for:
+
+* Extracting audio/video for machine learning
+* Converting formats for web deployment
+* Creating visualizations and presentations
+* Processing large media datasets
+
+Basic Operations:
+
+```
+# Basic conversion
+ffmpeg -i input.mp4 output.avi
+
+# Extract audio
+ffmpeg -i input.mp4 -vn output.mp3
+
+# Convert format without re-encoding
+ffmpeg -i input.mkv -c copy output.mp4
+
+# High quality encoding (crf: 0-51, lower is better)
+ffmpeg -i input.mp4 -preset slower -crf 18 output.mp4Copy to clipboardErrorCopied
+```
+
+Common Data Science Tasks:
+
+```
+# Extract frames for computer vision
+ffmpeg -i input.mp4 -vf "fps=1" frames_%04d.png    # 1 frame per second
+ffmpeg -i input.mp4 -vf "select='eq(n,0)'" -vframes 1 first_frame.jpg
+
+# Create video from image sequence
+ffmpeg -r 1/5 -i img%03d.png -c:v libx264 -vf fps=25 output.mp4
+
+# Extract audio for speech recognition
+ffmpeg -i input.mp4 -ar 16000 -ac 1 audio.wav      # 16kHz mono
+
+# Trim video/audio for training data
+ffmpeg -ss 00:01:00 -i input.mp4 -t 00:00:30 -c copy clip.mp4Copy to clipboardErrorCopied
+```
+
+Processing Multiple Files:
+
+```
+# Concatenate videos (first create files.txt with list of files)
+echo "file 'input1.mp4'
+file 'input2.mp4'" > files.txt
+ffmpeg -f concat -i files.txt -c copy output.mp4
+
+# Batch process with shell loop
+for f in *.mp4; do
+    ffmpeg -i "$f" -vn "audio/${f%.mp4}.wav"
+doneCopy to clipboardErrorCopied
+```
+
+Data Analysis Features:
+
+```
+# Get media file information
+ffprobe -v quiet -print_format json -show_format -show_streams input.mp4
+
+# Display frame metadata
+ffprobe -v quiet -print_format json -show_frames input.mp4
+
+# Generate video thumbnails
+ffmpeg -i input.mp4 -vf "thumbnail" -frames:v 1 thumb.jpgCopy to clipboardErrorCopied
+```
+
+Watch this introduction to FFmpeg (12 min):
+
+[![FFmpeg in 12 Minutes](https://i.ytimg.com/vi_webp/MPV7JXTWPWI/sddefault.webp)](https://youtu.be/MPV7JXTWPWI)
+
+Tools:
+
+* [ffmpeg.lav.io](https://ffmpeg.lav.io/): Interactive command builder
+* [FFmpeg Explorer](https://ffmpeg.guide/): Visual FFmpeg command generator
+* [FFmpeg Buddy](https://evanhahn.github.io/ffmpeg-buddy/): Simple command generator
+
+Tips:
+
+1. Use `-c copy` when possible to avoid re-encoding
+2. Monitor progress with `-progress pipe:1`
+3. Use `-hide_banner` to reduce output verbosity
+4. Test commands with small clips first
+5. Use hardware acceleration when available (-hwaccel auto)
+
+Error Handling:
+
+```
+# Validate file before processing
+ffprobe input.mp4 2>&1 | grep "Invalid"
+
+# Continue on errors in batch processing
+ffmpeg -i input.mp4 output.mp4 -xerror
+
+# Get detailed error information
+ffmpeg -v error -i input.mp4 2>&1 | grep -A2 "Error"Copy to clipboardErrorCopied
+```
+
+
+
+[Media tools: yt-dlp](#/extracting-audio-and-transcripts?id=media-tools-yt-dlp)
+-------------------------------------------------------------------------------
+
+[yt-dlp](https://github.com/yt-dlp/yt-dlp) is a feature-rich command-line tool for downloading audio/video from thousands of sites. It’s particularly useful for extracting audio and transcripts from videos.
+
+Install using your package manager:
+
+```
+# macOS
+brew install yt-dlp
+
+# Linux
+curl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o ~/.local/bin/yt-dlp
+chmod a+rx ~/.local/bin/yt-dlp
+
+# Windows
+winget install yt-dlpCopy to clipboardErrorCopied
+```
+
+Common operations for extracting audio and transcripts:
+
+```
+# Download audio only at lowest quality suitable for speech
+yt-dlp -f "ba[abr<50]/worstaudio" \
+       --extract-audio \
+       --audio-format mp3 \
+       --audio-quality 32k \
+       "https://www.youtube.com/watch?v=VIDEO_ID"
+
+# Download auto-generated subtitles
+yt-dlp --write-auto-sub \
+       --skip-download \
+       --sub-format "srt" \
+       "https://www.youtube.com/watch?v=VIDEO_ID"
+
+# Download both audio and subtitles with custom output template
+yt-dlp -f "ba[abr<50]/worstaudio" \
+       --extract-audio \
+       --audio-format mp3 \
+       --audio-quality 32k \
+       --write-auto-sub \
+       --sub-format "srt" \
+       -o "%(title)s.%(ext)s" \
+       "https://www.youtube.com/watch?v=VIDEO_ID"
+
+# Download entire playlist's audio
+yt-dlp -f "ba[abr<50]/worstaudio" \
+       --extract-audio \
+       --audio-format mp3 \
+       --audio-quality 32k \
+       -o "%(playlist_index)s-%(title)s.%(ext)s" \
+       "https://www.youtube.com/playlist?list=PLAYLIST_ID"Copy to clipboardErrorCopied
+```
+
+For Python integration:
+
+```
+# /// script
+# requires-python = ">=3.9"
+# dependencies = ["yt-dlp"]
+# ///
+
+import yt_dlp
+
+def download_audio(url: str) -> None:
+    """Download audio at speech-optimized quality."""
+    ydl_opts = {
+        'format': 'ba[abr<50]/worstaudio',
+        'postprocessors': [{
+            'key': 'FFmpegExtractAudio',
+            'preferredcodec': 'mp3',
+            'preferredquality': '32'
+        }]
+    }
+
+    with yt_dlp.YoutubeDL(ydl_opts) as ydl:
+        ydl.download([url])
+
+# Example usage
+download_audio('https://www.youtube.com/watch?v=VIDEO_ID')Copy to clipboardErrorCopied
+```
+
+Tools:
+
+* [ffmpeg](https://ffmpeg.org/): Required for audio extraction and conversion
+* [whisper](https://github.com/openai/whisper): Can be used with yt-dlp for speech-to-text
+* [gallery-dl](https://github.com/mikf/gallery-dl): Alternative for image-focused sites
+
+Note: Always respect copyright and terms of service when downloading content.
+
+[Whisper transcription](#/extracting-audio-and-transcripts?id=whisper-transcription)
+------------------------------------------------------------------------------------
+
+[Faster Whisper](https://github.com/SYSTRAN/faster-whisper) is a highly optimized implementation of OpenAI’s [Whisper model](https://github.com/openai/whisper), offering up to 4x faster transcription while using less memory.
+
+You can install it via:
+
+* `pip install faster-whisper`
+* [Download Windows Standalone](https://github.com/Purfview/whisper-standalone-win/releases)
+
+Here’s a basic usage example:
+
+```
+faster-whisper-xxl "video.mp4" --model medium --language enCopy to clipboardErrorCopied
+```
+
+Here’s my recommendation for transcribing videos. This saves the output in JSON as well as SRT format in the source directory.
+
+```
+faster-whisper-xxl --print_progress --output_dir source --batch_recursive \
+                   --check_files --standard --output_format json srt \
+                   --model medium --language en $FILECopy to clipboardErrorCopied
+```
+
+* `--model`: The OpenAI Whisper model to use. You can choose from:
+  + `tiny`: Fastest but least accurate
+  + `base`: Good for simple audio
+  + `small`: Balanced speed/accuracy
+  + `medium`: Recommended default
+  + `large-v3`: Most accurate but slowest
+* `--output_format`: The output format to use. You can pick multiple formats from:
+  + `json`: Has the most detailed information including timing, text, quality, etc.
+  + `srt`: A popular subtitle format. You can use this in YouTube, for example.
+  + `vtt`: A modern subtitle format.
+  + `txt`: Just the text transcript
+* `--output_dir`: The directory to save the output files. `source` indicates the source directory, i.e. where the input `$FILE` is
+* `--language`: The language of the input file. If you don’t specify it, it analyzes the first 30 seconds to auto-detect. You can speed it up by specifying it.
+
+Run `faster-whisper-xxl --help` for more options.
+
+[Gemini transcription](#/extracting-audio-and-transcripts?id=gemini-transcription)
+----------------------------------------------------------------------------------
+
+The [Gemini](https://gemini.google.com/) models from Google are notable in two ways:
+
+1. They have a *huge* input context window. Gemini 2.0 Flash can accept 1M tokens, for example.
+2. They can handle audio input.
+
+This allows us to use Gemini to transcribe audio files.
+
+LLMs are not good at transcribing audio *faithfully*. They tend to correct errors and meander from what was said. But they are intelligent. That enables a few powerful workflows. Here are some examples:
+
+1. **Transcribe into other languages**. Gemini will handle the transcription and translation in a single step.
+2. **Summarize audio transcripts**. For example, convert a podcast into a tutorial, or a meeting recording into actions.
+3. **Legal Proceeding Analysis**. Extract case citations, dates, and other details from a legal debate.
+4. **Medical Consultation Summary**. Extract treatments, medications, details of next visit, etc. from a medical consultation.
+
+Here’s how to use Gemini to transcribe audio files.
+
+1. Get a [Gemini API key](https://aistudio.google.com/app/apikey) from Google AI Studio.
+2. Set the `GEMINI_API_KEY` environment variable to the API key.
+3. Set the `MP3_FILE` environment variable to the path of the MP3 file you want to transcribe.
+4. Run this code:
+
+   ```
+   curl -X POST https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash-002:streamGenerateContent?alt=sse \
+     -H "X-Goog-API-Key: $GEMINI_API_KEY" \
+     -H "Content-Type: application/json" \
+     -d "$(cat << EOF
+   {
+     "contents": [
+       {
+         "role": "user",
+         "parts": [
+           {
+             "inline_data": {
+               "mime_type": "audio/mp3",
+               "data": "$(base64 --wrap=0 $MP3_FILE)"
+             }
+           },
+           {"text": "Transcribe this"}
+         ]
+       }
+     ]
+   }
+   EOF
+   )"Copy to clipboardErrorCopied
+   ```
+
+[Previous
+
+Transforming Images](#/transforming-images)
+
+[Next
+
+6. Data Analysis](#/data-analysis)

markdown_files/Forecasting_with_Excel.md

@@ -0,0 +1,25 @@
+---
+title: "Forecasting with Excel"
+original_url: "https://tds.s-anand.net/#/forecasting-with-excel?id=forecasting-with-excel"
+downloaded_at: "2025-06-08T23:26:21.478299"
+---
+
+[Forecasting with Excel](#/forecasting-with-excel?id=forecasting-with-excel)
+----------------------------------------------------------------------------
+
+[![Forecasting with Excel](https://i.ytimg.com/vi_webp/QrTimmxwZw4/sddefault.webp)](https://youtu.be/QrTimmxwZw4)
+
+Here are links used in the video:
+
+* [FORECAST reference](https://support.microsoft.com/en-us/office/forecast-and-forecast-linear-functions-50ca49c9-7b40-4892-94e4-7ad38bbeda99)
+* [FORECAST.ETS reference](https://support.microsoft.com/en-us/office/forecast-ets-function-15389b8b-677e-4fbd-bd95-21d464333f41)
+* [Height-weight dataset](https://docs.google.com/spreadsheets/d/1iMFVPh8q9KgnfLwBeBMmX1GaFabP02FK/view) from [Kaggle](https://www.kaggle.com/datasets/burnoutminer/heights-and-weights-dataset)
+* [Traffic dataset](https://docs.google.com/spreadsheets/d/1w2R0fHdLG5ZGW-papaK7wzWq_-WDArKC/view) from [Kaggle](https://www.kaggle.com/datasets/fedesoriano/traffic-prediction-dataset)
+
+[Previous
+
+Regression with Excel](#/regression-with-excel)
+
+[Next
+
+Outlier Detection with Excel](#/outlier-detection-with-excel)

markdown_files/Function_Calling.md

@@ -0,0 +1,184 @@
+---
+title: "Function Calling"
+original_url: "https://tds.s-anand.net/#/function-calling?id=function-calling-with-openai"
+downloaded_at: "2025-06-08T23:26:04.973121"
+---
+
+[Function Calling with OpenAI](#/function-calling?id=function-calling-with-openai)
+----------------------------------------------------------------------------------
+
+[Function Calling](https://platform.openai.com/docs/guides/function-calling) allows Large Language Models to convert natural language into structured function calls. This is perfect for building chatbots and AI assistants that need to interact with your backend systems.
+
+OpenAI supports [Function Calling](https://platform.openai.com/docs/guides/function-calling) – a way for LLMs to suggest what functions to call and how.
+
+[![OpenAI Function Calling - Full Beginner Tutorial](https://i.ytimg.com/vi_webp/aqdWSYWC_LI/sddefault.webp)](https://youtu.be/aqdWSYWC_LI)
+
+Here’s a minimal example using Python and OpenAI’s function calling that identifies the weather in a given location.
+
+```
+# /// script
+# requires-python = ">=3.11"
+# dependencies = [
+#   "httpx",
+# ]
+# ///
+
+import httpx
+import os
+from typing import Dict, Any
+
+
+def query_gpt(user_input: str, tools: list[Dict[str, Any]]) -> Dict[str, Any]:
+    response = httpx.post(
+        "https://api.openai.com/v1/chat/completions",
+        headers={
+            "Authorization": f"Bearer {os.getenv('OPENAI_API_KEY')}",
+            "Content-Type": "application/json",
+        },
+        json={
+            "model": "gpt-4o-mini",
+            "messages": [{"role": "user", "content": user_input}],
+            "tools": tools,
+            "tool_choice": "auto",
+        },
+    )
+    return response.json()["choices"][0]["message"]
+
+
+WEATHER_TOOL = {
+    "type": "function",
+    "function": {
+        "name": "get_weather",
+        "description": "Get the current weather for a location",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "location": {"type": "string", "description": "City name or coordinates"}
+            },
+            "required": ["location"],
+            "additionalProperties": False,
+        },
+        "strict": True,
+    },
+}
+
+if __name__ == "__main__":
+    response = query_gpt("What is the weather in San Francisco?", [WEATHER_TOOL])
+    print([tool_call["function"] for tool_call in response["tool_calls"]])Copy to clipboardErrorCopied
+```
+
+### [How to define functions](#/function-calling?id=how-to-define-functions)
+
+The function definition is a [JSON schema](https://json-schema.org/) with a few OpenAI specific properties.
+See the [Supported schemas](https://platform.openai.com/docs/guides/structured-outputs#supported-schemas).
+
+Here’s an example of a function definition for scheduling a meeting:
+
+```
+MEETING_TOOL = {
+    "type": "function",
+    "function": {
+        "name": "schedule_meeting",
+        "description": "Schedule a meeting room for a specific date and time",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "date": {
+                    "type": "string",
+                    "description": "Meeting date in YYYY-MM-DD format"
+                },
+                "time": {
+                    "type": "string",
+                    "description": "Meeting time in HH:MM format"
+                },
+                "meeting_room": {
+                    "type": "string",
+                    "description": "Name of the meeting room"
+                }
+            },
+            "required": ["date", "time", "meeting_room"],
+            "additionalProperties": False
+        },
+        "strict": True
+    }
+}Copy to clipboardErrorCopied
+```
+
+### [How to define multiple functions](#/function-calling?id=how-to-define-multiple-functions)
+
+You can define multiple functions by passing a list of function definitions to the `tools` parameter.
+
+Here’s an example of a list of function definitions for handling employee expenses and calculating performance bonuses:
+
+```
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_expense_balance",
+            "description": "Get expense balance for an employee",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "employee_id": {
+                        "type": "integer",
+                        "description": "Employee ID number"
+                    }
+                },
+                "required": ["employee_id"],
+                "additionalProperties": False
+            },
+            "strict": True
+        }
+    },
+    {
+        "type": "function",
+        "function": {
+            "name": "calculate_performance_bonus",
+            "description": "Calculate yearly performance bonus for an employee",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "employee_id": {
+                        "type": "integer",
+                        "description": "Employee ID number"
+                    },
+                    "current_year": {
+                        "type": "integer",
+                        "description": "Year to calculate bonus for"
+                    }
+                },
+                "required": ["employee_id", "current_year"],
+                "additionalProperties": False
+            },
+            "strict": True
+        }
+    }
+]Copy to clipboardErrorCopied
+```
+
+Best Practices:
+
+1. **Use Strict Mode**
+   * Always set `strict: True` to ensure valid function calls
+   * Define all required parameters
+   * Set `additionalProperties: False`
+2. **Use tool choice**
+   * Set `tool_choice: "required"` to ensure that the model will always call one or more tools
+   * The default is `tool_choice: "auto"` which means the model will choose a tool only if appropriate
+3. **Clear Descriptions**
+   * Write detailed function and parameter descriptions
+   * Include expected formats and units
+   * Mention any constraints or limitations
+4. **Error Handling**
+   * Validate function inputs before execution
+   * Return clear error messages
+   * Handle missing or invalid parameters
+
+[Previous
+
+Hybrid RAG with TypeSense](#/hybrid-rag-typesense)
+
+[Next
+
+LLM Agents](#/llm-agents)

markdown_files/Geospatial_Analysis_with_Excel.md

@@ -0,0 +1,33 @@
+---
+title: "Geospatial Analysis with Excel"
+original_url: "https://tds.s-anand.net/#/geospatial-analysis-with-excel?id=geospatial-analysis-with-excel"
+downloaded_at: "2025-06-08T23:26:02.659173"
+---
+
+[Geospatial Analysis with Excel](#/geospatial-analysis-with-excel?id=geospatial-analysis-with-excel)
+----------------------------------------------------------------------------------------------------
+
+[![Geospatial analysis with Excel](https://i.ytimg.com/vi_webp/49LjxNvxyVs/sddefault.webp)](https://youtu.be/49LjxNvxyVs)
+
+You’ll learn how to create a data-driven story about coffee shop coverage in Manhattan, covering:
+
+* **Data Collection**: Collect and scrape data for coffee shop locations and census population from various sources.
+* **Data Processing**: Use Python libraries like geopandas for merging population data with geographic maps.
+* **Map Creation**: Generate coverage maps using tools like QGIS and Excel to visualize coffee shop distribution and population impact.
+* **Visualization**: Create physical, Power BI, and video visualizations to present the data effectively.
+* **Storytelling**: Craft a narrative around coffee shop competition, including strategic insights and potential market changes.
+
+Here are links that explain how the video was made:
+
+* [The Making of the Manhattan Coffee Kings](https://blog.gramener.com/the-making-of-manhattans-coffee-kings/)
+* [Shaping and merging maps](https://blog.gramener.com/shaping-and-merging-maps/)
+* [Visualizing data on 3D maps](https://blog.gramener.com/visualizing-data-on-3d-maps/)
+* [Physical and digital 3D maps](https://blog.gramener.com/physical-and-digital-3d-maps/)
+
+[Previous
+
+Data Analysis with ChatGPT](#/data-analysis-with-chatgpt)
+
+[Next
+
+Geospatial Analysis with Python](#/geospatial-analysis-with-python)

markdown_files/Geospatial_Analysis_with_Python.md

@@ -0,0 +1,34 @@
+---
+title: "Geospatial Analysis with Python"
+original_url: "https://tds.s-anand.net/#/geospatial-analysis-with-python?id=geospatial-analysis-with-python"
+downloaded_at: "2025-06-08T23:22:52.295346"
+---
+
+[Geospatial Analysis with Python](#/geospatial-analysis-with-python?id=geospatial-analysis-with-python)
+-------------------------------------------------------------------------------------------------------
+
+[![Geospatial analysis with Python](https://i.ytimg.com/vi_webp/m_qayAJt-yE/sddefault.webp)](https://youtu.be/m_qayAJt-yE)
+
+You’ll learn how to perform geospatial analysis for location-based decision making, covering:
+
+* **Distance Calculation**: Compute distances between various store locations and a reference point, such as the Empire State Building.
+* **Data Visualization**: Visualize store locations on a map using Python libraries like Folium.
+* **Store Density Analysis**: Determine the number of stores within a specified radius.
+* **Proximity Analysis**: Identify the closest and farthest stores from a specific location.
+* **Decision Making**: Use geospatial data to assess whether opening a new store is feasible based on existing store distribution.
+
+Here are links used in the video:
+
+* [Jupyter Notebook](https://colab.research.google.com/drive/1TwKw2pQ9XKSdTUUsTq_ulw7rb-xVhays?usp=sharing)
+* Learn about the [`pandas` package](https://pandas.pydata.org/pandas-docs/stable/user_guide/10min.html) and [video](https://youtu.be/vmEHCJofslg)
+* Learn about the [`numpy` package](https://numpy.org/doc/stable/user/whatisnumpy.html) and [video](https://youtu.be/8JfDAm9y_7s)
+* Learn about the [`folium` package](https://python-visualization.github.io/folium/latest/) and [video](https://youtu.be/t9Ed5QyO7qY)
+* Learn about the [`geopy` package](https://pypi.org/project/geopy/) and [video](https://youtu.be/3jj_5kVmPLs)
+
+[Previous
+
+Geospatial Analysis with Excel](#/geospatial-analysis-with-excel)
+
+[Next
+
+Geospatial Analysis with QGIS](#/geospatial-analysis-with-qgis)

markdown_files/Geospatial_Analysis_with_QGIS.md

@@ -0,0 +1,32 @@
+---
+title: "Geospatial Analysis with QGIS"
+original_url: "https://tds.s-anand.net/#/geospatial-analysis-with-qgis?id=geospatial-analysis-with-qgis"
+downloaded_at: "2025-06-08T23:23:28.541219"
+---
+
+[Geospatial Analysis with QGIS](#/geospatial-analysis-with-qgis?id=geospatial-analysis-with-qgis)
+-------------------------------------------------------------------------------------------------
+
+[![Geospatial analysis with QGIS](https://i.ytimg.com/vi_webp/tJhehs0o-ik/sddefault.webp)](https://youtu.be/tJhehs0o-ik)
+
+You’ll learn how to use QGIS for geographic data processing, covering:
+
+* **Shapefiles and KML Files**: Create and manage shapefiles and KML files for storing and analyzing geographic information.
+* **Downloading QGIS**: Install QGIS on different operating systems and familiarize yourself with its interface.
+* **Geospatial Data**: Access and utilize shapefiles from sources like Diva-GIS and integrate them into QGIS projects.
+* **Creating Custom Shapefiles**: Learn how to create custom shapefiles when existing ones are unavailable, including creating a shapefile for South Sudan.
+* **Editing and Visualization**: Use QGIS tools to edit shapefiles, add attributes, and visualize geographic data with various styling and labeling options.
+* **Exporting Data**: Export shapefiles or KML files for use in other applications, such as Google Earth.
+
+Here are links used in the video:
+
+* [QGIS Project](https://www.qgis.org/en/site/)
+* [Shapefile Data](https://www.diva-gis.org/gdata)
+
+[Previous
+
+Geospatial Analysis with Python](#/geospatial-analysis-with-python)
+
+[Next
+
+Network Analysis in Python](#/network-analysis-in-python)

markdown_files/Hybrid_RAG_with_TypeSense.md

@@ -0,0 +1,154 @@
+---
+title: "Hybrid RAG with TypeSense"
+original_url: "https://tds.s-anand.net/#/hybrid-rag-typesense?id=install-and-run-typesense"
+downloaded_at: "2025-06-08T23:25:43.332058"
+---
+
+[Hybrid Retrieval Augmented Generation (Hybrid RAG) with TypeSense](#/hybrid-rag-typesense?id=hybrid-retrieval-augmented-generation-hybrid-rag-with-typesense)
+--------------------------------------------------------------------------------------------------------------------------------------------------------------
+
+Hybrid RAG combines semantic (vector) search with traditional keyword search to improve retrieval accuracy and relevance. By mixing exact text matches with embedding-based similarity, you get the best of both worlds: precision when keywords are present, and semantic recall when phrasing varies. [TypeSense](https://typesense.org/) makes this easy with built-in hybrid search and automatic embedding generation.
+
+Below is a fully self-contained Hybrid RAG tutorial using TypeSense, Python, and the command line.
+
+### [Install and run TypeSense](#/hybrid-rag-typesense?id=install-and-run-typesense)
+
+[Install TypeSense](https://typesense.org/docs/guide/install-typesense.html).
+
+```
+mkdir typesense-data
+
+docker run -p 8108:8108 \
+  -v typesense-data:/data typesense/typesense:28.0 \
+  --data-dir /data \
+  --api-key=secret-key \
+  --enable-corsCopy to clipboardErrorCopied
+```
+
+* **`docker run`**: spins up a containerized TypeSense server on port 8108
+  + `-p 8108:8108` maps host port to container port.
+  + `-v typesense-data:/data` mounts a Docker volume for persistence.
+  + `--data-dir /data` points TypeSense at that volume.
+  + `--api-key=secret-key` secures your API.
+  + `--enable-cors` allows browser-based requests.
+
+**Expected output:**
+
+* Docker logs showing TypeSense startup messages, such as `Started Typesense API server`.
+* Listening on `http://0.0.0.0:8108`.
+
+### [Embed and import documents into TypeSense](#/hybrid-rag-typesense?id=embed-and-import-documents-into-typesense)
+
+Follow the steps in the [RAG with the CLI](#/rag-cli) tutorial to create a `chunks.json` that has one `{id, content}` JSON object per line.
+
+[TypeSense supports automatic embedding of documents](https://typesense.org/docs/28.0/api/vector-search.html#option-b-auto-embedding-generation-within-typesense). We’ll use that capability.
+
+Save the following as `addnotes.py` and run it with `uv run addnotes.py`.
+
+```
+# /// script
+# requires-python = ">=3.13"
+# dependencies = ["httpx"]
+# ///
+import json
+import httpx
+import os
+
+headers = {"X-TYPESENSE-API-KEY": "secret-key"}
+
+schema = {
+    "name": "notes",
+    "fields": [
+        {"name": "id", "type": "string", "facet": False},
+        {"name": "content", "type": "string", "facet": False},
+        {
+            "name": "embedding",
+            "type": "float[]",
+            "embed": {
+                "from": ["content"],
+                "model_config": {
+                    "model_name": "openai/text-embedding-3-small",
+                    "api_key": os.getenv("OPENAI_API_KEY"),
+                },
+            },
+        },
+    ],
+}
+
+with open("chunks.json", "r") as f:
+    chunks = [json.loads(line) for line in f.readlines()]
+
+with httpx.Client() as client:
+    # Create the collection
+    if client.get(f"http://localhost:8108/collections/notes", headers=headers).status_code == 404:
+        r = client.post("http://localhost:8108/collections", json=schema, headers=headers)
+
+    # Embed the chunks
+    result = client.post(
+        "http://localhost:8108/collections/notes/documents/import?action=emplace",
+        headers={**headers, "Content-Type": "text/plain"},
+        data="\n".join(json.dumps(chunk) for chunk in chunks),
+    )
+    print(result.text)Copy to clipboardErrorCopied
+```
+
+* **`httpx.Client`**: an HTTP client for Python.
+* **Collection schema**: `id` and `content` fields plus an `embedding` field with auto-generated embeddings from OpenAI.
+* **Auto-embedding**: the `embed` block instructs TypeSense to call the specified model for each document.
+* **`GET /collections/notes`**: checks existence.
+* **`POST /collections`**: creates the collection.
+* **`POST /collections/notes/documents/import?action=emplace`**: bulk upsert documents, embedding them on the fly.
+
+**Expected output:**
+
+* A JSON summary string like `{"success": X, "failed": 0}` indicating how many docs were imported.
+* (On timeouts, re-run until all chunks are processed.)
+
+### [4. Run a hybrid search and answer a question](#/hybrid-rag-typesense?id=_4-run-a-hybrid-search-and-answer-a-question)
+
+Now, we can use a single `curl` against the Multi-Search endpoint to combine keyword and vector search as a [hybrid search](https://typesense.org/docs/28.0/api/vector-search.html#hybrid-search):
+
+```
+Q="What does the author affectionately call the => syntax?"
+
+payload=$(jq -n --arg coll "notes" --arg q "$Q" \
+  '{
+     searches: [
+       {
+         collection: $coll,
+         q:           $q,
+         query_by:    "content,embedding",
+         sort_by:     "_text_match:desc",
+         prefix:      false,
+         exclude_fields: "embedding"
+       }
+     ]
+   }'
+)
+curl -s 'http://localhost:8108/multi_search' \
+  -H "X-TYPESENSE-API-KEY: secret-key" \
+  -d "$payload" \
+  | jq -r '.results[].hits[].document.content' \
+  | llm -s "${Q} - \$Answer ONLY from these notes. Cite verbatim from the notes." \
+  | uvx streamdownCopy to clipboardErrorCopied
+```
+
+* **`query_by: "content,embedding"`**: tells TypeSense to score by both keyword and vector similarity.
+* **`sort_by: "_text_match:desc"`**: boosts exact text hits.
+* **`exclude_fields: "embedding"`**: keeps responses lightweight.
+* **`curl -d`**: posts the search request.
+* **`jq -r`**: extracts each hit’s `content`. See [jq manual](https://stedolan.github.io/jq/manual/)
+* **`llm -s`** and **`uvx streamdown`**: generate and stream a grounded answer.
+
+**Expected output:**
+
+* The raw matched snippets printed first.
+* Then a concise, streamed LLM answer citing the note verbatim.
+
+[Previous
+
+RAG with the CLI)](#/rag-cli)
+
+[Next
+
+Function Calling](#/function-calling)

markdown_files/Images__Compression.md

@@ -0,0 +1,83 @@
+---
+title: "Images: Compression"
+original_url: "https://tds.s-anand.net/#/image-compression?id=images-compression"
+downloaded_at: "2025-06-08T23:26:15.003538"
+---
+
+[Images: Compression](#/image-compression?id=images-compression)
+----------------------------------------------------------------
+
+Image compression is essential when deploying apps. Often, pages have dozens of images. Image analysis runs over thousands of images. The cost of storage and bandwidth can grow over time.
+
+Here are things you should know when you’re compressing images:
+
+* **Image dimensions** are the width and height of the image in pixels. This impacts image size a lot
+* **Lossless** compression (PNG, WebP) preserves exact data
+* **Lossy** compression (JPEG, WebP) removes some data for smaller files
+* **Vector** formats (SVG) scale without quality loss
+* **WebP** is the modern standard, supporting both lossy and lossless
+
+Here’s a rule of thumb you can use as of 2025.
+
+* Use SVG if you can (i.e. if it’s vector graphics or you can convert it to one)
+* Else, reduce the image to as small as you can, and save as (lossy or lossless) WebP
+
+Common operations with Python:
+
+```
+from pathlib import Path
+from PIL import Image
+import io
+
+async def compress_image(input_path: Path, output_path: Path, quality: int = 85) -> None:
+    """Compress an image while maintaining reasonable quality."""
+    with Image.open(input_path) as img:
+        # Convert RGBA to RGB if needed
+        if img.mode == 'RGBA':
+            img = img.convert('RGB')
+        # Optimize for web
+        img.save(output_path, 'WEBP', quality=quality, optimize=True)
+
+# Batch process images
+paths = Path('images').glob('*.jpg')
+for p in paths:
+    await compress_image(p, p.with_suffix('.webp'))Copy to clipboardErrorCopied
+```
+
+Command line tools include [cwebp](https://developers.google.com/speed/webp/docs/cwebp), [pngquant](https://pngquant.org/), [jpegoptim](https://github.com/tjko/jpegoptim), and [ImageMagick](https://imagemagick.org/).
+
+```
+# Convert to WebP
+cwebp -q 85 input.png -o output.webp
+
+# Optimize PNG
+pngquant --quality=65-80 image.png
+
+# Optimize JPEG
+jpegoptim --strip-all --all-progressive --max=85 image.jpg
+
+# Convert and resize
+convert input.jpg -resize 800x600 output.jpg
+
+# Batch convert
+mogrify -format webp -quality 85 *.jpgCopy to clipboardErrorCopied
+```
+
+Watch this video on modern image formats and optimization (15 min):
+
+[![Modern Image Optimization (15 min)](https://i.ytimg.com/vi_webp/F1kYBnY6mwg/sddefault.webp)](https://youtu.be/F1kYBnY6mwg)
+
+Tools for image optimization:
+
+* [squoosh.app](https://squoosh.app/): Browser-based compression
+* [ImageOptim](https://imageoptim.com/): GUI tool for Mac
+* [sharp](https://sharp.pixelplumbing.com/): Node.js image processing
+* [Pillow](https://python-pillow.org/): Python imaging library
+
+[Previous
+
+Markdown](#/markdown)
+
+[Next
+
+Static hosting: GitHub Pages](#/github-pages)

markdown_files/Interactive_Notebooks__Marimo.md

@@ -0,0 +1,58 @@
+---
+title: "Interactive Notebooks: Marimo"
+original_url: "https://tds.s-anand.net/#/marimo?id=interactive-notebooks-marimo"
+downloaded_at: "2025-06-08T23:25:35.286078"
+---
+
+[Interactive Notebooks: Marimo](#/marimo?id=interactive-notebooks-marimo)
+-------------------------------------------------------------------------
+
+[Marimo](https://marimo.app/) is a new take on notebooks that solves some headaches of Jupyter. It runs cells reactively - when you change one cell, all dependent cells update automatically, just like a spreadsheet.
+
+Marimo’s cells can’t be run out of order. This makes Marimo more reproducible and easier to debug, but requires a mental shift from the Jupyter/Colab way of working.
+
+It also runs Python directly in the browser and is quite interactive. [Browse the gallery of examples](https://marimo.io/gallery). With a wide variety of interactive widgets, It’s growing popular as an alternative to Streamlit for building data science web apps.
+
+Common Operations:
+
+```
+# Create new notebook
+uvx marimo new
+
+# Run notebook server
+uvx marimo edit notebook.py
+
+# Export to HTML
+uvx marimo export notebook.pyCopy to clipboardErrorCopied
+```
+
+Best Practices:
+
+1. **Cell Dependencies**
+
+   * Keep cells focused and atomic
+   * Use clear variable names
+   * Document data flow between cells
+2. **Interactive Elements**
+
+   ```
+   # Add interactive widgets
+   slider = mo.ui.slider(1, 100)
+   # Create dynamic Markdown
+   mo.md(f"{slider} {"🟢" * slider.value}")Copy to clipboardErrorCopied
+   ```
+3. **Version Control**
+
+   * Keep notebooks are Python files
+   * Use Git to track changes
+   * Publish on [marimo.app](https://marimo.app/) for collaboration
+
+[!["marimo: an open-source reactive notebook for Python" - Akshay Agrawal (Nbpy2024)](https://i.ytimg.com/vi_webp/9R2cQygaoxQ/sddefault.webp)](https://youtu.be/9R2cQygaoxQ)
+
+[Previous
+
+Narratives with LLMs](#/narratives-with-llms)
+
+[Next
+
+HTML Slides: RevealJS](#/revealjs)

markdown_files/JSON.md

@@ -0,0 +1,8 @@
+---
+title: "JSON"
+original_url: "https://tds.s-anand.net/#/revealjs"
+downloaded_at: "2025-06-08T23:25:13.176607"
+---
+
+404 - Not found
+===============

markdown_files/JavaScript_tools__npx.md

@@ -0,0 +1,47 @@
+---
+title: "JavaScript tools: npx"
+original_url: "https://tds.s-anand.net/#/npx?id=javascript-tools-npx"
+downloaded_at: "2025-06-08T23:21:38.208039"
+---
+
+[JavaScript tools: npx](#/npx?id=javascript-tools-npx)
+------------------------------------------------------
+
+[npx](https://docs.npmjs.com/cli/v8/commands/npx) is a command-line tool that comes with npm (Node Package Manager) and allows you to execute npm package binaries and run one-off commands without installing them globally. It’s essential for modern JavaScript development and data science workflows.
+
+For data scientists, npx is useful when:
+
+* Running JavaScript-based data visualization tools
+* Converting notebooks and documents
+* Testing and formatting code
+* Running development servers
+
+Here are common npx commands:
+
+```
+# Run a package without installing
+npx http-server .                # Start a local web server
+npx prettier --write .           # Format code or docs
+npx eslint .                     # Lint JavaScript
+npx typescript-node script.ts    # Run TypeScript directly
+npx esbuild app.js               # Bundle JavaScript
+npx jsdoc .                      # Generate JavaScript docs
+
+# Run specific versions
+npx prettier@3.2 --write .        # Use prettier 3.2
+
+# Execute remote scripts (use with caution!)
+npx github:user/repo            # Run from GitHubCopy to clipboardErrorCopied
+```
+
+Watch this introduction to npx (6 min):
+
+[![What you can do with npx (6 min)](https://i.ytimg.com/vi_webp/55WaAoZV_tQ/sddefault.webp)](https://youtu.be/55WaAoZV_tQ)
+
+[Previous
+
+Python tools: uv](#/uv)
+
+[Next
+
+Unicode](#/unicode)

markdown_files/LLM_Agents.md

@@ -0,0 +1,123 @@
+---
+title: "LLM Agents"
+original_url: "https://tds.s-anand.net/#/llm-agents?id=command-line-agent-example"
+downloaded_at: "2025-06-08T23:25:53.665479"
+---
+
+[LLM Agents: Building AI Systems That Can Think and Act](#/llm-agents?id=llm-agents-building-ai-systems-that-can-think-and-act)
+-------------------------------------------------------------------------------------------------------------------------------
+
+LLM Agents are AI systems that can define and execute their own workflows to accomplish tasks. Unlike simple prompt-response patterns, agents make multiple LLM calls, use tools, and adapt their approach based on intermediate results. They represent a significant step toward more autonomous AI systems.
+
+[![Building LLM Agents with LangChain (13 min)](https://i.ytimg.com/vi_webp/DWUdGhRrv2c/sddefault.webp)](https://youtu.be/DWUdGhRrv2c)
+
+### [What Makes an Agent?](#/llm-agents?id=what-makes-an-agent)
+
+An LLM agent consists of three core components:
+
+1. **LLM Brain**: Makes decisions about what to do next
+2. **Tools**: External capabilities the agent can use (e.g., web search, code execution)
+3. **Memory**: Retains context across multiple steps
+
+Agents operate through a loop:
+
+* Observe the environment
+* Think about what to do
+* Take action using tools
+* Observe results
+* Repeat until task completion
+
+### [Command-Line Agent Example](#/llm-agents?id=command-line-agent-example)
+
+We’ve created a minimal command-line agent called [`llm-cmd-agent.py`](llm-cmd-agent.py) that:
+
+1. Takes a task description from the command line
+2. Generates code to accomplish the task
+3. Automatically extracts and executes the code
+4. Passes the results back to the LLM
+5. Provides a final answer or tries again if the execution fails
+
+Here’s how it works:
+
+```
+uv run llm-cmd-agent.py "list all Python files under the current directory, recursively, by size"
+uv run llm-cmd-agent.py "convert the largest Markdown file to HTML"Copy to clipboardErrorCopied
+```
+
+The agent will:
+
+1. Generate a shell script to list files with their sizes
+2. Execute the script in a subprocess
+3. Capture the output (stdout and stderr)
+4. Pass the output back to the LLM for interpretation
+5. Present a final answer to the user
+
+Under the hood, the agent follows this workflow:
+
+1. Initial prompt to generate a shell script
+2. Code extraction from the LLM response
+3. Code execution in a subprocess
+4. Result interpretation by the LLM
+5. Error handling and retry logic if needed
+
+This demonstrates the core agent loop of:
+
+* Planning (generating code)
+* Execution (running the code)
+* Reflection (interpreting results)
+* Adaptation (fixing errors if needed)
+
+### [Agent Architectures](#/llm-agents?id=agent-architectures)
+
+Different agent architectures exist for different use cases:
+
+1. **ReAct** (Reasoning + Acting): Interleaves reasoning steps with actions
+2. **Reflexion**: Adds self-reflection to improve reasoning
+3. **MRKL** (Modular Reasoning, Knowledge and Language): Combines neural and symbolic modules
+4. **Plan-and-Execute**: Creates a plan first, then executes steps
+
+### [Real-World Applications](#/llm-agents?id=real-world-applications)
+
+LLM agents can be applied to various domains:
+
+1. **Research assistants** that search, summarize, and synthesize information
+2. **Coding assistants** that write, debug, and explain code
+3. **Data analysis agents** that clean, visualize, and interpret data
+4. **Customer service agents** that handle queries and perform actions
+5. **Personal assistants** that manage schedules, emails, and tasks
+
+### [Project Ideas](#/llm-agents?id=project-ideas)
+
+Here are some practical agent projects you could build:
+
+1. **Study buddy agent**: Helps create flashcards, generates practice questions, and explains concepts
+2. **Job application assistant**: Searches job listings, tailors resumes, and prepares interview responses
+3. **Personal finance agent**: Categorizes expenses, suggests budgets, and identifies savings opportunities
+4. **Health and fitness coach**: Creates workout plans, tracks nutrition, and provides motivation
+5. **Course project helper**: Breaks down assignments, suggests resources, and reviews work
+
+### [Best Practices](#/llm-agents?id=best-practices)
+
+1. **Clear instructions**: Define the agent’s capabilities and limitations
+2. **Effective tool design**: Create tools that are specific and reliable
+3. **Robust error handling**: Agents should recover gracefully from failures
+4. **Memory management**: Balance context retention with token efficiency
+5. **User feedback**: Allow users to correct or guide the agent
+
+### [Limitations and Challenges](#/llm-agents?id=limitations-and-challenges)
+
+Current LLM agents face several challenges:
+
+1. **Hallucination**: Agents may generate false information or tool calls
+2. **Planning limitations**: Complex tasks require better planning capabilities
+3. **Tool integration complexity**: Each new tool adds implementation overhead
+4. **Context window constraints**: Limited memory for long-running tasks
+5. **Security concerns**: Tool access requires careful permission management
+
+[Previous
+
+Function Calling](#/function-calling)
+
+[Next
+
+LLM Image Generation](#/llm-image-generation)