Investigating with AI

Commands worth knowing

You do not need to memorise this. Skim it once, then come back when you need something. Everything here is already installed on the course server.

A convention used below: <like-this> means replace this with your own value, brackets included.

Getting around

Command What it does
pwd Where am I?
ls What is here? ls -l for details, ls -lh for readable sizes, ls -a to include hidden files
cd <folder> Go into a folder. cd .. goes up one. cd alone goes home
tree -L 2 Show the folder structure, two levels deep
mkdir <name> Make a folder. mkdir -p a/b/c makes the whole chain

Press Tab to complete a name you have started typing. Press the Up arrow to bring back a previous command. These two habits save more time than anything else here.

Looking at files without opening an editor

Command What it does
bat <file> Show a file with syntax highlighting and line numbers
head -20 <file> First 20 lines — the fastest way to check a CSV's columns
tail -20 <file> Last 20 lines. tail -f <log> follows a file as it grows
less <file> Page through a big file. Arrow keys scroll, /word searches, q quits
wc -l <file> Count lines — how many rows does this dataset have?

Finding things

Command What it does
fd <name> Find files by name, recursively. fd '\.csv$' finds every CSV
rg <text> Search inside files for text. Fast, skips junk automatically
rg -l <text> Just list which files contain it
fzf Fuzzy-pick a file interactively. Try cd $(fd -t d \| fzf)
history \| rg ssh What was that command I ran last week?

Data files from the command line

Often faster than opening Python, especially for a first look.

Command What it does
csvlook <file>.csv Print a CSV as a readable aligned table
csvstat <file>.csv Per-column summary statistics
mlr --icsv --opprint head -n 5 <file>.csv First rows, nicely formatted
mlr --icsv --ojson cut -f age,sex <file>.csv Pull specific columns out
duckdb -c "SELECT count(*) FROM '<file>.csv'" Run SQL directly on a CSV or parquet file, no import step
pdftotext <file>.pdf - Extract the text of a PDF to the screen
tesseract <scan>.png - OCR a scanned image (English and French installed)

duckdb is the one to remember. It reads CSV and parquet files far larger than memory and answers in SQL, which is often quicker than writing pandas.

Keeping your work alive

Your session dies when your connection drops — unless you use tmux.

Command What it does
tmux Start a persistent session
Ctrl-b then d Detach. Everything keeps running
tmux attach Reattach, from any computer
tmux ls List your sessions
Ctrl-b then c / n / p New window / next / previous

Start any job longer than a coffee inside tmux. There is no way to rescue a long job that was not.

Space and system

Command What it does
quota -s How much of your 25 GB is left
du -sh ~/* \| sort -h What is using it, biggest last
ncdu ~ Interactive disk browser — navigate and delete. The best tool for a full quota
duf Free space per filesystem
btop Live CPU, memory and process view. q quits
nproc / free -h How many cores / how much memory the server has

The server has 8 cores shared by the whole course. If btop shows it pinned, someone is running something heavy — that is usually why things feel slow.

Files and transfers

Command What it does
cp <a> <b> Copy. cp -r for folders
mv <a> <b> Move or rename
rm <file> Delete. There is no recycle bin. rm -r for folders
rsync -av <src>/ <dest>/ Copy a folder efficiently; resumable, skips unchanged files
zip -r out.zip <folder> Make a zip to hand in

Before any rm -r, run the same path through ls first. This costs two seconds and has saved a great many dissertations.

Editing text

Command What it does
nano <file> Simplest editor. Ctrl-O saves, Ctrl-X exits; shortcuts shown at the bottom
micro <file> Like nano but with normal Ctrl-C / Ctrl-V / Ctrl-S
vim <file> Powerful, steep. If you land in it by accident: press Esc, then type :q! and Enter

Most of the time you will ask Claude Code to make the edit instead.

Python

Command What it does
uv sync Build this project's environment from pyproject.toml
uv run python <script>.py Run a script inside that environment
uv add <package> Add a dependency and record it
uv run jupyter lab --no-browser --port 8888 Start Jupyter for this project

Never pip install. uv records what you installed so the project stays reproducible; pip leaves no trace of how the environment was built.

JupyterLab in a browser

Go to https://investigatingwithai.com/jupyter and sign in with your normal server username and password. Choose the kernel Course (data science).

To make your own project environment available there as a kernel:

uv add --dev ipykernel
uv run python -m ipykernel install --user --name <my-project> --display-name "<My project>"

The shared environment

Every common package is already installed in a shared environment, so you do not have to build one to try something out:

course-python                    # a Python REPL with everything available
course-python myscript.py        # run a script with it
course-jupyter lab --no-browser --port 8888

In JupyterLab it appears as the kernel Course (data science).

It holds pandas, polars, duckdb, statsmodels, linearmodels, scikit-learn, xgboost, lightgbm, shap, PyTorch, transformers, sentence-transformers, spaCy (English and French), NLTK, gensim, BERTopic, geopandas, networkx, plotly, altair and the Claude API client.

Use it for exploring. Use uv inside a project for anything you will hand in — the shared environment is read-only and shared, so it cannot record what your analysis depends on. That distinction is the difference between a result someone can reproduce and one they cannot.

Git and GitHub

Command What it does
git status What have I changed? Run this constantly
git add -A && git commit -m "<message>" Save a checkpoint
git log --oneline History
git diff What exactly changed since the last commit
git checkout -- <file> Throw away my changes to that file
gh auth login Connect your GitHub account (once)
gh repo create <name> --private --source=. --push Publish this folder

Commit before asking Claude Code to change several files at once. It is the only reliable undo you have.

Claude Code

Command What it does
claude Start it in the current folder
/status Which account and configuration are active
/permissions What it may and may not touch here
/usage How much of your weekly allowance is left
/explore-data <path> First-pass description of a dataset
/check-repro Does this project actually run end to end?
/explain Plain-language reading of the last result

When you are stuck

Command What it does
man <command> The manual. q quits
<command> --help Usually shorter and more useful than man
Ctrl-C Stop whatever is running
Ctrl-D or exit Close the session

And if a command is doing something you did not intend: Ctrl-C. Nothing here is so urgent that it is worth letting it finish.