diff --git a/.gitignore b/.gitignore index ad8261f96..162cc1476 100644 --- a/.gitignore +++ b/.gitignore @@ -2,4 +2,6 @@ /docs/all_md.txt /docs/basic_skeleton.sh /.vscode -.cache \ No newline at end of file +.cache +.claude/ +.opencode/ diff --git a/docs/Tools/VSCode/cluster_node.md b/docs/Tools/VSCode/cluster_node.md index 416223bee..da61a798f 100644 --- a/docs/Tools/VSCode/cluster_node.md +++ b/docs/Tools/VSCode/cluster_node.md @@ -1,192 +1,354 @@ -# VSCode in interactive node +# VSCode in IRB Cluster + +!!! warning "VPN Required" + For any case, you need to activate the VPN. ## Description These are the instructions to use Visual Studio Code to run and debug scripts/notebooks -within an **interactive node** from the cluster. +within the IRB cluster. Three approaches are documented: + +- **Login Node**: For non-computationally expensive tasks +- **Interactive Node – Method A (Dynamic Node)**: For computationally expensive tasks with flexible resource allocation +- **Interactive Node – Method B (Fixed Node)**: For computationally expensive tasks, always connecting to the same node + +## Not Computationally Expensive Tasks - Login Node + +1. Open VSCode in your local computer +2. On the sidebar, click the icon that looks like a screen +3. Click on the option that says `irbcluster02`, either the right arrow (open in current window) or the +other icon (new window). + +## Computationally Expensive Tasks - Interactive Node + +Both methods below run VSCode connected to a compute node via SSH. Choose based on your workflow: + +- **Method A (Dynamic Node)**: Allocates a new node each time via the `interactive` command. +Better for one-off jobs with flexible resource needs. +- **Method B (Fixed Node)**: Always submits to the same node via `sbatch`. +Better for ongoing work where you want to preserve your VSCode session history (recently opened files, workspace state). + +===+ "Method A: Dynamic Node (interactive)" + + !!! tip "When to use Method A" + Use this method when you need **flexible resources** or a different node each time. + Keep in mind that every time you are allocated a different node, VSCode will not remember + your previously opened files and folders for that session. + + #### Step 1: [CLUSTER] Open interactive session -## Using VSCode App + !!! note "Existing screen session" + If you already have a screen session opened in the cluster with the interactive session, go directly to step 4. -### Step 1: [CLUSTER] Open interactive session + The first step is to allocate the resources you will need for your executions. + We can do this with the `interactive` command. -The first step is to allocate the resources you will need for your executions. -We can do this with the `interactive` command. + !!! tip "Tip: Use `screen`" + Consider launching the interactive session in a screen so that it doesn't get killed when the terminal is closed. -!!! tip "Tip: Use `screen`" - Consider launching the interactive session in a screen so that it doesn't get killed when the terminal is closed. + ```bash + screen -S vscode + ``` + + Launch an interactive session within a given node, allocating some computing resources: ```bash - screen -S vscode + interactive -c 4 -m 16 ``` -Launch an interactive session within a given node, allocating some computing resources: + Then, check on which node you are. You can look at the prompt, or use the `hostname` command. -```bash -interactive -c 6 -m 20 -``` + ```bash + hostname + ``` -Then, check on which node you are. You can look at the prompt, or use the `hostname` command. + Example: -```bash -hostname -``` + ```bash + $ hostname + irbccn39.hpc.irbbarcelona.pcb.ub.es + ``` -Example: + #### Step 2: [CLUSTER] Execute job vscode-interactive-job.sh -```bash -$ hostname -irbccn43.hpc.irbbarcelona.pcb.ub.es -``` + !!! note "First time setup" + If it is the first time you are using this method, copy the following lines into a + file called `vscode-interactive-job.sh` in your home directory. -### Step 2: [CLUSTER] Execute job vscode-interactive-job.sh + **File: `vscode-interactive-job.sh`** -!!! note "First time setup" - If it is the first time you are using this method, copy the following lines into a - file called `vscode-interactive-job.sh` in your home directory. + ```bash + #!/bin/bash + #SBATCH --job-name="tunnel" + #SBATCH --time=8:00:00 # walltime - **File: `vscode-interactive-job.sh`** + /usr/sbin/sshd -D -p 2222 -f /dev/null -h ${HOME}/.ssh/id_ecdsa # uses the user key as the host key + ``` + + Execute the `vscode-interactive-job.sh` script with: ```bash - #!/bin/bash - #SBATCH --job-name="tunnel" - #SBATCH --time=8:00:00 # walltime - - /usr/sbin/sshd -D -p 2222 -f /dev/null -h ${HOME}/.ssh/id_ecdsa # uses the user key as the host key + bash vscode-interactive-job.sh ``` -Execute the `vscode-interactive-job.sh` script with: + !!! failure "`id_ecdsa` not found" + If you get an error saying that the file `id_ecdsa` is not found, you can generate it with: -```bash -bash vscode-interactive-job.sh -``` + ```bash + ssh-keygen -t ecdsa + ``` -!!! failure "`id_ecdsa` not found" - If you get an error saying that the file `id_ecdsa` is not found, you can generate it with the following command: + It will prompt you several times to confirm the generation of the key. You can just press `Enter` to accept the default values. + + The terminal will look like it got stuck, but what is happening is that it is waiting for + the SSH tunnel to be established. + + #### Step 3: [LOCAL] [ONE-TIME-STEP] Add the SSH configuration + + Modify the file `~/.ssh/config` in your local machine to include the following configuration: ```bash - ssh-keygen -t ecdsa + Host irbccn* + HostName %h + ProxyJump irblogin02.sc.irbbarcelona.org + User + Port 22 ``` - It will prompt you several times to confirm the generation of the key. You can just press `Enter` to accept the default values. + Example: -The terminal will look like it got stuck, but what is happening is that it is waiting for -the SSH tunnel to be established. + ```bash + Host irbccn39 + HostName %h + ProxyJump irblogin02.sc.irbbarcelona.org + User clopeze + Port 22 + ``` -### Step 3: [LOCAL] Open SSH tunnel + !!! warning "Node change" + This is marked as a "*ONE-TIME-STEP*", but in reality it depends on the node you are allocated. + If you need to change the node or add a new node, **the configuration will need to be updated**. + + !!! tip "Tip (Optional): Helper function to add SSH configuration" + Here is a helper function that you can add to your `.bashrc` file to make it easier to + setup the node and the tunnel. + + Steps the function does: + + 1. Prompts only for the **host alias**. + 2. Checks if a config entry for that **host already exists** in `~/.ssh/config` (and skips adding it if so). + 3. Uses the fixed configuration values (with HostName as "%h", the fixed ProxyJump, port 22, and the current computer user). + 4. Computes the full hostname for the tunnel (by appending ".hpc.irbbarcelona.pcb.ub.es" to the alias) and creates the SSH tunnel (running in the background). + + ```bash + add_cluster_ssh_host_with_tunnel() { + read -p "Enter host alias (e.g., irbccn39): " host_alias + current_user=$(whoami) + + # Check if SSH config for this host already exists + if grep -qE "^Host[[:space:]]+${host_alias}\$" ~/.ssh/config; then + echo "SSH configuration for host '${host_alias}' already exists. Skipping configuration update." + else + new_entry="Host ${host_alias} + HostName %h + ProxyJump irblogin02.sc.irbbarcelona.org + User ${current_user} + Port 22 + " + echo -e "\n${new_entry}" >> ~/.ssh/config + echo "SSH configuration added for host '${host_alias}'." + fi + + # Create the SSH tunnel. + # Assuming the full hostname is .hpc.irbbarcelona.pcb.ub.es + computed_hostname="${host_alias}.hpc.irbbarcelona.pcb.ub.es" + tunnel_cmd="ssh -f -N -L 2222:${computed_hostname}:22 ${current_user}@irblogin02.sc.irbbarcelona.org" + echo "Establishing SSH tunnel with command:" + echo "${tunnel_cmd}" + eval ${tunnel_cmd} + if [ $? -eq 0 ]; then + echo "SSH tunnel established for host '${host_alias}'." + else + echo "Failed to establish SSH tunnel." + fi + } + + alias addsshcluster='add_cluster_ssh_host_with_tunnel' + ``` + + #### Step 3.1: [LOCAL] [ONE-TIME-STEP] Access through the terminal to the node + + To make sure the configuration is working, access the node through the terminal with: -In your local machine, you need to open the SSH tunnel with the following command: + ```bash + ssh irbccn* + ``` -```bash -ssh -L 2222::22 @irblogin01.irbbarcelona.pcb.ub.es -``` + Example: -Example: + ```bash + ssh irbccn39 + ``` + + This is needed the first time you connect to a new node, so that the node is added to the known hosts. + + #### Step 4: [VSCODE] Connect VSCode to the interactive node + + Open Visual Studio Code and make sure you have installed the `Remote - SSH` extension. + + Open the side bar at the "Remote - SSH" panel, and then click the `irbccn*` option (e.g., `irbccn39`), either the right + arrow (open in current window) or the other icon (new window). A new window will open with the + terminal connected to the interactive node. All the execution of notebooks and debuggers will be done from this node. -```bash -ssh -L 2222:irbccn43.hpc.irbbarcelona.pcb.ub.es:22 clopeze@irblogin01.irbbarcelona.pcb.ub.es -``` + !!! note "Connection retry" + You might have to click **Retry** a few times, as sometimes it doesn't connect at the first try. -### Step 4: [LOCAL] [ONE-TIME-STEP] Add the SSH configuration + !!! tip "Tip: Create a 'Profile' to keep the VSCode settings and extensions" + You can create a profile in VSCode to keep the settings and extensions for the connection to the cluster. + This works similar to the idea of conda environments, but for the VSCode settings. -Modify the file `~/.ssh/config` in your local machine to include the following configuration: + To create a profile, click on the gear icon in the bottom left corner of VSCode, and then click on Profiles and create a new profile. -```bash -Host irbccn* - HostName %h - ProxyJump irblogin01.irbbarcelona.pcb.ub.es - User - Port 2222 -``` + #### [OPTIONAL] Open SSH tunnel — only if the connection from VSCode doesn't work -Example: + !!! note "If VSCode connection fails" + If opening VSCode on the local computer and trying to connect to the cluster doesn't work, + run the following command in a new terminal on your local machine: -```bash -Host irbccn43 - HostName %h - ProxyJump irblogin01.irbbarcelona.pcb.ub.es - User clopeze - Port 2222 -``` + ```bash + ssh -L 2222::22 @irblogin02.sc.irbbarcelona.org + ``` -!!! warning "Node change" - This is marked as a "*ONE-TIME-STEP*", but in reality it depends on the node you are allocated. - If you need to change the node or add a new node, **the configuration will need to be updated**. + Example: -!!! tip "Tip (Optional): Helper function to add SSH configuration" - Here is a helper function that you can add to your `.bashrc` file to make it easier to - setup the node and the tunnel. + ```bash + ssh -L 2222:irbccn39.hpc.irbbarcelona.pcb.ub.es:22 clopeze@irblogin02.sc.irbbarcelona.org + ``` - Steps the function does: + Replace `` with your allocated node (e.g., `irbccn39.hpc.irbbarcelona.pcb.ub.es`) and `` with your username. - 1. Prompts only for the **host alias**. - 2. Checks if a config entry for that **host already exists** in `~/.ssh/config` (and skips adding it if so). - 3. Uses the fixed configuration values (with HostName as "%h", the fixed ProxyJump, port 2222, and the current computer user). - 4. Computes the full hostname for the tunnel (by appending ".hpc.irbbarcelona.pcb.ub.es" to the alias) and creates the SSH tunnel (running in the background). +=== "Method B: Fixed Node (sbatch)" + + !!! tip "When to use Method B" + Use this method when you want to **always connect to the same node**, preserving your VSCode + session history (recently opened files, workspace state). It also requires fewer steps since + the job is submitted directly from the login node — no `screen` or `interactive` session needed. + + #### Step 1: [ONE-TIME-STEP] Create the SBATCH script + + Create the directory if it doesn't exist: ```bash - add_cluster_ssh_host_with_tunnel() { - read -p "Enter host alias (e.g., irbccn43): " host_alias - current_user=$(whoami) - - # Check if SSH config for this host already exists - if grep -qE "^Host[[:space:]]+${host_alias}\$" ~/.ssh/config; then - echo "SSH configuration for host '${host_alias}' already exists. Skipping configuration update." - else - new_entry="Host ${host_alias} + mkdir -p ~/.local/bin + ``` + + Then create the file `~/.local/bin/vscode-it` with the following content: + + **File: `~/.local/bin/vscode-it`** + + ```bash + #!/bin/bash + + #SBATCH --job-name="code-client" + #SBATCH --cpus-per-task=4 + #SBATCH --nodelist= + #SBATCH --partition=bbg_cpu_zen4 + #SBATCH --mem=16G + #SBATCH --qos=interactive + #SBATCH --time=7-00:00:00 # walltime + + set -xe + + /usr/sbin/sshd -D -p 2222 -f /dev/null -h ${HOME}/.ssh/id_ecdsa + ``` + + !!! warning "Adjust resources to your needs" + Edit `--nodelist`, `--cpus-per-task`, `--mem`, and `--partition` to match the node and + resources you want to reserve. The `--nodelist` value pins the job to a specific node — + this is the key feature of this method. + + !!! failure "`id_ecdsa` not found" + If you get an error saying that the file `id_ecdsa` is not found, you can generate it with: + + ```bash + ssh-keygen -t ecdsa + ``` + + It will prompt you several times to confirm the generation of the key. You can just press `Enter` to accept the default values. + + #### Step 2: [CLUSTER] Submit the job from the login node + + From the IRB cluster login node, run: + + ```bash + sbatch ~/.local/bin/vscode-it + ``` + + The job will be submitted to the SLURM queue and start running on the node specified in `--nodelist`. + You can verify it is running with: + + ```bash + squeue -u + ``` + + #### Step 3: [LOCAL] [ONE-TIME-STEP] Add the SSH configuration + + Modify the file `~/.ssh/config` in your local machine to include the following configuration: + + ```bash + Host irbccn* HostName %h - ProxyJump irblogin01.irbbarcelona.pcb.ub.es - User ${current_user} + ProxyJump irblogin02.sc.irbbarcelona.org + User Port 2222 - " - echo -e "\n${new_entry}" >> ~/.ssh/config - echo "SSH configuration added for host '${host_alias}'." - fi - - # Create the SSH tunnel. - # Assuming the full hostname is .hpc.irbbarcelona.pcb.ub.es - computed_hostname="${host_alias}.hpc.irbbarcelona.pcb.ub.es" - tunnel_cmd="ssh -f -N -L 2222:${computed_hostname}:22 ${current_user}@irblogin01.irbbarcelona.pcb.ub.es" - echo "Establishing SSH tunnel with command:" - echo "${tunnel_cmd}" - eval ${tunnel_cmd} - if [ $? -eq 0 ]; then - echo "SSH tunnel established for host '${host_alias}'." - else - echo "Failed to establish SSH tunnel." - fi - } - - alias addsshcluster='add_cluster_ssh_host_with_tunnel' ``` -#### Step 4.1: [LOCAL] [ONE-TIME-STEP] Access though the terminal to the node + Example (using the same node as the `--nodelist` in the script): + + ```bash + Host irbccn25 + HostName %h + ProxyJump irblogin02.sc.irbbarcelona.org + User clopeze + Port 3131 + ``` + + !!! note "Fixed node advantage" + Since this method always targets the same node, you only need to set up this + configuration once and it will continue to work across all future sessions. -To make sure the configuration is working, you can access the node through the terminal with the following command: + #### Step 3.1: [LOCAL] [ONE-TIME-STEP] Access through the terminal to the node -```bash -ssh irbccn* -``` + To make sure the configuration is working, access the node through the terminal with: -Example: + ```bash + ssh irbccn* + ``` + + Example: + + ```bash + ssh irbccn25 + ``` -```bash -ssh irbccn43 -``` + This is needed the first time you connect to the node, so that the node is added to the known hosts. -This is needed the first time you connect to a new node, so that the node is added to the known hosts. + #### Step 4: [VSCODE] Connect VSCode to the node -### Step 5: [VSCODE] Connect VSCode to the interactive node + Open Visual Studio Code and make sure you have installed the `Remote - SSH` extension. -Open Visual Studio Code and make sure you have installed the `Remote - SSH` extension. + Open the side bar at the "Remote - SSH" panel, and then click the `irbccn*` option (e.g., `irbccn25`), either the right + arrow (open in current window) or the other icon (new window). A new window will open with the + terminal connected to the node. All the execution of notebooks and debuggers will be done from this node. -Open the side bar at the "Remote - SSH" panel, and then click the `irbccn*` option. A new window will open with the -terminal connected to the interactive node. All the execution of notebooks and debuggers will be done from this node. + !!! note "Connection retry" + You might have to click **Retry** a few times, as sometimes it doesn't connect at the first try. -!!! tip "Tip: Create a 'Profile' to keep the VSCode settings and extensions" - You can create a profile in VSCode to keep the settings and extensions for the connection to the cluster. - This works similar to the idea of conda environments, but for the VSCode settings. + !!! tip "Tip: Create a 'Profile' to keep the VSCode settings and extensions" + You can create a profile in VSCode to keep the settings and extensions for the connection to the cluster. + This works similar to the idea of conda environments, but for the VSCode settings. - To create a profile, click on the gear icon in the bottom left corner of VSCode, and then click on Profiles and create a new profile. + To create a profile, click on the gear icon in the bottom left corner of VSCode, and then click on Profiles and create a new profile. ## From Browser (*code-server*)