README.md

A set of scripts to automatically transfer data from rig computers to the fileshare and analyze them with the cluster.

An alternative to transfero. Uses WSL2 instead of cygwin on the rig computer. Further, the rig computer, not the cluster, has the cron job which does the copy. All in shell instead of python.

Installation

In a PowerShell terminal:

wsl --install

After rebooting, open a WSL terminal.

If the D: drive on the Windows side is not already mounted to /mnt/d then:

sudo mkdir /mnt/d
sudo echo 'D: /mnt/d drvfs defaults 0 0' >> /etc/fstab
sudo mount -a

Add a user for the robot:

sudo adduser otopalikrobot

The scripts use zsh (not bash), so install it and set it as the default shell:

sudo apt install zsh
su -l otopalikrobot
chsh -s $(which zsh)

Copy "diff-rsync.sh" from this repo to the robot's home folder. Then set its execution bit:

chmod a+x diff-rsync.sh

Enable password-less ssh access to the cluster login node by first generating a RSA key pair on the rig computer:

ssh-keygen -t rsa -P "" -f ~/.ssh/id_rsa

And then manually copying .ssh/id_rsa.pub on the rig computer to .ssh/authorized_keys on the cluster fileshare.

While logged into the cluster, copy "classify-ethogram.sh" from this repo into the robot's home folder, and set its execution bit too:

chmod a+x classify-ethogram.sh

Examine the various configuration parameters stored in variables at the top of each of these two scripts. Change as necessary. Hopefully their names of descriptive enough.

If there are already data to analyze, check the everything works by doing so manually:

./diff-rsync.sh

Automatic Execution

Run the following in a PowerShell terminal with administrator privilege.

First, enable wake timers in the active power plan so scheduled tasks are allowed to wake the machine from sleep:

powercfg /SETACVALUEINDEX SCHEME_CURRENT SUB_SLEEP RTCWAKE 1
powercfg /SETACTIVE SCHEME_CURRENT

Set the WSL distribution and user that will run the script:

$distroName = "Ubuntu"     # Change if your distro name is different
$wslUser    = "otopalikrobot"

Define the action: launch diff-rsync.sh inside WSL via a login shell, writing stdout and stderr to a timestamped log file in the WSL user's home directory:

$action = New-ScheduledTaskAction `
    -Execute "C:\Windows\System32\wsl.exe" `
    -Argument "-d $distroName -u $wslUser -- zsh -lc `"`$HOME/diff-rsync.sh > `$HOME/diff-rsync.`$(date +`%Y`%m`%d`%H`%M`%S).log 2>&1`""

Define the trigger: every day at 2:00 AM:

$trigger = New-ScheduledTaskTrigger -Daily -At 2:00am

Define the task settings: wake the machine from sleep to run, run a missed task as soon as the machine is next available, ignore the trigger if a previous run is still going, retry up to 3 times at 10-minute intervals on failure, and kill the task if it runs longer than 6 hours:

$settings = New-ScheduledTaskSettingsSet `
    -WakeToRun `
    -StartWhenAvailable `
    -MultipleInstances IgnoreNew `
    -RestartCount 3 `
    -RestartInterval (New-TimeSpan -Minutes 10) `
    -ExecutionTimeLimit (New-TimeSpan -Hours 6)

Prompt for the current Windows user's password so the task can run unattended, without anyone being logged in:

$cred = Get-Credential -UserName $env:USERNAME -Message "Enter password for $env:USERNAME to allow unattended wake runs"

Finally, register the task with Windows Task Scheduler:

Register-ScheduledTask `
    -TaskName "Otopalik-Diff-Rsync" `
    -Action $action `
    -Trigger $trigger `
    -Settings $settings `
    -User $cred.UserName `
    -Password $cred.GetNetworkCredential().Password `
    -RunLevel Highest `
    -Description "Nightly diff-rsync data transfer and cluster ethogram analysis"

Once registered, the task persists across reboots — Windows stores it permanently, so no re-registration is needed. If the machine is asleep at 2:00 AM, it will be woken to run the task; if it is fully powered off, it cannot be woken, but the missed run will execute shortly after the next boot. Note that the task is registered with the Windows user's password, so if that password changes, the task will silently fail until re-registered.

Monitoring

On a different machine, say your Linux workstation, add this to its crontab:

0 6 * * * /groups/scicompsoft/home/arthurb/projects/otopalik/monitor-heartbeat.sh

You will receive an email in the morning if the Windows box's scheduled task isn't working.

You really want to do this because things can break. A simple reboot of Windows typically fixes it.

Notes

Previously I scheduled it to automatically run with crontab in WSL:

crontab -e
# 0 2 * * * $HOME/diff-rsync.sh > $HOME/diff-rsync.$(date +\%Y\%m\%d\%H\%M\%S).log 2>&1

And tried to ensure these scripts would continue to run automatically if the computer sleeps:

Open the Task Scheduler

Create a new task:
Click "Action" > "Create Basic Task"
Give your task a name like "Start WSL on Startup".

Set trigger:
Select "When the computer starts" as the trigger.

Set action:
Select "Start a program" as the action.
Browse to the "wsl.exe" executable file (usually located in "C:\Windows\System32\wsl.exe").

Set permissions (important):
Go to the "General" tab of the task properties.
Check the "Run whether user is logged on or not" option.
Set "Run with highest privileges".

On Windows 11 I also updated the WSL configuration by adding these lines to C:\Users<UserName>.wslconfig:

[boot]
command="service cron start"

[wsl2]
vmIdleTimeout=-1

And tried to make it reboot-safe with the following

Make a file called perpetual-wsl.bat in Windows with the following contents:
	wsl.exe --exec dbus-launch true
Add a shortcut to perpetual-wsl.bat in the windows start-up folder:
	`run shell:startup` and drag and drop

But none of this worked consistently.