On 2022-08-10 we have set up the same ssh key across the 4 complexes using the ac-login node as a master key. This change addressed the issue of ssh errors when connecting regarding host key changes for a given host after an update, such as:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: POSSIBLE DNS SPOOFING DETECTED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The ECDSA host key for hpc-login has changed,
and the key for the corresponding IP address 10.100.192.100
is unknown. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:QdNPyN2jAR5m7ngLbtIUjc2JgzknvFP2flMOGbd1i5k.
Please contact your system administrator.
Add correct host key in /home/user/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/user/.ssh/known_hosts:4
ECDSA host key for hpc-login has changed and you have requested strict checking.
Host key verification failed.

You may need to delete old entries on your ~/.ssh/known_hosts when connecting after the session to make sure the new host keys are added, but you should not need to do this again thereafter. For example, you may use the following command to remove the entries for hpc-login:

ssh-keygen -R hpc-login

A similar command may be run for other hostnames with the same problem. After that, a new connection should prompt you to accept the new key.

Cron service

The cron service on the hpc-cron and ecs-cron node. Any tasks that need to be added into a crontab should be placed one of these nodes. In case of node failure or system session, those crontabs would automatically be moved to another node.

Minor bugfixes and updates

  • The ab-login/ab-batch (ab6-100) login was rebooted to benefit from the last updates. Any sessions active at the time were terminated.
  • SLURM_EXPORT_ENV will be defined to ALL for non-interactive shells, addressing the non-consistent behaviour for remote submission of ksh jobs.
  • Lmod module system version update to 8.7.5 addressing a minor bug. Reverted due to a bug discovered.