Issues installing gdal-bin (libmysqlclient21 dependency) on 20.04.3 (databricks job clusters)

Question

I've had, in the past, gdal utilities installed successfully on a Databricks Cluster running 20.04.3 LTS (focal).

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"

$ sudo apt-get install -y gdal-bin

Each time a databricks job cluster startups, my sample notebook attempts to install GDAL. I've attempted a series of apt-get updates/upgrades; and just when I think the issue has gone away; the next day it comes back again (no change of code, cluster spec, etc.)

Most recently the issue occurs once it hits installation for libmysqlclient21, e.g:

Err:27 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmysqlclient21 amd64 8.0.26-0ubuntu0.20.04.3
  404  Not Found [IP: 91.189.88.142 80]

My understanding is that these issues arise when the packaging versions are no longer supported, and noticed that libmysqlclient21 has come out with a recent patch as of the 25th of October (when the issue started arising).

I've noticed when I run %sh sudo apt-get --fix-missing -y update and apt policy libmysqlclient21, prior to a successful installation, I can see it's found 8.0.27-0ubuntu0.20.04.1:

Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
Hit:3 https://repos.azul.com/zulu/deb stable InRelease
Get:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:5 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [628 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,635 kB]
Get:8 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [801 kB]
Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,183 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,086 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [33.4 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [679 kB]
Fetched 6,374 kB in 19s (329 kB/s)
Reading package lists...

libmysqlclient21:
Installed: (none)
Candidate: 8.0.27-0ubuntu0.20.04.1
Version table:
  8.0.27-0ubuntu0.20.04.1 500
    500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
    500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
  8.0.19-0ubuntu5 500
    500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

On an unsuccessful run, it's returning:

Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
Hit:3 https://repos.azul.com/zulu/deb stable InRelease
Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Reading package lists...

libmysqlclient21:
Installed: (none)
Candidate: 8.0.26-0ubuntu0.20.04.3
Version table:
  8.0.26-0ubuntu0.20.04.3 500
    500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
  8.0.26-0ubuntu0.20.04.2 500
    500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
  8.0.19-0ubuntu5 500
    500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

I've been scratching my head trying to understand what could be causing this, and what else to do to work around the problem. I'd prefer it to handle the installation automatically without having to hardcode specific versions (in case other issues were to arise); but not against targeting specific versions if I know they'll be supported on the current platform.

Any advice appreciated.

Answer 1

I had success by putting

sudo apt clean && sudo apt update --fix-missing -y && sudo apt install -y libmysqlclient21

before trying to install gdal and it seems to be working now....

...nevermind began failing the next day again...