Android ExoPlayer delay when playing audio

Question

In my app, I'm trying to use ExoPlayer to play audio and video together at the same time using MergingMediaSource.

The problem is that it buffers for several minutes, and when it starts playing the audio is really distorted, sometimes sped up for a few seconds or crackly.

I've spent hours searching for a solution and I've come up with nothing.

The player

val player = ExoPlayer.Builder(application)
    .setRenderersFactory(
        DefaultRenderersFactory(application)
            .setEnableAudioTrackPlaybackParams(true)
            .setEnableAudioOffload(true)
            .setExtensionRendererMode(DefaultRenderersFactory.EXTENSION_RENDERER_MODE_ON)
    )
    .build()

This is the code where I load the streams

val factory = ProgressiveMediaSource.Factory(DefaultHttpDataSource.Factory())
val videoItem = factory.createMediaSource(MediaItem.fromUri(streams.filterIsInstance<DomainStream.Video>().first().url))
val soundItem = factory.createMediaSource(MediaItem.fromUri(streams.filterIsInstance<DomainStream.Audio>().first().url))
val mergedSource = MergingMediaSource(true, videoItem, soundItem)

viewModel.player.setMediaSource(mergedSource)
viewModel.player.prepare()
viewModel.player.playWhenReady = true

Here's a log of what happens when I start the player

surfaceSize [eventTime=0.10, mediaPos=0.00, window=0, 1080, 608]
timeline [eventTime=0.86, mediaPos=0.00, window=0, periodCount=1, windowCount=1, reason=PLAYLIST_CHANGED
  period [?]
  window [?, seekable=false, dynamic=true]
]
mediaItem [eventTime=0.86, mediaPos=0.00, window=0, reason=PLAYLIST_CHANGED]
state [eventTime=0.86, mediaPos=0.00, window=0, BUFFERING]
playWhenReady [eventTime=0.86, mediaPos=0.00, window=0, true, USER_REQUEST]
timeline [eventTime=0.95, mediaPos=0.00, window=0, period=0, periodCount=1, windowCount=1, reason=SOURCE_UPDATE
  period [?]
  window [?, seekable=false, dynamic=false]
]
loading [eventTime=0.95, mediaPos=0.00, window=0, period=0, true]
timeline [eventTime=1.21, mediaPos=0.00, window=0, period=0, periodCount=1, windowCount=1, reason=SOURCE_UPDATE
  period [2048.92]
  window [2048.92, seekable=true, dynamic=false]
]
videoEnabled [eventTime=2.01, mediaPos=0.00, window=0, period=0]
audioEnabled [eventTime=2.02, mediaPos=0.00, window=0, period=0]
tracks [eventTime=2.02, mediaPos=0.00, window=0, period=0, []]
downstreamFormat [eventTime=2.02, mediaPos=0.00, window=0, period=0, id=1, mimeType=video/avc, codecs=avc1.640028, res=1920x1080]
videoDecoderInitialized [eventTime=2.03, mediaPos=0.00, window=0, period=0, c2.goldfish.h264.decoder]
videoInputFormat [eventTime=2.03, mediaPos=0.00, window=0, period=0, id=1, mimeType=video/avc, codecs=avc1.640028, res=1920x1080]
audioDecoderInitialized [eventTime=2.04, mediaPos=0.00, window=0, period=0, c2.android.aac.decoder]
audioInputFormat [eventTime=2.05, mediaPos=0.00, window=0, period=0, id=1, mimeType=audio/mp4a-latm, codecs=mp4a.40.5, channels=2, sample_rate=44100, language=en]
videoSize [eventTime=88.44, mediaPos=0.00, window=0, period=0, 1920, 1080]
renderedFirstFrame [eventTime=88.44, mediaPos=0.00, window=0, period=0, Surface(name=null)/@0xed8b97b]

EDIT: I also attempted to load just the audio using Androids built-in MediaPlayer API, and the same issue also occurred. Testing the same URL on my PC plays fine, so it couldn't be an issue with the source.