AVAudioSession .defaultToSpeaker changes mic input

Question

I have an app taps the microphone and also play sounds depending on mic input(don't have to be simultaneously tho) This code below works. But one problem is the output plays on the small top speaker and not the bottom real loud speakers. I could solve this problem strangely by putting the 3 lines below just before the player starts, Then I can hear the sound on speakers. But then the microphone stops listening! Even after the player stops playing. Basically mic does not like when it is

.defaultToSpeaker

Any idea?

Here also documented what I am trying to do is correct:

https://developer.apple.com/documentation/avfoundation/avaudiosession/categoryoptions/1616462-defaulttospeaker

UPDATE: I minimized the problem. No Player just mic. Code below, mic does not "work" when it is ".defaultToSpeaker". After some debugging I realized that defaultToSpeaker switches the mic from "bottom" to "front". And

 try preferredPort.setPreferredDataSource(source)

Cant seem to change it to bottom again. (I can provide code for this) And when category is defaultToSpeaker apperantly the tap buffer framelength is 4800 and not 4410. This difference seem causes trouble in my code because I need exactly 44100. So mic is actually working, but later in code it fails to do its job due to different SR. Below code can explain more.

 func tapMicrophone() {
    try? AVAudioSession.sharedInstance().setActive(false)
    try? AVAudioSession.sharedInstance().setCategory(.playAndRecord,  options: [.defaultToSpeaker])
    //setBottomMic()
    try? AVAudioSession.sharedInstance().setActive(true)

    //tracker.start()
    let input = engine.inputNode
    let inputFormat = input.outputFormat(forBus: 0)
    let sampleRate = Double(11025)
    let outputFormat = AVAudioFormat(commonFormat: .pcmFormatFloat32, sampleRate: sampleRate, channels: 1, interleaved: true)!
    let converter = AVAudioConverter(from: inputFormat, to: outputFormat)!
    let inputBufferSize = 44100 //  100ms of 44.1K = 4410 samples.
    let sampleRateRatio = 44100 / sampleRate

    input.installTap(onBus: 0, bufferSize: AVAudioFrameCount(inputBufferSize), format: inputFormat) {
        buffer, time in
        var error: NSError? = nil
        let capacity = Int(Double(buffer.frameCapacity) / sampleRateRatio)
        let bufferPCM16 = AVAudioPCMBuffer(pcmFormat: outputFormat, frameCapacity: AVAudioFrameCount(capacity))!
        converter.convert(to: bufferPCM16, error: &error) { inNumPackets, outStatus in
            outStatus.pointee = AVAudioConverterInputStatus.haveData
            return buffer
        }
    }

    engine.prepare()
    try! engine.start()

}

In this case I seem to have 2 options. Either solve problem on mic level, if possible make this code work with ".defaultToSpeaker". Or don't use category .playandrecord But switch between .playback and .record when mic is not needed. This didn't seem to be easy too, since it requires a lot of starting/stopping of all audio, which is necessary for activate and deactive AVAudioSession. But if this is the way to go I can provide more code.