Disable .htaccess redirect for internal links does not work - Is caching the problem?

Question

Assume a french webpage is available under https://example.com/ and an english version under https://example.com/en.

If https://example.com/ is visited directly (enter the address manually or by some external link) the visitor should be redirected automatically to https://example.com/en if fr is not included in the Accept-Language header.

However, when calling https://example.com/ from some internal link, there should be NO automatic redirect. Otherwise it would not be possible to manually switch from /en to the french main page.

I tried so solve this using the following code in .htaccess

# Check if NOT an internal link. Since it is not possible use environment
# vars on both sides at the same time, construct an string by combining
# HOST + REFERER and then checking if the first part (the host) is contained
# in the second part (the referer)
RewriteCond %{HTTP_HOST}@@%{HTTP_REFERER} !^([^@]*)@@https?://\1/.*

# Apply to main page only. Do not inlude subpages.
RewriteCond %{REQUEST_URI} ^/$ [NC]

# Redirect if FR is not accepted
RewriteCond %{HTTP:Accept-Language} !fr [NC]

# Redirect to english main page
RewriteRule ^$ /en [L,R]

The first time I tried this, it worked as expected. However, once I call https://example.com/ manually and I am redirected to /en it does not work anymore.

Why is this?

Is it possible that there is some caching involved which simply check "Ok, last call to / was redirected to /en, so just redirect the next calls there as well without checking .htaccess again"? Or is there any other problem with these rules?

The page is running on a shared Server where I cannot access any config files. Is it possible to solve this in .htaccess only?