how to achieve multi-user isolation for kubeflow pipelines

Question

I recently updated kubeflow version to 1.3 but this does not seem to work for API authentication.

When I call kubeflow endpoint from my springboot service, it gives below error

nested exception is org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 Internal Server Error: [{"error":"Internal error: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.\nFailed to authorize with API resource refer... (10791 bytes)]] with root cause

After going through documentation, I found we can add AuthorizationPolicy to support request.headers so I did and now the error thrown is

[Request processing failed; nested exception is org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden: [RBAC: access denied]] with root cause

Also I can see in v1.3 documentation, there is limitation for In-cluster API request authentication

Does anybody know how to achieve this?

Answer 1

I found a way to achieve this by passing the kubeflow auth service session token as a header
-H "Cookie: authservice_session=$token

Answer 2

there is a workaround can achieve the kfp API authentication under multi-user mode.

https://github.com/kubeflow/kfctl/issues/140#issuecomment-719894529

basically this approach is to retrieve the authservice_session by an additional post request, and then put the authservice_session to the kfp client headers.

Answer 3

In case you still looking for the answer, here is what I did to solve the problem:

1. Apply AuthorizationPolicy (just like you said)

please refer here for more detail

2. create new notebook server with the new option enabled.

There should be a new configure options for you to enable, you have to tick it (not selected by default).

the new pod that notebook server is running should have a file contains token file.

However, it is not injected to your request automatically, you have to read it explicitly.

3. use the token

like this:

token_file = os.getenv("KF_PIPELINES_SA_TOKEN_PATH")
with open(token_file) as f:
    token = f.readline()
client = kfp.Client(host='http://ml-pipeline-ui.kubeflow/pipeline',
               existing_token=token)