Google Admin SDK: Limited impersonation for service accounts

Question

I'm building a simple Java application where I need to retrieve the list of groups the user is subscribed to in my G Suite domain. I have gone through a bunch of answers on Stackoverflow, but am still a bit confused for the usage.

There is https://developers.google.com/admin-sdk/directory/v1/guides/manage-groups#get_all_member_groups but I can't seem to get it to work without a service account and impersonating an admin/

There is https://developers.google.com/admin-sdk/directory/v1/guides/manage-users#retrieve_users_non_admin, but it doesn't give me groups.

I have everything working with a service account, but it looks like once I do that, I can impersonate pretty much any admin under my account. That seems a little scary to me. Although I can limit the capabilities of the service account to just read the list of users and groups, I would like to go further and restrict the impersonation capabilities to only a user that has a custom role I created with just readonly privileges.

Is this the right way (service account with admin impersonation) or is there an easier way to do this ?