Using AWS cognito id token instead of access token

Question

After login, AWS cognito provides access token and id token. In the backend I was wondering if I can use ID token instead of access token for authorization. Some blogs suggests that ID token should not be passed to the server. However aws jwt verifier provides option to verify ID token signature. I need some values that are provided in ID token like custom fields. Is there any security issue in using ID token ?