Why endorser_proposals_recieved (Hyperledger fabric peer-metric) is giving different values at different peers?

Question

We are using Hyperledger fabric 2.2.3 with 3 organizations 2 peers each, a total 6 endorsing peers,3 orderers, and using the Prometheus tool for monitoring.

using the endorser_proposals_recieved metric for calculating the transactions arrival rate at endorsing peer, when 10 transactions are issued to the network we observed the following values.

peer0-org1 : 2 transactions
peer1-org1 : 1 transaction
peer0-org2 : 3 transactions
peer1-org2 : 5 transactions
peer0-org3 : 6 transactions
peer1-org3 : 3 transactions

where broadcast_processed_count and ledger_transaction_count metrics are giving all 10 transactions at all peers and all transactions are successful, using majority endorsement policy.

we would like to understand how the endorser_proposals_recieved metric is giving values. we are using rate(endorser_proposals_recieved[1m]) for calculating arrival rate at endorser

Answer 1

It is worth understanding the Fabric transaction flow, documented here:

https://hyperledger-fabric.readthedocs.io/en/release-2.4/txflow.html

Your majority endorsement policy means that, for a transaction to successfully update the ledger, the client must obtain endorsements from at least two (of three) organisations. The client is free to send proposals to any (or all) available peers in order to receive (at least) enough matching endorsed proposal responses to meet this endorsement policy. Depending on the client implementation, it might be picking peers (and organisations) based on their block height, random selection, or some other strategy.

The key take-away is that you can reasonably expect proposals for different transactions to be spread across different organisations and different peers within those organisations. The distribution of proposals is not necessarily even across available organisations or peers.

Note that endorsement policies specified at the chaincode, collection and key level can further impact the distribution of proposals across organisations and peers.