Distributed lock with TTL

Question

When we have a distributed lock with TTL, it is possible that lock will expire because of TTL config and the process which had that lock has not finished computation and it will continue to manipulate the object for which it acquired lock as it doesn't know that lock has already expired. How can we avoid that scenario?

Answer 1

The solution you are looking for is called "fencing token". Long story short - every mutation command/operation should include the token and the executor should check if the token is still valid.

The token is just a number, call it "term", every time a new lock is issued, the term get increased.

The executor has a simple logic, it never accepts commands with an old term.

Arguably, this is the only option to truly avoid any lock related issues. Stuff like having timestamps, or explicit lock releases - all of them are inherently prone to various race issues.

Another pointer I recommend to look at - Red Lock algorithm; and the issues it has - more on this here: https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html