Thought of sharing a relatively simple gist of Retry akka pattern I came up recently. It could be useful in case Client actor sends request to another Service provider actor and request may have to be retried due to different reasons (network failure in distributed environment, service down or other...). To increase resilience Client attempts to re-try operation until either response is received or after number of retries exceeded Client simply gives up.
The pattern is quite generic and I thought it would be cool to have a pluggable re-tries for akka actors.
The implementation assumes following:
1) Service provider actor suppose to respond to a Client actor. (it is not just fire and forget scenario)
2) Client attempts to send multiple requests: Each request is given an individual Timeout and requests are retried after Retry interval
3) Each request completes as Time out (service did not respond) or actual response message from Service provider. Time outs are simply logged. While response propagated to an original sender actor and Retry proxy is stopped.
4) In case of total failure - after retries exceeded exception sent to an original sender actor.
If you need to implement retries here is how you could do that.
Example 1. Existing code using tell:
worker ! someWorkCan be modified to include re-tries as:
context.actorOf(ReTry.props(tries = 2, retryTimeOut = 1000 millis, retryInterval = 100 millis, worker)) ! SomeWorkUsually response will be listened in receive loop
Exmaple 2. Or using akka ask pattern:
val futureResult = worker ? someWorkCould be modified to include re-tries as:
val futureResult = context.actorOf(ReTry.props(tries = 10, retryTimeOut = 1000 millis, retryInterval = 100 millis, worker)) ? SomeWork
Above will be re-tried 10 times with individual time out of 1 sec per try and with 100 milliseconds interval between tries.
Gist below:
And several test cases here...
Hi Jen,
ReplyDeletethanks for the pointer at letitcrash.com! There are a few issues which you might want to fix before we link to it. The biggest one is that you access the actor’s state from an onSuccess handler, which breaks actor encapsulation. You should replace both registered callbacks with pipeTo(self) and handling the resulting messages (which is Status.Failure in the onFailure case). The other issue is that you should make it clear that all of this only works if the actor is designed to reply. This should be obvious, but the waters are muddied by showing the fire-and-forget message send first.
Regards,
Roland
HI Roland,
DeleteFirst of all its great to hear from you! :) And thanks for pointing out the issue... I have made some changes and wrote couple of tests. Please let me know if I've left out anything else.
By the way thanks for all the great work you doing at TypeSafe and on Coursera - I truly enjoy all about Scala and technology stack! If there is anything I could do to help will be glad to get involved!
Thanks,
JenShen
Can I link another take on the same subject from my blog?
ReplyDeletehttp://blog.astrac.me/coding/2014/02/19/akka-actor-retry/
Sure please! Just wanted to highlight slight difference in my implementation: Each retry has individual time out. And tries/calls can overlap base on the individual timeout and retry intervals (two different params). Meaning we can execute first try and after retry interval while still waiting for the first response or timeout we can execute second try and so on depending on the number of retries. The time out for each try is defined as param during ReTry actor creation.
DeleteThanks,
JenShen