Implementing HA Active/Active with distributed file system

[saipradeepkumar]

Hi kevin,

We are trying to implement the HA active/active.as you mentioned here #562 (comment) we are able to handle the scenario3" logic to the kernel web handlers such that if a request comes in for an unknown kernel ID, we attempt to re-establish connections to that kernel using persisted state (if any)".we had tested one scenario by running 2 instances of enterprise gateway and launched 4 spark-python yarn cluster kernels.in that i killed killed one instance by using kill-9 process id .after that i checked my kernels out of them two are responding other two are not responding.if i do reconnect or close & open my notebook it is working.

I need your help in understanding how only two of them are working with out reconnect.

[kevin-bates]

@saipradeepkumar - this is really great news actually! Thank you for taking a deeper look into this.

Regarding the timing of things... It sounds like you ...

1. Started 2 EGs before any kernels were started.
2. Then you started 4 kernels.
3. Then killed one EG.
4. Then you issued subsequent requests for each of the 4 kernels and found 2 required reconnects while the other 2 did not.

Is that a correct reflection of the scenario?

Off the top of my head, I'm not sure. Have a couple questions...

Could the 2 kernels that didn't require reconnects been serviced by the "other" EG from the outset? I.e., are these behind an LB or reverse proxy of some sort?

Are each of the "subsequent requests" the same kind of thing?

Do the logs for the "other" EG indicate any kind of "auto-reconnect" entries happening at the time the "subsequent requests" are happening (or prior)?

Did you move the handlers from JKG to do this?

[saipradeepkumar]

@saipradeepkumar - this is really great news actually! Thank you for taking a deeper look into this.

Regarding the timing of things... It sounds like you ...

1. Started 2 EGs before any kernels were started.
2. Then you started 4 kernels.
3. Then killed one EG.
4. Then you issued subsequent requests for each of the 4 kernels and found 2 required reconnects while the other 2 did not.

Is that a correct reflection of the scenario?

Off the top of my head, I'm not sure. Have a couple questions...

Could the 2 kernels that didn't require reconnects been serviced by the "other" EG from the outset? I.e., are these behind an LB or reverse proxy of some sort?

Are each of the "subsequent requests" the same kind of thing?

Do the logs for the "other" EG indicate any kind of "auto-reconnect" entries happening at the time the "subsequent requests" are happening (or prior)?

Did you move the handlers from JKG to do this?

@kevin-bates thank you for your quick reply. your understanding is correct.
below i am attaching screenshot of how i am creating [HA]

I had moved the handlers from JKG and created enterprise_websocket module and changed the api module to the package enterprise_websocket
As explained in the image i am using ha proxy to route the traffic to the enterprise gateway instance using round robin algorithm.so kernel start can be on one instance and api request may be served by other instance of EG.whenever an instance of EG gets the request if the kernel is not present in that instance, it will try to refresh the session from the distributed store and start the kernel from session.
so subsequent request can be served by this instance.

so in testing scenario, as i explained i had created 4 kernels ,out of them one kernel got created on instance 1,Other three on instance 2 .i killed the instance 2 .now when i sent the request from the kernels only two of them are responding .
here what i am wondering is out of four kernels two kernels expecting reconnect command to make them active .But other two kernels one from instance 1 and one from instance 2 are working with reconnect command .

[kevin-bates]

Thank you for the picture - truly worth a thousand words! 😄

ok - so 3 of the four kernels originated on instance 1 (which was killed) and, following their subsequent requests - which were then directed to the "long-lived" instance 2, one of the three didn't require reconnection. Correct? Yeah, that seems odd to me.

Are you sure that 3 of the 4 were actually on instance 1 and not, 2 on each? That would explain everything if the latter and that should have been the case in a RR configuration.

When the instance encounters a kernel id that it hadn't seen before, do you simply call into the kernel-session manager to reload the session?

It would be great to somehow get the reconnect to happen automatically, but I think that needs to come from the client. So, the next best thing would be know if the client can determine a reconnect is necessary, so it could auto-enable that action.

However, having this working even with manual intervention is pretty cool!

[saipradeepkumar]

@kevin-bates

Thank you for the picture - truly worth a thousand words! 😄

ok - so 3 of the four kernels originated on instance 1 (which was killed) and, following their subsequent requests - which were then directed to the "long-lived" instance 2, one of the three didn't require reconnection. Correct? Yeah, that seems odd to me.

Are you sure that 3 of the 4 were actually on instance 1 and not, 2 on each? That would explain everything if the latter and that should have been the case in a RR configuration.

When the instance encounters a kernel id that it hadn't seen before, do you simply call into the kernel-session manager to reload the session?

It would be great to somehow get the reconnect to happen automatically, but I think that needs to come from the client. So, the next best thing would be know if the client can determine a reconnect is necessary, so it could auto-enable that action.

However, having this working even with manual intervention is pretty cool!

@kevin-bates
as you said When the instance encounters a kernel id that it hadn't seen before, i am refreshing the kernel session by calling _load_sessions after that i am calling start_kernel_from_session.
so that subsequent request are served

[kevin-bates]

Great. And you're sure that the original spread of kernels wasn't 2 and 2, but actually 3 and 1?

If so, I'm not sure why 1 of the 3 failed-over kernels didn't require a reconnect.

[saipradeepkumar]

Yeah i am sure because once the kernel has been started note book will start sending request for the status of kernel .by the time i create another kernel multiple request has been served by the two instances .in that case this scenario will happen.

[saipradeepkumar]

if i have to determine the reconnect required on the client side ,I have to implement the reconnect automatically in NB2KG.am i right?

[kevin-bates]

That's probably the right place to start.

Please note that NB2KG is now embedded in the notebook master branch - 6.0 release. So we'll need to "backport" any pure NB2KG changes. In addition, some of the handlers aren't present in the embedded version because the NB handlers simply call into the "substituted" kernel manager classes - that then forward the requests to the gateway.

[saipradeepkumar]

@kevin-bates Thanks a lot for the response ,i will look in to NB2KG module . if any changes were made to automatically detect the reconnect we will definitely backport .

[gss2002]

So does this support HA HDFS or no?

[kevin-bates]

I don't think HDFS is an immediate goal. Ideally we'd prefer a NoSQL DB to make the persisted kernel sessions accessible from other EG nodes. However, if HDFS is the sharing "vehicle" and we use a file-based persistence model, that should work as well. I'm not confident this answers your question though. If not, could you please be more specific how you see HDFS coming into play here?