Skip to content

Bug in emergency mode capability #446

@AVSBharadwaj

Description

@AVSBharadwaj

Background

This bug was observed in mercari's platform when we released the recent emergency mode capability of tortoise into our infrastructure. The emergency mode PR is this .

It was observed for cases where tortoise was in auto mode and the scaling policies in tortoise were all vertical something like

status:
  autoscalingPolicy:
  - policy:
      cpu: Vertical
      memory: Vertical
    containerName: app
  - policy:
      cpu: Vertical
      memory: Vertical
    containerName: istio-proxy

In such cases, we observed tortoise going to Emergency phase as soon as the next reconcile happened after the deployment of emergency mode feature.

Reason

The bug was at the line of hte isHpaMetricAvailable function which would return always false for tortoises which had all vertical scaling and hence no hpa associated.

Action taken ( PR )

so to handle the above all-vertical cases, a check is added to the isHpaMetricAvailable method which will now return true in allvertical cases and it will continue to remain in its original phase and not enter emergency mode wrongly.

Future TODOs

add a emergency mode feature for the vertical scaling case as well i.e when the recommendations from teh VPA arent working as expected, then the tortoise must enter emergency mdoe.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions