RFC: Improving the diagnosis of fibers backpressing

[rorororom]

Reviewers

• Main Reviewer: @drewdzzz
• Second Reviewer: @nshy
• Third Reviewer: @Totktonada
• Team Lead: @sergepetrenko
• CTO: @sergos

Tickets

core: fibers engine backpressure #10169

Goal

Past discussion of this tickets

In the initial implementation, it was decided not to set a limit on the number of fibers.

Instead of imposing a strict limit on the number of fibers, which can lead to a violation of acceptable workloads, we propose to simplify the failure path by issuing a single diagnostic error with wide formatting.

Thanks to this change, a single error log line allows you to instantly get a clear and accurate error.

Diagnostic Information

1) Process-wide memory stats

Field	Description
vmem	Total virtual memory in bytes (Linux: first field of /proc/self/statm; macOS: task_info.virtual_size).
rss	Resident Set Size in bytes.
vmem_maps	(Linux only) Current number of VM mappings (/proc/self/maps).
vmem_maps_max	(Linux only) System limit of VM mappings (/proc/sys/vm/max_map_count).

2) Fiber statistics

Field	Description
total	Total number of fibers (alive + dead).
running	Number of alive fibers not in a wait state.
suspended	Number of alive fibers currently waiting.
dead	Number of fibers in the dead pool (ready for reuse).
csw	Total context switches since process start.
stack_allocated	Bytes reserved for all fiber stacks (stack_size * total).
stack_used	Bytes actually used by all fiber stacks.
region_allocated	Bytes allocated via the region allocator (region_total()).
region_used	Bytes currently in use by the region allocator (region_used()).
mem_bookkeeping	Memory overhead for fiber structs & metadata (replacement for vmem_structs).
pool.iproto.total	Total fibers in the “iproto” pool.
pool.iproto.running	Running fibers in the “iproto” pool.
pool.iproto.suspended	Suspended fibers in the “iproto” pool.
pool.iproto.idle	Idle fibers in the “iproto” pool (suspended with no work).
pool.tx.total	Total fibers in the “tx” pool.
pool.tx.running	Running fibers in the “tx” pool.
pool.tx.suspended	Suspended fibers in the “tx” pool.
pool.tx.idle	Idle fibers in the “tx” pool (suspended with no work).

Proposed Lua API

-- 1) Process-wide memory stats
local proc = box.process.stat()
-- returns:
{
    vmem           = <number>,
    rss            = <number>,
    vmem_maps      = <number>,   -- Linux only
    vmem_maps_max  = <number>,   -- Linux only
}

-- 2) Fiber statistics
local fs = fiber.stat()
-- returns:
{
    total            = <number>,
    running          = <number>,
    suspended        = <number>,
    dead             = <number>,
    csw              = <number>,
    stack_allocated  = <number>,
    stack_used       = <number>,
    region_allocated = <number>,
    region_used      = <number>,
    mem_bookkeeping  = <number>,
    pool = {
        iproto = {
            total     = <number>,
            running   = <number>,
            suspended = <number>,
            idle      = <number>,
        },
        tx = {
            total     = <number>,
            running   = <number>,
            suspended = <number>,
            idle      = <number>,
        },
    },
}

Error message

When fiber distribution fails, emit a single concise log line:

error: Fiber distribution failed — run box.process.stat() and fiber.stat() for diagnostic details.

[Totktonada]

I like the idea in general. Please, consider my remarks below.

1. VMA term

proc_vma
vma_maps/max_map_count
vma_stacks
vma_structs
vma_other

The vma term is used in different meanings. And, TBH, I don't understand why 'total virtual memory' is abbreviated as VMA.

2. Bytes/pages

Total virtual memory (VMA) in kB (from /proc/self/statm).

All the configuration parameters and metrics in tarantool that are about a size of data are in bytes (not kilo-, mega-, giga-, etc).

3. API

I would like to discuss an API. I guess that the general statistics about a process will be in some <something>.stat() call and the fiber related statistics in fiber.stat(). Let's propose some variant and we'll start to discuss.

4. vma_other

Field	Description
vma_other	Remaining VMA not counted by stacks or structs.

At first, it seems to be about a memory (amount of bytes), not VMAs. I wrote the next points in this context.

I would like to know, what exactly is counted in vma_other. An informal description is OK, but it should describe certain things. There are two reasons why I would make it more explicit:

• There are things that are related to a fiber from a user point of view, but I guess that they're not in this proposed metric. For example, a memory for Lua objects which are created and lost during a request that is served by this fiber.
• Now vma_stacks + vma_structs + vma_other means some 'all fiber related memory'. Imagine that we add one more metric in a future, for example guard_page_size. Should we count it in vma_other as well? The given description of vma_other means that we should. But this makes these metrics non-orthogonal.

Maybe it worth to track <...>_total instead of <...>_other.

5. fibers.*

At first, it seems that we use two pretty orthogonal property sets here. The first one is a status: total, running, suspended, pooled. I thnk we shouldn't mix it with other properties. And I would rename pooled to something else, because now it looks like it is related to some of the given two pools above. <fiber object>:status() reports it as dead, so I would use this word here (if it fits here -- please, verify my words).

tarantool> fiber.stat().by_status
- total: 42
  running: 1
  suspended: 40
  dead: 1

Next, the second property set is from where (from which pool) the fiber comes.

tarantool> fiber.stat().by_pool
- total: 42
  tx_pool: 0
  iproto_pool: 20

How reusable fits here? Is it total - tx_pool - iproto_pool? Or there are reusable and non-reusable fibers? I would like to have some kind of Euler diagram here (how these sets are intersected).

Next, system/non-system fits bad here. But since amount of system fibers is likely limited by some constent like 5 or 10, I think we can just ignore this property.

Also, it is interesting, how many active fibers are in each pool. So we possibly want some more flexible API to filter and group the given properties.

tarantool> fiber.stat({
         >     filter = {'status', '!=', 'dead'},
         >     group_by = 'pool'})
- total: 41
  tx_pool: 0
  iproto_pool: 10

[rorororom]

1. You are right, avoid overloading “VMA” with two meanings, then I suggest renaming everything to vmem instead of vma(except vma_maps), as if it would make it clearer what is meant by the size of the virtual memory, not the number of areas.
2. Thanks, I'll keep that in mind.
3. See the updated text of the RFC
4. I think it's better to just remove the other field.
5. Yes, you have proposed a very successful and understandable structure.

fiber object>:status() reports it as dead, so I would use this word here (if it fits here -- please, verify my words).

You are right, it is more correct to call this field dead, just like the fiber object in :status()`

I would like to have some kind of Euler diagram here (how these sets are intersected).

As I understand it, nothing intersects.

                             total   
-----------------------------------------------------------------
|                                                               |
|         ---------------      ---------------                  |
|         | iproto_pool |      | tx_pool     |                  |  
|         ---------------      ---------------                  |
|                                                               |
|         non_reusable = (total − iproto_pool − tx_pool)        |
-----------------------------------------------------------------

it might not even be worth outputting non_reusable, everyone can figure it out for themselves)

[nshy]

1. AFAIU opts argument is not used currently so let's not mention it.

local fs = fiber.stat([opts])

2. AFAIU fiber.stat() should return a map like below:

{
    by_status = {
        total     = 200,
        running   = 180,
        suspended = 10,
        dead      = 10,
    },
    by_pool = {
        total         = 200,
        iproto_pool   =   8,
        tx_pool       =   2,
    },
    memory = {
        stacks_bytes    = 104857000,
        structs_bytes   =    8192,
    },
    -- fs.metrics we are just pulling up existing counters from the metrics module
    metrics = {
        amount          = ...,
        memused         = ...,
        memalloc        = ...,
        csw             = ...,
    },
}

I'd prefer to group stats semantically and not by source (like metrics) or filter-like names (like by-pool):

{
    total = 200,
    running = 100,
    suspended = 100,
    dead = 1,
    csw = 23513515,
    stack_allocated = 12513531451,
    stack_used = 1234514,
    region_allocated = 123451243,
    region_used = 3251324,
    mem_bookkeeping = 33242143,
    pool = {
        iproto  = {
            total = 10,
            running = 2,
            suspended = 2,
            idle = 2,
        },
        tx  = {
            total = 9,
            running = 2,
            suspended = 2,
            idle = 3,
        },
    },
}

In more details:

• let's not use _bytes suffixes as we don't use it in other stats
• let's add detailed info for pools (instead of adding filtering to stats to get detailed information for pool)
• add idle stat for pool. Idle fiber is suspended but because of there is no tasks to do
• region_used and region_allocated corresponds to region functions region_used() and region_total()
• mem_bookkeeping instead of structs_bytes. The latter is too technical name IMHO

Previous metrics.memused and metrics.memalloc should be covered in more details by stacks_*, region_* and mem_bookkeeping.

3. Similarly for box.process.stat().
   Instead of

{
    vmem = {
        total_bytes = <number>,
        rss_bytes = <number>,
        vma_count = <number>,
        max_map_count = <number>,
    },
}

I'd prefer:

{
    vmem = 1234124,
    rss = 12342143,
    vmem_maps = 12341234,
    vmem_maps_max = 70870897,
}

It is different from linux names but more consitent. Also I'd not sure we need extra nesting vmem here.

[rorororom]

Thanks for the feedback — I took into account all the comments (removed opts, flat semantics, renamed fields and divided tables), the RFC has been updated.

[drewdzzz]

1. User can look at vmem_maps, compare it with vmem_maps_max and find out that the instance has run out of VMAs. But what can user do with rss and vmem metrics? I don't see how they improve diagnostics.
2. Are we talking only about the TX thread in terms of fiber statistics?
3. "Running fibers" - only one fiber can be running at a time (actually, the one obtaining statistics). Is there any point to add it to the statistics?
4. fiber.stack_used - how are you going to measure amount of bytes actually used by the stacks?
5. region_allocated/region_used - do we need these statistics here at all? I bet 99% of users don't even know what region is. Also, we have other allocators, for example, mempool, that use the same slabs as the regions, so I don't see how these fields can be useful for diagnostics.
6. What is mem_bookkeeping? Just sizeof(struct fiber) * total? A weird name, TBO.

[nshy]

1. and 5. I guess some info like this is added to give a bigger picture and stats completeness. Not for the particular case of exhausted VMAs.
2. Other threads has only system fibers AFAIK. So other threads fibers stats will help developer more rather then user I guess, in case of issues in Tarantool itself.
3. True. Yet I would name fiber that do fiber_sleep(0), call or start another fiber running too. They just give other fibers opportunity to run and not wait for some condition. But it is not a matter of this RFC.
4. I guess it is possible. We can dig for stack pointer of any fiber.
5. Yes. And yes name is weird but I did not found a better one. structs is too C-bound I guess.

[drewdzzz]

I guess it was a mistake to write a comment with several points because they share the same thread now 😄

1. Hm, OK, they can be useful in general. Let's leave them.
2. Yeah, I agree - I would account TX thread only (partially, just to make it simple, without need of multithreaded synchronization). Let's just mention it in the RFC.
3. So, do you think we should leave them in statistics keeping in mind that definition of "running fibers" can be extended in the future?
4. Hm, good idea. Let's leave this entry.
5. Still, does region_used show anything to a user? It's just internal allocator, bound to a fiber. Also, we would need to iterate over all fibers to collect this statistic since region_alloc doesn't report the allocation to the fiber subsystem. And it sounds expensive - statistics should be obtained in a constant amount of time because a monitoring demon can read it very often.
6. Isn't it better to export the whole fiber_mempool statistics then? IMHO, amount of occupied bytes doesn't show much.

fiber.stats()
-- returns
{
    total = 2,
    suspended = 1,
    ...
    mempool = {
        mem_free: 16232
        mem_used: 48
        item_count: 2
        item_size: 24
        slab_count: 1
        slab_size: 16384
    }
}

[Totktonada]

I think that a fiber region statistic may be useful. It is not so hard to explain in the documentation: we can just say that it is a memory that is used by tarantool for temporary fiber-local objects.

[nshy]

3. After thinking a bit I'd keep 'running' in stats. This way no one have questions why total = suspended + dead + 1).
4. Currently we need to iterate fibers to get at least some other stats. If we need to make fiber stats that fast we can later update stats somewhere in cord on every fiber switch/creation/dead I guess, it should not give much overhead.
5. Let's skip details here and leave only mem_used and mem_allocated to be on par with other stats. Without extra mempool layer.

[drewdzzz]

You say:

Instead of imposing a strict limit on the number of fibers, which can lead to a violation of acceptable workloads, we propose to simplify the failure path by issuing a single diagnostic error with wide formatting.
Thanks to this change, a single error log line allows you to instantly get a clear and accurate error.

And later you show Lua API (box.process.stat() and fiber.stat()). So what path do we choose?

1. A huge error with all these fields.
2. A simple error referring to the new Lua API.

If the second path is chosen, please show the new error message.

[rorororom]

I suggest choosing the second, simplified version. The error will look like this:

error: Fiber distribution failed — run box.process.stat() and fiber.stat() for diagnostic details.

[drewdzzz]

When it will be thrown? Only when mptrotect has failed, or when allocation fails as well?

[rorororom]

It is logical to throw a diagnostic error in both cases — when mprotect() fails and when memory allocation fails. In both situations, a fiber cannot be created, and from the user's point of view, this is a single fiber distribution error.

[Totktonada]

I'm OK with the proposal. Thank you for working on it!

If you're going to elaborate it further, please, add a changelog section into the document. It helps to recap the discussion after a delay.