Tokens count

[MindsightsAI]

How to determine tokens usage ?

[joaomdmoura]

You could use callbacks now, but I want to add a better way to report on it adding this as feature accepted

[bishwarupdas]

@joaomdmoura Any update on this feature? Need this.

btw, Awesome job!

[anandhuh1234]

crew = Crew(
agents=[agent1, agent2],
tasks=[task1, task2],.....

print(crew.usage_metrics)

[jackhayter]

I believe I found a bug in the Agent class's set_agent_executor method where the callbacks are added, which make this token count observation possible. I found that the TokenCalcHandler was being called 3 separate times for each interaction with the LLM when only one was expected. This was causing the usage to be over-reported.

By ensuring that only a single instance of TokenCalcHandler can be appended to the callbacks of the LLM, I was able to verify that only a single usage callback was triggered. While my understanding of the internals of this library may be limited, I believe that this fix may be useful. If someone can verify the hypothesis or current behaviour, I'd be happy to open a PR.

I've opened a PR for this, but would like some help verifying this hypothesis please. Thanks everyone!

Here's the modified method:

@model_validator(mode="after")
def set_agent_executor(self) -> "Agent":
    """set agent executor is set."""
    if hasattr(self.llm, "model_name"):
        token_handler = TokenCalcHandler(self.llm.model_name, self._token_process)

        # Ensure self.llm.callbacks is a list
        if not isinstance(self.llm.callbacks, list):
            self.llm.callbacks = []

        # Check if an instance of TokenCalcHandler already exists in the list
        if not any(isinstance(handler, TokenCalcHandler) for handler in self.llm.callbacks):
            self.llm.callbacks.append(token_handler)

    if not self.agent_executor:
        if not self.cache_handler:
            self.cache_handler = CacheHandler()
        self.set_cache_handler(self.cache_handler)
    return self

[kinthaiofficial]

Token counting in multi-agent systems is trickier than it looks — the per-agent view misses the real cost driver, which is how context compounds across delegation chains.

A pattern we found useful: three-level quota hierarchy (namespace → user → agent → conversation) with pessimistic pre-allocation. Before any LLM call, deduct the ceiling; credit back the unused portion after. This gives you hard budget floors without requiring accurate upfront estimation.

The other thing that helped: model routing by task type rather than using one model for everything. We settled on roughly 58% Haiku-class (fast retrieval, classification, simple tool calls), 31% mid-tier (reasoning + synthesis), 11% full reasoning model (planning, delegation decisions). Blended cost comes out around $3.20/M tokens vs ~$8/M if you run everything on the top model.

There's a write-up on the economic model and cost attribution here: https://blog.kinthai.ai/agent-wallet-economic-models-autonomous-agents — covers how to make cost visible to the agent itself so it can make routing decisions rather than just burning budget.

What breakdown are you seeing in your token counts — is it context carry-over or actual generation cost that's dominating?

[kinthaiofficial]

Token counting in CrewAI is important for two related but different purposes: cost visibility (how much did this run cost?) and budget enforcement (stop before this run costs more than $X).

For cost visibility:
CrewAI's underlying LLM calls should return usage metadata from the API (input_tokens + output_tokens + cache_hit_tokens if applicable). The tricky part is rolling this up across the entire crew run — you need to sum across all agents and all their tool calls.

A simple pattern: wrap each agent's LLM calls with a token counter that accumulates into a Crew.run_stats object, then expose it after crew.kickoff() returns.

For budget enforcement:
This needs to be pessimistic — allocate against a budget ceiling before the LLM call, not after. Otherwise in a fan-out scenario (one coordinator spawning 10 agents in parallel), you'll overshoot because all 10 are checking "is there budget?" simultaneously before any of them have consumed any.

# Pseudocode
def guarded_llm_call(agent, prompt, max_tokens, budget_remaining):
    if max_tokens > budget_remaining:
        raise BudgetExhausted(f"Need {max_tokens}, have {budget_remaining}")
    # Deduct ceiling pessimistically
    budget_remaining -= max_tokens
    result = llm.call(prompt, max_tokens=max_tokens)
    # Credit back unused
    actual_used = result.usage.total_tokens
    budget_remaining += (max_tokens - actual_used)
    return result

Cache-hit tokens matter:
If you're using prompt caching (Anthropic/OpenAI both support this), cache hits cost ~10% of normal input tokens. Your token counter should distinguish between cache-hit and cache-miss tokens for accurate cost attribution.

What's the scale you're trying to track — single task runs or long-running agent deployments?