Plan adherence feedback function doesn't identify plan recorded in Root input

[Milanomac]

I am adding plan adherence feedback function to my agentic workflow where Plan is defined dynamically depending on the analyzed scenario. In all scenarios, plan that I want to use as the Plan for the feedback function is called Monitoring Strategy Report and is always recorded as the Record Input of the Record Root. I added custom_instructions to the feedback score explaining this, but still, everytime when I run my workflow the feedback score is always 0 and in the Trulens dashboard I always see this justification for it:
"Supporting Evidence: The trace contains only a list of spans (tool calls, LLM generations, searches, etc.) and does not include any section labeled PLAN or any inferred checklist of steps. According to the extraction procedure, when a plan cannot be found the evaluator must state that no plan is present. Without a plan, there is nothing for the agent to adhere to, so adherence is effectively zero."

I checked the class PlanAdherence defined under this link https://github.com/truera/trulens/blob/main/src/feedback/trulens/feedback/v2/feedback.py, and explored Trulens documentation but I wasn't able to find how to ensure that the evaluation function identifies the plan from my record input. Say I have this class below used for instrumentation of my agent, do I need to add attributes or some Span type there to help the evaluation function identify that plan is located there. In the exaple below, obs_input is the argument that gets logged as the Record Input and contains the Monitoring Strategu Report (my Plan).

class TruReActAgent:
    """Track spans for ReActAgent application."""

    def __init__(self, agent: ReActAgent):
        self.agent = agent

    @instrument
    async def get_response(
        self,
        obs_input: str,
        question: str,
        memory: Memory,
        thread_id: str | None = None,
    ) -> str:
        """Record the root."""
        result = await self.agent.run(question, memory=memory)
        return str(result)

[dosubot[bot]]

The PlanAdherence feedback function isn't detecting your plan because it expects the plan to be explicitly labeled in the trace, typically as a span attribute like RECORD_ROOT.INPUT on the root span. Just passing the plan as an argument (e.g., obs_input) isn't always enough for the evaluator to find it unless it's mapped to the correct span attribute.

To fix this, update your instrumentation so that the plan is stored as the root input in the trace. You can do this by using the @instrument decorator with span_type=SpanAttributes.SpanType.RECORD_ROOT and attributes={SpanAttributes.RECORD_ROOT.INPUT: "obs_input"}. Here’s how you can modify your TruReActAgent:

from trulens.core.otel.instrument import instrument
from trulens.otel.semconv.trace import SpanAttributes

class TruReActAgent:
    def __init__(self, agent: ReActAgent):
        self.agent = agent

    @instrument(
        span_type=SpanAttributes.SpanType.RECORD_ROOT,
        attributes={SpanAttributes.RECORD_ROOT.INPUT: "obs_input"}
    )
    async def get_response(
        self,
        obs_input: str,
        question: str,
        memory: Memory,
        thread_id: str | None = None,
    ) -> str:
        result = await self.agent.run(question, memory=memory)
        return str(result)

This ensures the plan (your "Monitoring Strategy Report") is stored in the trace as the root input, making it accessible to the PlanAdherence evaluator docs.

When configuring your feedback function, use a Selector that points to RECORD_ROOT.INPUT to extract the plan for evaluation. For example:

from trulens.core.feedback.selector import Selector
from trulens.otel.semconv.trace import SpanAttributes

plan_selector = Selector(
    span_type=SpanAttributes.SpanType.RECORD_ROOT,
    span_attribute=SpanAttributes.RECORD_ROOT.INPUT
)

If you make these changes and the plan is still not detected, double-check that the plan text is actually being passed as obs_input and that the decorator is applied to the correct method. The PlanAdherence evaluator is strict about trace structure and labeling, so this explicit mapping is necessary reference.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.