A correct routing label can lose part of the request

Both reviewers can have read the message correctly

“I cannot log in and I was charged twice” can quite reasonably receive an access label. That gets the message to the first queue, but it does not get the double charge handled. Two reviewers may disagree because one is choosing who leads and the other is recording everything the customer needs. Before changing the classifier, I would ask which of those jobs its label is meant to do.

A single category is perfectly capable of naming a lead queue. It cannot also describe two issues unless something else carries the second one forward. That makes the apparent annotation disagreement useful: it may be showing a missing agreement between classification and the work created afterwards. Making the reviewers choose the same answer could improve a score while leaving the customer’s other problem just as absent.

scikit-learn’s documentation distinguishes multiclass problems, where a sample has one class, from multilabel problems, where more than one label can apply. [1] The terminology is useful, but choosing a representation is a business decision before it becomes an estimator setting. The software needs to preserve what the team intends to do for the person, not merely produce a tidy target column for training.

Count the work inside the messages

The original teaching set contains six messages. M1 asks for an invoice copy; M2 cannot log in; M3 combines login trouble and a double charge; M4 asks to cancel and receive a refund; M5 asks to change a delivery address; M6 asks to cancel because the delivery address was wrong. Under the authored issue definitions, these contain nine issues across four types: access, billing, cancellation and delivery.

Under the explicit priority rule—cancellation before access, then billing, then delivery—M3 goes to access and M4 and M6 go to cancellation. If that single route is treated as the complete work description, billing disappears from M3 and M4 and delivery disappears from M6. The route choices can all satisfy their rule while omitting three relevant issues. The intended meaning of the label determines whether that is acceptable.

The downloadable comparison keeps the priority choices beside all nine authored issues. Read its omitted-issue list as well as the route counts: three messages contain work a primary category does not express. That is a property of this teaching representation, not an observed classifier error rate. If the real assignment is only to choose the lead queue, the next check belongs at the handoff: can its owner still find and progress the other issues? The same labels can serve that narrower job without pretending to describe the whole request.

One lead route can still work well when its owner takes responsibility for the whole case. Call the label lead queue, and let the full issue list or original message travel with it. The evaluation can then check that routing job without advertising it as recognition of every required action. Put the same meaning in the dataset instructions, interface and team agreement, or three sensible interpretations of one field can recreate the disagreement downstream.

More labels preserve information and create another job

A multilabel representation can retain access plus billing for M3 and cancellation plus billing for M4. That avoids the artificial need to choose which true issue exists. It does not decide who coordinates the response, whether the tasks can run independently or what order their effects require. Adding labels repairs one information loss while making the coordination question visible.

M4’s cancellation and refund need coordination even after both have been found. The refund may depend on cancellation state and the policy that applies. Two independent queue messages could lead one colleague to promise a refund while another sends a continuation notice. The customer would have received two competently categorised answers to one incoherently handled case. Keep an accountable parent case and the dependencies visible, with a shared reference where both owners can record decisions before the final reply is assembled.

There is also judgement in what constitutes an issue. In M6, the wrong address might be background explaining the cancellation, or it might be a separate request to correct the address if cancellation is unavailable. The literal wording should drive the annotation rule. For this exercise, delivery is retained as relevant context needing disposition, not a claim that every mention of an address should create another task. Some cases should ask for clarification instead of multiplying work automatically.

An issue-splitting system can create linked subtasks with evidence spans, proposed next actions and relationships to the original message. The nine issue records in this exercise still belong to six parent cases, with shared commitments and coordinated responses. More detected issues should not automatically mean more independent conversations. Repeated messages may also concern an existing unresolved issue, so task deduplication needs its own identity rules alongside issue coverage.

Change the evaluation unit deliberately

A single-label dataset asks whether the chosen route matches the expected route. An issue-level evaluation asks which issues were found, which were missed and which were invented. A workflow evaluation asks whether the resulting case reached the correct coordinated outcome. Those are different questions. One aggregate accuracy figure cannot carry all three meanings just because the same incoming message started them.

For the six-message exercise, inspect M3 through M6 in their full context. Did the system preserve the double charge while routing the login problem? Did it distinguish the request for a refund from a guarantee that one is due? Did the cancellation owner retain the address concern for the appropriate disposition? These checks focus on information and commitments, rather than awarding extra labels indiscriminately.

Revise the reference data when the intended task changes. Do not compare a new multilabel system against old single-label targets and call every second correct label a false positive. Equally, do not rewrite the historical targets solely to make the new system look successful. Record the new task definition, annotate the cases under it and retain enough history to explain why the scores before and after answer different questions.

A useful adjudication record says why reviewers disagreed. Was one following lead-route priority while the other captured all issues? Was the customer genuinely ambiguous? Did the taxonomy lack a category, or did a reviewer miss a sentence? These causes need different responses. More examples may help a missed pattern. A better definition is needed when two careful people were answering different legitimate questions.

Let disagreement improve the service design

Before changing a model, run the intended labels through the next operational step. Show an owner the case they would receive with the primary route alone, then with the issue list. Ask what action they can justify and what they would need to recover from the original text. It makes the cost of the representation concrete before training or deployment turns it into routine behaviour.

Do not charge every ambiguity to model uncertainty. A confident answer may follow a taxonomy that cannot express the whole request, while a hesitant answer may correctly expose two plausible intentions. The remedy could be a larger case representation or one clarifying question. Tuning a threshold until it selects a label more decisively would not recover the meaning that representation leaves out.

A team can choose one lead route and still handle multi-issue work well. It can also adopt multiple labels and handle it badly. What matters is the contract between classification and action: which information survives, who owns it and how related decisions remain consistent. That contract makes an evaluation useful because a correct result now corresponds to work the team actually intends to do.

Follow M3 into the access queue and ask the owner where the double charge went. If it remains visible and somebody is responsible for it, access can be the right lead route. If it has disappeared, agreement on that label is a poor acceptance test. The same exercise separates lead routing, issue coverage and coordinated resolution without making every case create several conversations. What matters is that the customer does not have to send the missing half again.