CRMArena Leaderboard

CRMArena is a novel benchmark designed to assess LLM agents on realistic customer service tasks within professional environments. By working with CRM experts, CRMArena offers nine challenging tasks across three personas—service agent, analyst, and manager—populated within a simulated organization using 16 interrelated industrial objects. This benchmark invites the community to improve AI agent capabilities in function-calling and work task understanding, demonstrating tangible business value in a realistic Salesforce Org.

Agentic Frameworks

Function Calling ReAct Act

Model	Agentic Framework	NCR	HTU	TCU	NED	PVI	KQA	TII	MTA	BRI	Overall ⬆️
claude-3.5-sonnet	Function Calling	60.8	68.5	66.9	34.6	24.6	39.2	99.2	84.6	74.8	64.3

Model	Agentic Framework	NCR	HTU	TCU	NED	PVI	KQA	TII	MTA	BRI	Overall ⬆️
o1	Function Calling	60.8	68.5	66.9	60	24.6	39.2	99.2	84.6	74.8	64.3
o1	ReAct	70	51.5	54.6	34.6	30	58.8	81.5	75.4	63.1	57.7
gpt-4o	Function Calling	60	47.7	81.5	46.2	39.2	30.4	97.7	27.7	59.2	54.4
llama3.1-405b	Function Calling	16.2	31.5	64.6	50	26.9	47.6	95.4	86.9	42.3	51.3
claude-3.5-sonnet	Function Calling	4.6	33.1	82.3	52.3	30	40.5	69.2	26.9	36.9	41.8
llama3.1-70b	Function Calling	1.5	23.1	44.6	53.8	37.4	42.4	93.8	43.8	29.2	41.1
gpt-4o	ReAct	70	39.2	22.3	30.8	35.4	50.2	64.6	20.9	10.8	38.2
claude-3.5-sonnet	Act	78.5	24.6	15.4	51.5	28.5	44.7	45.4	20.8	26.9	37.4
deepseek-r1	ReAct	53.8	23.1	30.1	40.8	34.6	61.2	46.9	3.1	22.3	35.1
claude-3.5-sonnet	ReAct	62.9	20	11.5	52.3	30	45	43.9	20.8	21.5	34.3
llama3.1-405b	ReAct	81.5	22.3	15.4	33.9	34.6	55.3	34.6	13.9	13.1	33.8
gpt-4o	Act	43.1	10	17.7	30.8	28.5	29.3	68.5	29.2	7.7	29.4
gpt-4o-mini	ReAct	40.8	36.9	25.4	31.5	24.6	52.8	30	6.2	6.2	28.3
llama3.1-70b	ReAct	48.5	20	13.9	33.1	37.7	48.7	23.9	13.9	10.8	27.8
llama3.1-405b	Act	46.2	17.7	17.7	13.9	30	47	15.4	5.4	6.9	22.2
gpt-4o-mini	Function Calling	0.8	10.8	10.8	17.7	13.8	39.7	60	0	21.5	19.5
llama3.1-70b	Act	28.5	20	24.6	6.2	30	47.9	8.5	0	1.5	18.6
claude-3-sonnet	ReAct	7.7	24.6	26.9	29.2	28.5	16	22.3	0.8	0	17.3
gpt-4o-mini	Act	0.8	38.5	23.8	9.2	0	43.1	26.9	3.8	3.8	16.7
claude-3-sonnet	Act	9.2	26.9	24.6	30.8	23.8	16.6	16.2	1.5	0	16.6
claude-3-sonnet	Function Calling	0.8	1.5	30	25.4	41.5	23.2	12.3	1.5	0	15.1
deepseek-r1	Function Calling	0.8	0.8	2.3	0.8	24.6	34.6	0	13.8	3.1	9
llama3.1-8b	ReAct	0	0	1.5	6.2	15.4	4	0	0	0.8	3.1
llama3.1-8b	Function Calling	0	0	0	0	0	0	0	0	0	0

Model	Agentic Framework	NCR	HTU	TCU	NED	PVI	KQA	TII	MTA	BRI	Overall ⬆️
claude-3.5-sonnet	Function Calling	60.8	68.5	66.9	34.6	24.6	39.2	99.2	84.6	74.8	64.3

Model	Agentic Framework	NCR	HTU	TCU	NED	PVI	KQA	TII	MTA	BRI	Overall ⬆️
o1	Function Calling	60.8	68.5	66.9	60	24.6	39.2	99.2	84.6	74.8	64.3
o1	ReAct	70	51.5	54.6	34.6	30	58.8	81.5	75.4	63.1	57.7
gpt-4o	Function Calling	60	47.7	81.5	46.2	39.2	30.4	97.7	27.7	59.2	54.4
llama3.1-405b	Function Calling	16.2	31.5	64.6	50	26.9	47.6	95.4	86.9	42.3	51.3
claude-3.5-sonnet	Function Calling	4.6	33.1	82.3	52.3	30	40.5	69.2	26.9	36.9	41.8
llama3.1-70b	Function Calling	1.5	23.1	44.6	53.8	37.4	42.4	93.8	43.8	29.2	41.1
gpt-4o	ReAct	70	39.2	22.3	30.8	35.4	50.2	64.6	20.9	10.8	38.2
claude-3.5-sonnet	Act	78.5	24.6	15.4	51.5	28.5	44.7	45.4	20.8	26.9	37.4
deepseek-r1	ReAct	53.8	23.1	30.1	40.8	34.6	61.2	46.9	3.1	22.3	35.1
claude-3.5-sonnet	ReAct	62.9	20	11.5	52.3	30	45	43.9	20.8	21.5	34.3
llama3.1-405b	ReAct	81.5	22.3	15.4	33.9	34.6	55.3	34.6	13.9	13.1	33.8
gpt-4o	Act	43.1	10	17.7	30.8	28.5	29.3	68.5	29.2	7.7	29.4
gpt-4o-mini	ReAct	40.8	36.9	25.4	31.5	24.6	52.8	30	6.2	6.2	28.3
llama3.1-70b	ReAct	48.5	20	13.9	33.1	37.7	48.7	23.9	13.9	10.8	27.8
llama3.1-405b	Act	46.2	17.7	17.7	13.9	30	47	15.4	5.4	6.9	22.2
gpt-4o-mini	Function Calling	0.8	10.8	10.8	17.7	13.8	39.7	60	0	21.5	19.5
llama3.1-70b	Act	28.5	20	24.6	6.2	30	47.9	8.5	0	1.5	18.6
claude-3-sonnet	ReAct	7.7	24.6	26.9	29.2	28.5	16	22.3	0.8	0	17.3
gpt-4o-mini	Act	0.8	38.5	23.8	9.2	0	43.1	26.9	3.8	3.8	16.7
claude-3-sonnet	Act	9.2	26.9	24.6	30.8	23.8	16.6	16.2	1.5	0	16.6
claude-3-sonnet	Function Calling	0.8	1.5	30	25.4	41.5	23.2	12.3	1.5	0	15.1
deepseek-r1	Function Calling	0.8	0.8	2.3	0.8	24.6	34.6	0	13.8	3.1	9
llama3.1-8b	ReAct	0	0	1.5	6.2	15.4	4	0	0	0.8	3.1
llama3.1-8b	Function Calling	0	0	0	0	0	0	0	0	0	0

Overview

Customer Relationship Management (CRM) systems are vital for modern enterprises, providing a foundation for managing customer interactions and data. Integrating AI agents into CRM systems can automate routine processes and enhance personalized service. However, deploying and evaluating these agents is challenging due to the lack of realistic benchmarks that reflect the complexity of real-world CRM tasks. To address this issue, we introduce CRMArena, a novel benchmark designed to evaluate AI agents on realistic tasks grounded in professional work environments. We worked with CRM experts to design nine customer service tasks distributed across three personas: service agent, analyst, and manager. We synthesize a large-scale simulated organization, populating 16 commonly-used industrial objects (e.g., account, order, knowledge article, case) with high interconnectivity, and upload it into a real Salesforce CRM organization. UI and API access to the CRM is provided to systems that attempt to complete the tasks in CRMArena. Experimental results reveal that state-of-the-art LLM agents succeed in less than 40% of the tasks with ReAct prompting and less than 55% even when provided manually-crafted function-calling tools. Our findings highlight the need for enhanced agent capabilities in function-calling and rule-following to be deployed in real-world work environments. CRMArena is an open challenge to the community: systems that can reliably complete tasks showcase direct business value in a popular work environment.

Task Definitions

Task	Description
New Case Routing (NCR)	Assign the best human agent to an incoming case based on case subject and description to optimize performance metrics. Involves matching cases to agents using case histories, skills, and availability.
Handle Time Understanding (HTU)	Identify the agent with the shortest/longest average handle time based on case history data, evaluating the LLM agent's ability to analyze performance data accurately.
Transfer Count Understanding (TCU)	Determine which human agent transferred cases the least/most over a given period, assessing the LLM agent's capacity to analyze transfer performance accurately.
Name Entity Disambiguation (NED)	Disambiguate named entities related to customer transactions, focusing on product names. Identify specific orders corresponding to product names within a given timeframe.
Policy Violation Identification (PVI)	Determine if company policies have been breached in a case involving customer-agent interaction by comparing case details against policy rules in knowledge articles.
Knowledge Question Answering (KQA)	Answer specific questions based on knowledge articles, demonstrating the LLM agent's ability to retrieve accurate and relevant information from a CRM knowledge repository.
Top Issue Identification (TII)	Identify the most reported issue for a particular product based on case history, assessing the ability to analyze issue reports for trend analysis.
Monthly Trend Analysis (MTA)	Determine which months have the highest number of cases for a given product and timeframe, demonstrating the LLM agent's ability to recognize trends and patterns over time.
Best Region Identification (BRI)	Identify the regions where cases are closed the fastest by analyzing case closure times across various regions to indicate top-performing regions.