TLDR: I’ve tested 17 LLMs against 4 typed programming languages to see which excels at what.

The biggest takeaways are that there is a huge gap between the same LLM performing well in different languages. Take Sonnet 3.5 for example:

Python: 81% success rate
Typescript: 81% success rate
Swift: 71% success rate
Rust: 62% success rate

For the details of how this was measured, read on.

Intro

I have lately been thinking about the role of LLMs in software development. I use LLMs every day during my work and I couldn’t fathom doing my job as a software engineer without them. However, I do work with multiple different programming languages and I have noticed that the quality of the LLM’s output varies greatly between languages. This was just a feeling I had, not grounded in any empirical evidence. So I decided to do something about it.

The Idea

There’re different amounts of source code out there. Github publishes the state of the Octoverse every year, and while they don’t show the amount of code per language, they show distinct users contributing code per language and we can take that as an indicator for the amount of code out there.

By that metric, there is far more Python and JavaScript code than there is Rust or Swift code. As LLMs are being trained on code, and become better with more data, we can assume that they’re better at the languages with more code.

In addition, some languages - such as Rust - have a high complexity stemming from a trait based system where the correctness of the code depends not just on the currently visible code (e.g. a function), but a potpourri of other parts of the codebase. I assume that this is more difficult to reason about for a LLM.

With that in mind, I set out to compare four typed languages across multiple LLMs.

The Language x LLM Benchmark

I implemented a small Python project together with a database of programming problems. Then, I iterated over a list of LLMs and asked them to solve the problems. The outcome of this can then be used to answer the following questions:

See which LLM is the best for a given programming language
See which programming language is the best for ai-supported programming

Particular the second point is something I’d like to stress:

LLMs might enforce a language gap

If LLMs are far better at solving problems in a certain language, developers might be incentivized to use that language. This might lead to a language gap, where certain languages are used more often than others. There is a chance that future, better, LLMs will support inferiorely supported languages better, but by then the languages that are already supported very well will be supported even better.

The tasks

I’m keeping the database of LLM tasks private for now do decrease the likelihood that LLMs are being trained on this data. Each task consists out of the prompt as well as a description for how to evaluate the solution:

Check: I’m just running a type check on the code.
Eval: The code is run with a given input, and I’m expecting a given output.
Contains: I’m just checking if the code contains a given string (or multiple strings).
Run: The code should implement a server and I’m doing http requests to it.

I’ve included the Contains task to also test for the knowledge that a LLM has about the given ecosystem. E.g. how do I do the following in this third party library?

The tasks are split up into general tasks, which can be solved by any language, and language specific tasks which are only solvable in a certain language. One example is a task that requires the solution of a complex Rust lifetime issue. This is only solvable in Rust.

The set of specific tasks exist in order to answer question 2: Which LLM is the best for a given programming language?

An example of a general task is the following:


Can you write a {{lang}} function that recieves
a {{string_type}} as a parameter and returns the
same {{string_type}} uppercased?

In this case, {{lang}} is a placeholder for the programming language, {{string_type}} is a placeholder for the type of the string.

I would love to support morge languages, but these are the ones I’m most familiar with.

Each task is being executed three times to see how stable the results are for the given case (this is important as LLMs are not deterministic).

The contenders

In order to simplify the whole process, I’m using OpenRouter to access the LLMs. This allows me to use the same API for all LLMs.

amazon/nova-pro-v1
anthropic/claude-3.5-haiku-20241022:beta
anthropic/claude-3.5-sonnet:beta
deepseek/deepseek-chat
deepseek/deepseek-r1
deepseek/deepseek-r1-distill-llama-70b
meta-llama/llama-3.3-70b-instruct
microsoft/phi-4
mistralai/codestral-2501
mistralai/mistral-large-2411
mistralai/mistral-small-24b-instruct-2501
openai/gpt-4o-2024-11-20
openai/gpt-4o-mini
openai/o3-mini
qwen/qwen-max
qwen/qwen-plus
qwen/qwen-turbo
qwen/qwq-32b-preview

Results

Now the most interesting part, the results. I will try to answer various questions based on the data, including the two original questions:

See which LLM is the best for a given programming language
See which programming language is the best for ai-supported programming

Compare languages across LLMs

Name	Python	Rust	Typescript	Swift
amazon/nova-pro-v1	76.2%	35.7%	40.5%	42.9%
anthropic/claude-3.5-haiku-20241022:beta	76.2%	45.2%	71.4%	61.9%
anthropic/claude-3.5-sonnet:beta	81.0%	61.9%	81.0%	71.4%
deepseek/deepseek-chat	92.9%	61.9%	88.1%	64.3%
deepseek/deepseek-r1	69.0%	54.8%	59.5%	52.4%
deepseek/deepseek-r1-distill-llama-70b	61.9%	38.1%	45.2%	28.6%
meta-llama/llama-3.3-70b-instruct	73.8%	33.3%	57.1%	35.7%
microsoft/phi-4	64.3%	26.2%	50.0%	40.5%
mistralai/codestral-2501	73.8%	59.5%	71.4%	66.7%
mistralai/mistral-large-2411	78.6%	50.0%	69.0%	64.3%
mistralai/mistral-small-24b-instruct-2501	64.3%	21.4%	31.0%	28.6%
openai/gpt-4o-mini	71.4%	54.8%	71.4%	61.9%
openai/gpt-4o-2024-11-20	78.6%	45.2%	64.3%	59.5%
openai/o3-mini	97.6%	76.2%	95.2%	92.9%
qwen/qwen-max	85.7%	59.5%	54.8%	38.1%
qwen/qwen-plus	38.1%	42.9%	16.7%	47.6%
qwen/qwen-turbo	76.2%	45.2%	61.9%	35.7%
qwen/qwq-32b-preview	61.9%	23.8%	38.1%	38.1%

Here you can see the success rate of each LLM for each language. o3-mini is really good at Swift. What I find particularly interesting, is that some much cheaper LLMs perform almost as good as more expensive ones. Lets hone in on this by taking costs and duration into account.

Compare costs and duration across LLMs and languages

Name	Python	Python Duration	Python Costs	Rust	Rust Duration	Rust Costs	Typescript	Typescript Duration	Typescript Costs	Swift	Swift Duration	Swift Costs
amazon/nova-pro-v1	76.2%	78.9s	$0.0224	35.7%	115.9s	$0.0333	40.5%	106.1s	$0.0282	42.9%	165.3s	$0.0392
anthropic/claude-3.5-haiku-20241022:beta	76.2%	135.1s	$0.0373	45.2%	179.4s	$0.0489	71.4%	169.2s	$0.0450	61.9%	199.9s	$0.0571
anthropic/claude-3.5-sonnet:beta	81.0%	109.6s	$0.1215	61.9%	156.6s	$0.1792	81.0%	139.8s	$0.1539	71.4%	180.9s	$0.2109
deepseek/deepseek-chat	92.9%	1130.1s	$0.0087	61.9%	1289.4s	$0.0124	88.1%	1069.6s	$0.0103	64.3%	1953.8s	$0.0130
deepseek/deepseek-r1	69.0%	11049.6s	$0.1499	54.8%	15493.3s	$0.1795	59.5%	12264.8s	$0.1599	52.4%	14094.4s	$0.1594
deepseek/deepseek-r1-distill-llama-70b	61.9%	1501.1s	$0.0209	38.1%	2021.6s	$0.0237	45.2%	1292.6s	$0.0225	28.6%	2005.8s	$0.0323
meta-llama/llama-3.3-70b-instruct	73.8%	367.7s	$0.0024	33.3%	436.0s	$0.0033	57.1%	339.4s	$0.0027	35.7%	550.8s	$0.0039
microsoft/phi-4	64.3%	183.9s	$0.0015	26.2%	325.8s	$0.0025	50.0%	192.7s	$0.0017	40.5%	342.9s	$0.0023
mistralai/codestral-2501	73.8%	37.0s	$0.0063	59.5%	55.9s	$0.0094	71.4%	54.1s	$0.0077	66.7%	88.1s	$0.0108
mistralai/mistral-large-2411	78.6%	111.6s	$0.0512	50.0%	188.3s	$0.0811	69.0%	158.3s	$0.0661	64.3%	238.7s	$0.0888
mistralai/mistral-small-24b-instruct-2501	64.3%	55.0s	$0.0023	21.4%	74.5s	$0.0036	31.0%	74.3s	$0.0030	28.6%	106.5s	$0.0041
openai/gpt-4o-mini	71.4%	83.7s	$0.0696	54.8%	133.8s	$0.1071	71.4%	120.5s	$0.0919	61.9%	190.7s	$0.1287
openai/gpt-4o-2024-11-20	78.6%	85.2s	$0.0033	45.2%	127.5s	$0.0052	64.3%	116.1s	$0.0043	59.5%	216.5s	$0.0067
openai/o3-mini	97.6%	486.8s	$0.1960	76.2%	410.3s	$0.2757	95.2%	443.1s	$0.1905	92.9%	364.7s	$0.2395
qwen/qwen-max	85.7%	138.4s	$0.0367	59.5%	249.3s	$0.0611	54.8%	157.3s	$0.0371	38.1%	267.4s	$0.0579
qwen/qwen-plus	38.1%	95.8s	$0.0052	42.9%	216.8s	$0.0119	16.7%	80.9s	$0.0035	47.6%	284.5s	$0.0128
qwen/qwen-turbo	76.2%	100.3s	$0.0013	45.2%	161.8s	$0.0022	61.9%	131.2s	$0.0017	35.7%	193.3s	$0.0022
qwen/qwq-32b-preview	61.9%	1496.0s	$0.0108	23.8%	1401.2s	$0.0106	38.1%	1439.4s	$0.0101	38.1%	1514.7s	$0.0118

The big surprise here is just how slow the deepseek models are. These values are real. I ran the whole suite 3 times, so it took deepseek r1 more than 9 hours to complete. Compared to 110 seconds for Claude Sonnet or 487 seconds for o3 mini.

Stability

Another thing I tested for is the stability of the results. If I ask the same question, how often do I get a correct answer? So the whole benchmark suite ran 3 times. The next table shows the successful runs per language.

R1 means first run
R2 means second run
R3 means third run

Name	Python	Python R1	Python R2	Python R3	Rust	Rust R1	Rust R2	Rust R3	Typescript	Typescript R1	Typescript R2	Typescript R3	Swift	Swift R1	Swift R2	Swift R3
amazon/nova-pro-v1	76.2%	14	14	16	35.7%	9	7	11	40.5%	5	7	8	42.9%	6	8	10
anthropic/claude-3.5-haiku-20241022:beta	76.2%	15	14	15	45.2%	12	12	11	71.4%	13	12	11	61.9%	10	12	12
anthropic/claude-3.5-sonnet:beta	81.0%	17	15	16	61.9%	13	13	14	81.0%	11	14	13	71.4%	14	15	14
deepseek/deepseek-chat	92.9%	17	18	19	61.9%	16	15	12	88.1%	14	14	15	64.3%	11	11	13
deepseek/deepseek-r1	69.0%	14	16	14	54.8%	13	12	15	59.5%	9	10	12	52.4%	13	8	11
deepseek/deepseek-r1-distill-llama-70b	61.9%	13	13	12	38.1%	10	10	10	45.2%	7	6	9	28.6%	4	7	7
meta-llama/llama-3.3-70b-instruct	73.8%	16	16	11	33.3%	12	12	9	57.1%	10	9	9	35.7%	5	9	7
microsoft/phi-4	64.3%	14	13	12	26.2%	9	7	10	50.0%	7	7	9	40.5%	7	5	7
mistralai/codestral-2501	73.8%	14	14	14	59.5%	13	13	12	71.4%	11	10	14	66.7%	11	9	12
mistralai/mistral-large-2411	78.6%	15	15	15	50.0%	13	15	11	69.0%	11	9	11	64.3%	11	12	10
mistralai/mistral-small-24b-instruct-2501	64.3%	10	13	14	21.4%	6	8	7	31.0%	8	4	4	28.6%	4	6	7
openai/gpt-4o-mini	71.4%	14	14	15	54.8%	13	16	12	71.4%	10	10	12	61.9%	12	11	11
openai/gpt-4o-2024-11-20	78.6%	16	15	14	45.2%	9	12	13	64.3%	10	11	11	59.5%	9	9	11
openai/o3-mini	97.6%	19	19	18	76.2%	19	18	20	95.2%	15	14	15	92.9%	16	15	17
qwen/qwen-max	85.7%	17	16	15	59.5%	17	14	12	54.8%	1	13	12	38.1%	10	9	5
qwen/qwen-plus	38.1%	6	16	6	42.9%	14	12	12	16.7%	3	3	3	47.6%	9	10	7
qwen/qwen-turbo	76.2%	16	14	14	45.2%	12	11	12	61.9%	10	10	7	35.7%	4	6	6
qwen/qwq-32b-preview	61.9%	12	13	14	23.8%	6	8	5	38.1%	4	7	7	38.1%	7	5	6

Best language for LLMs

Back to the original question. Which language is best for LLMs? To answer that, we’re going beyond the general success rate and will look at the specific success rate which includes language-specific questions about the ecosystem and tricky type errors.

“Python (Gen)” means “Generic” questions, whereas “Python” means “Language specific” questions.

Name	Python	Python (Gen)	Rust	Rust (Gen)	Typescript	Typescript (Gen)	Swift	Swift (Gen)
amazon/nova-pro-v1	69.8%	76.2%	31.0%	35.7%	33.3%	40.5%	34.8%	42.9%
anthropic/claude-3.5-haiku-20241022:beta	69.8%	76.2%	40.2%	45.2%	60.0%	71.4%	49.3%	61.9%
anthropic/claude-3.5-sonnet:beta	76.2%	81.0%	46.0%	61.9%	63.3%	81.0%	62.3%	71.4%
deepseek/deepseek-chat	85.7%	92.9%	49.4%	61.9%	71.7%	88.1%	50.7%	64.3%
deepseek/deepseek-r1	69.8%	69.0%	46.0%	54.8%	51.7%	59.5%	46.4%	52.4%
deepseek/deepseek-r1-distill-llama-70b	60.3%	61.9%	34.5%	38.1%	36.7%	45.2%	26.1%	28.6%
meta-llama/llama-3.3-70b-instruct	68.3%	73.8%	37.9%	33.3%	46.7%	57.1%	30.4%	35.7%
microsoft/phi-4	61.9%	64.3%	29.9%	26.2%	38.3%	50.0%	27.5%	40.5%
mistralai/codestral-2501	66.7%	73.8%	43.7%	59.5%	58.3%	71.4%	46.4%	66.7%
mistralai/mistral-large-2411	71.4%	78.6%	44.8%	50.0%	51.7%	69.0%	47.8%	64.3%
mistralai/mistral-small-24b-instruct-2501	58.7%	64.3%	24.1%	21.4%	26.7%	31.0%	24.6%	28.6%
openai/gpt-4o-mini	68.3%	71.4%	47.1%	54.8%	53.3%	71.4%	49.3%	61.9%
openai/gpt-4o-2024-11-20	71.4%	78.6%	39.1%	45.2%	53.3%	64.3%	42.0%	59.5%
openai/o3-mini	88.9%	97.6%	65.5%	76.2%	73.3%	95.2%	69.6%	92.9%
qwen/qwen-max	76.2%	85.7%	49.4%	59.5%	43.3%	54.8%	34.8%	38.1%
qwen/qwen-plus	44.4%	38.1%	43.7%	42.9%	15.0%	16.7%	37.7%	47.6%
qwen/qwen-turbo	69.8%	76.2%	40.2%	45.2%	45.0%	61.9%	23.2%	35.7%
qwen/qwq-32b-preview	61.9%	61.9%	21.8%	23.8%	30.0%	38.1%	26.1%	38.1%

Here’s the current languages sorted by the success rate of the highest performing LLM (always o3 mini).

Python: 89%
Typescript: 73%
Swift: 69%
Rust: 65%

A couple of other observations

Qwen-Max is as good at Python as Sonnet but costs only 30%. The duration is also only slightly slower
Deepseek-chat is really good and really cheap but also really slow.
Deepseek-r1 is far worse than o3-mini but only slightly cheaper (and so unbelievably slow)
o3-mini is the best in all categories.

All Data

Here you can see all data in one huge table.

Name	Python	Python (Gen)	Python Duration	Python Costs	Python Quality	Rust	Rust (Gen)	Rust Duration	Rust Costs	Rust Quality	Typescript	Typescript (Gen)	Typescript Duration	Typescript Costs	Typescript Quality	Swift	Swift (Gen)	Swift Duration	Swift Costs	Swift Quality	Swift R1	Swift R2	Swift R3	Rust R1	Rust R2	Rust R3	Typescript R1	Typescript R2	Typescript R3	Python R1	Python R2	Python R3
amazon/nova-pro-v1	69.8%	76.2%	78.9s	$0.0224	68.0%	31.0%	35.7%	115.9s	$0.0333	30.0%	33.3%	40.5%	106.1s	$0.0282	30.0%	34.8%	42.9%	165.3s	$0.0392	31.1%	6	8	10	9	7	11	5	7	8	14	14	16
anthropic/claude-3.5-haiku-20241022:beta	69.8%	76.2%	135.1s	$0.0373	70.1%	40.2%	45.2%	179.4s	$0.0489	40.9%	60.0%	71.4%	169.2s	$0.0450	62.1%	49.3%	61.9%	199.9s	$0.0571	47.2%	10	12	12	12	12	11	13	12	11	15	14	15
anthropic/claude-3.5-sonnet:beta	76.2%	81.0%	109.6s	$0.1215	77.6%	46.0%	61.9%	156.6s	$0.1792	45.3%	63.3%	81.0%	139.8s	$0.1539	60.7%	62.3%	71.4%	180.9s	$0.2109	62.1%	14	15	14	13	13	14	11	14	13	17	15	16
deepseek/deepseek-chat	85.7%	92.9%	1130.1s	$0.0087	83.7%	49.4%	61.9%	1289.4s	$0.0124	52.2%	71.7%	88.1%	1069.6s	$0.0103	70.7%	50.7%	64.3%	1953.8s	$0.0130	49.1%	11	11	13	16	15	12	14	14	15	17	18	19
deepseek/deepseek-r1	69.8%	69.0%	11049.6s	$0.1499	69.4%	46.0%	54.8%	15493.3s	$0.1795	44.8%	51.7%	59.5%	12264.8s	$0.1599	48.6%	46.4%	52.4%	14094.4s	$0.1594	49.1%	13	8	11	13	12	15	9	10	12	14	16	14
deepseek/deepseek-r1-distill-llama-70b	60.3%	61.9%	1501.1s	$0.0209	61.2%	34.5%	38.1%	2021.6s	$0.0237	34.5%	36.7%	45.2%	1292.6s	$0.0225	35.0%	26.1%	28.6%	2005.8s	$0.0323	23.0%	4	7	7	10	10	10	7	6	9	13	13	12
meta-llama/llama-3.3-70b-instruct	68.3%	73.8%	367.7s	$0.0024	72.8%	37.9%	33.3%	436.0s	$0.0033	39.9%	46.7%	57.1%	339.4s	$0.0027	47.9%	30.4%	35.7%	550.8s	$0.0039	28.0%	5	9	7	12	12	9	10	9	9	16	16	11
microsoft/phi-4	61.9%	64.3%	183.9s	$0.0015	63.9%	29.9%	26.2%	325.8s	$0.0025	29.6%	38.3%	50.0%	192.7s	$0.0017	36.4%	27.5%	40.5%	342.9s	$0.0023	28.0%	7	5	7	9	7	10	7	7	9	14	13	12
mistralai/codestral-2501	66.7%	73.8%	37.0s	$0.0063	66.7%	43.7%	59.5%	55.9s	$0.0094	44.3%	58.3%	71.4%	54.1s	$0.0077	55.7%	46.4%	66.7%	88.1s	$0.0108	46.0%	11	9	12	13	13	12	11	10	14	14	14	14
mistralai/mistral-large-2411	71.4%	78.6%	111.6s	$0.0512	71.4%	44.8%	50.0%	188.3s	$0.0811	45.8%	51.7%	69.0%	158.3s	$0.0661	52.1%	47.8%	64.3%	238.7s	$0.0888	48.4%	11	12	10	13	15	11	11	9	11	15	15	15
mistralai/mistral-small-24b-instruct-2501	58.7%	64.3%	55.0s	$0.0023	54.4%	24.1%	21.4%	74.5s	$0.0036	23.2%	26.7%	31.0%	74.3s	$0.0030	31.4%	24.6%	28.6%	106.5s	$0.0041	21.7%	4	6	7	6	8	7	8	4	4	10	13	14
openai/gpt-4o-mini	68.3%	71.4%	83.7s	$0.0696	67.3%	47.1%	54.8%	133.8s	$0.1071	47.3%	53.3%	71.4%	120.5s	$0.0919	51.4%	49.3%	61.9%	190.7s	$0.1287	50.3%	12	11	11	13	16	12	10	10	12	14	14	15
openai/gpt-4o-2024-11-20	71.4%	78.6%	85.2s	$0.0033	73.5%	39.1%	45.2%	127.5s	$0.0052	36.0%	53.3%	64.3%	116.1s	$0.0043	52.1%	42.0%	59.5%	216.5s	$0.0067	40.4%	9	9	11	9	12	13	10	11	11	16	15	14
openai/o3-mini	88.9%	97.6%	486.8s	$0.1960	89.8%	65.5%	76.2%	410.3s	$0.2757	65.0%	73.3%	95.2%	443.1s	$0.1905	73.6%	69.6%	92.9%	364.7s	$0.2395	68.9%	16	15	17	19	18	20	15	14	15	19	19	18
qwen/qwen-max	76.2%	85.7%	138.4s	$0.0367	78.2%	49.4%	59.5%	249.3s	$0.0611	53.2%	43.3%	54.8%	157.3s	$0.0371	30.0%	34.8%	38.1%	267.4s	$0.0579	39.1%	10	9	5	17	14	12	1	13	12	17	16	15
qwen/qwen-plus	44.4%	38.1%	95.8s	$0.0052	42.2%	43.7%	42.9%	216.8s	$0.0119	45.3%	15.0%	16.7%	80.9s	$0.0035	15.0%	37.7%	47.6%	284.5s	$0.0128	39.1%	9	10	7	14	12	12	3	3	3	6	16	6
qwen/qwen-turbo	69.8%	76.2%	100.3s	$0.0013	72.1%	40.2%	45.2%	161.8s	$0.0022	40.4%	45.0%	61.9%	131.2s	$0.0017	47.9%	23.2%	35.7%	193.3s	$0.0022	21.1%	4	6	6	12	11	12	10	10	7	16	14	14
qwen/qwq-32b-preview	61.9%	61.9%	1496.0s	$0.0108	59.9%	21.8%	23.8%	1401.2s	$0.0106	22.2%	30.0%	38.1%	1439.4s	$0.0101	26.4%	26.1%	38.1%	1514.7s	$0.0118	27.3%	7	5	6	6	8	5	4	7	7	12	13	14

Adding more languages

If you’re curious about adding a language, let me know on GitHub.. Note that the actual tasks are kept private to prevent LLMs training on them.