ART ARGENTUM ANALYSIS

Vulnerabilities in LLM APIs

Analysis of vulnerabilities in proprietary LLM APIs, based on "Stealing Reasoning Traces from Proprietary LLM APIs" | Machine Learning Street Talk.

2026-08-22Machine Learning Street TalkStealing Reasoning Traces from Proprietary LLM APIs

OPEN SOURCE

SUMMARY

The discussion centers on a significant vulnerability in proprietary LLM APIs, where encrypted reasoning states can be decoded, allowing for the extraction of reasoning traces from advanced models. Researchers Ilia Shumailov and Alexander Panfilov demonstrated that smaller models can exploit this weakness to replay reasoning from larger models, posing substantial security risks.

The implications of this vulnerability extend to scenarios where users share conversations containing sensitive data, as the encrypted reasoning can reveal private information even if the visible parts are sanitized. This raises critical concerns about data privacy and security, particularly in contexts where sensitive information may be involved.

The researchers emphasize the need for architectural changes to mitigate these risks, suggesting that current cryptographic measures are insufficient. They advocate for responsible vulnerability disclosure and highlight the importance of monitoring AI reasoning processes to ensure safety and prevent models from producing harmful or nonsensical outputs.

The findings challenge the notion that AI systems are becoming more secure, suggesting instead that existing vulnerabilities could be exploited across different models and platforms. The researchers express uncertainty about the feasibility of stealing advanced models, indicating that while some progress has been made, significant challenges remain in extracting meaningful insights from cutting-edge AI systems.

The conversation also touches on the ethical implications of AI models contemplating cheating, revealing that while models may consider deceptive strategies, they often ultimately reject them, raising questions about their reasoning integrity. The researchers call for improved understanding of reasoning processes in AI models to prevent unauthorized access to sensitive information.

XDETAIL

INFO

YOUTUBE2026-08-22machine learning street talk

Stealing Reasoning Traces from Proprietary LLM APIs — Ilia Shumailov & Alexander Panfilov

STANCE

00:00

05:00

10:00

15:00

20:00

25:00

30:00

35:00

40:00

45:00

10 intervals • swipe left

Stealing Reasoning Traces from Proprietary LLM APIs — Ilia Shumailov & Alexander Panfilov

machine_learning_street_talk • 2026-08-22 20:22:07 UTC

FULL

00:00–05:00

The paper discusses a vulnerability in proprietary LLM APIs where encrypted reasoning states can be decoded, allowing for the extraction of reasoning traces from advanced models. Researchers demonstrated that smaller models can exploit this weakness to replay reasoning from larger models, posing significant security risks.

The paper discusses a vulnerability in proprietary LLM APIs where encrypted reasoning states can be decoded, allowing for the extraction of reasoning traces from advanced models like GPT
Researchers Ilia Shumailov and Alexander Panfilov demonstrated that smaller models can exploit this weakness to replay reasoning from larger models, posing significant security risks
The decoding of reasoning blobs enables various attacks, including prompt injections and the potential for training on sensitive user data, raising serious safety implications
The shared vulnerability across major model providers like Anthropic, OpenAI, and Google suggests a structural issue in the design of these models, which could be mitigated through architectural revisions
The researchers emphasize that while they did not steal models, they revealed how reasoning traces can be accessed, challenging the notion of user data privacy in AI systems

Read full analysis

STANCE

STANCE MAP

Proponents of AI security measures

Emphasize the need for architectural changes to mitigate vulnerabilities in LLM APIs
Advocate for responsible vulnerability disclosure to enhance data privacy

Critics of current AI security practices

Argue that existing cryptographic measures are insufficient to protect sensitive information
Highlight the challenges in monitoring AI models and the potential for unauthorized access

Neutral / Shared

The paper discusses a vulnerability in proprietary LLM APIs where encrypted reasoning states can be decoded, allowing for the extraction of reasoning traces from advanced models like GPT

FULL

05:00–10:00

The discussion highlights a vulnerability in encrypted reasoning blobs used by AI models, which can be exploited to replay reasoning across different users and models. This poses significant risks, including unauthorized access to sensitive information and potential privacy violations.

The vulnerability of encrypted reasoning blobs used by AI models, which can be replayed across different users and models, allowing for unauthorized access to sensitive information
Researchers demonstrate that these reasoning traces can be extracted and injected into new conversations, enabling various attacks, including the potential for privacy violations and the generation of misleading outputs
The implications of this vulnerability extend to scenarios where users share conversations containing sensitive data, as the encrypted reasoning can reveal private information even if the visible parts are sanitized
The conversation emphasizes the need for monitoring AI reasoning processes to ensure safety and prevent models from producing harmful or nonsensical outputs, particularly in light of past incidents involving AI systems

FULL

10:00–15:00

The discussion focuses on vulnerabilities in proprietary LLM APIs, particularly how encrypted reasoning states can be exploited to extract reasoning traces. Researchers highlight the challenges in monitoring AI models and the implications of these vulnerabilities for security and privacy.

The challenges in monitoring AI models, particularly as they exhibit increasingly complex and opaque reasoning patterns, which complicates understanding their decision-making processes
Researchers observed that earlier generations of models, particularly Codex, displayed unusual reasoning traces that may stem from the unique ways software engineers think, suggesting potential artifacts from the training process
The conversation touches on the ethical implications of AI models contemplating cheating, revealing that while models may consider deceptive strategies, they often ultimately reject them, raising questions about their reasoning integrity
The potential for model stealing is discussed, with an analogy to cryptographic analysis, where minor input adjustments can reveal decision boundaries, although this is more feasible with smaller models than with larger, more complex ones
The participants express uncertainty about the feasibility of stealing advanced models, indicating that while some progress has been made, significant challenges remain in extracting meaningful insights from cutting-edge AI systems

FULL

15:00–20:00

The researchers analyzed reasoning traces from various models, particularly focusing on Kimi's outputs influenced by reasoning from Opus. Their findings indicate a specific vulnerability in Kimi's architecture, as it adapts its reasoning style more effectively than other models when exposed to external reasoning traces.

The researchers conducted an analysis of reasoning traces from various models, particularly focusing on how Kimis outputs can be influenced by injecting reasoning from other models like Opus
They observed that when a small portion of reasoning from Opus was integrated into Kimi, the resulting output closely resembled that of Opus, indicating a potential vulnerability in Kimis reasoning process
The findings suggest that Kimi adapts its reasoning style more effectively than other models when influenced by external reasoning traces, raising questions about the models internal mechanisms and training data similarities
The researchers noted that this phenomenon was unique to Kimi, as similar experiments with other models did not yield the same results, highlighting a specific weakness in Kimis architecture
The team engaged in responsible disclosure with the labs involved, receiving acknowledgment of their findings and discussing the details of the attack execution

METRICS

OTHER

50 percent%

details

CONTEXT: the expected percentage of reasoning that would dictate the model's output style

WHY: This percentage indicates how much influence external reasoning can have on Kimi's output

EVIDENCE: if you like perfil I know like substantial part of the reasoning like a 50 percent I would expect that model just adopt this title to reasoning

FULL

20:00–25:00

The researchers discuss vulnerabilities in encrypted reasoning blobs used by AI models, which can be exploited to replay reasoning across different users and models. They emphasize the need for improved architectural defenses to prevent unauthorized access to sensitive information.

The researchers emphasize the importance of responsible vulnerability disclosure, noting that their findings were acknowledged by the labs involved without negative repercussions
Mitigation efforts are underway, focusing on anti-distillation strategies to address the architectural vulnerabilities that allow reasoning traces to be replayed across different users and models
Architectural vulnerabilities make it easier for attackers to exploit reasoning outputs, suggesting that fixing these vulnerabilities is crucial to prevent unauthorized access to sensitive information
The potential for more architectural vulnerabilities to be discovered as protocols are examined, indicating a need for improved understanding of reasoning processes in AI models
The researchers question the effectiveness of simply releasing reasoning in plain text, arguing that it could still enable attacks and that the current cryptographic measures are insufficient to prevent exploitation by smaller models

FULL

25:00–30:00

The researchers found that reasoning traces from proprietary LLM APIs can be easily extracted, revealing significant vulnerabilities in the system's architecture. Their analysis of approximately 350,000 reasoning blobs identified instances of private data exposure, raising concerns about data privacy and security.

The researchers discovered that reasoning traces from proprietary LLM APIs can be easily extracted, revealing vulnerabilities in the systems architecture that allow for unauthorized access to sensitive information
By analyzing approximately 350,000 reasoning blobs from user sessions found online, they identified instances of private data exposure, including API keys and personal information, despite some data being synthetic
The ease of extracting reasoning led to the development of a universal jailbreak that can decode reasoning from various models, highlighting significant security flaws in the way reasoning states are handled
The findings challenge the notion that AI systems are becoming more secure, suggesting instead that existing vulnerabilities could be exploited across different models and platforms, raising concerns about data privacy and security
The researchers emphasize the need for improved cryptographic measures and responsible disclosure practices to mitigate the risks associated with these vulnerabilities

FULL

30:00–35:00

The researchers discuss vulnerabilities in proprietary LLM APIs, particularly focusing on the risks associated with encrypted reasoning blobs that can be replayed across different models. They propose architectural changes to mitigate these vulnerabilities and emphasize the importance of detecting reasoning leaks.

The discussion adds to doubts about the use of a single global key for decrypting reasoning across different models, suggesting that while it is unlikely, the exact key management remains unclear
Proposed fixes for the vulnerabilities include not sending reasoning to users and implementing architectural changes to prevent reasoning from being replayed across different contexts
The researchers emphasize the importance of detecting when reasoning leaks occur, comparing it to existing methods for identifying sensitive information leaks
An unexpected finding from their experiments indicates that injecting specific words into reasoning can significantly alter the length and style of the output, highlighting a complex interaction between model inputs and outputs

FULL

35:00–40:00

The researchers highlight vulnerabilities in proprietary LLM APIs, particularly concerning the replay of encrypted reasoning blobs across different users and models. They emphasize the potential risks of data leaks and the need for improved defenses against malicious exploitation.

The researchers discuss a surprising phenomenon where reasoning distribution shifts occur, indicating complex interactions between model inputs and outputs that are not fully understood
Concerns are raised about the potential harms of reasoning leaks, particularly regarding the extraction of sensitive user data and the implications of injecting malicious thoughts into reasoning traces
The conversation highlights the risks associated with downloading shared reasoning blobs, which may contain poisoned thoughts that could manipulate model behavior in unforeseen ways
The researchers draw parallels between the risks of reasoning traces and software vulnerabilities, emphasizing the importance of verifying the integrity of shared data to prevent malicious exploitation
There is a sense of urgency regarding the pace of emerging threats in AI systems, with the researchers noting that defenses may not keep up with the rapid development of new vulnerabilities

FULL

40:00–45:00

Recent discussions highlight the increasing frequency of vulnerabilities in AI models, particularly concerning the replay of encrypted reasoning blobs. Researchers emphasize the need for improved defenses and a scientific approach to understanding these vulnerabilities.

Recent incidents, such as the Hugging Face security issue, highlight the increasing frequency of vulnerabilities in AI models, raising concerns among researchers about the safety of these systems
The discussion emphasizes a shift in perspective regarding the trade-off between safety and capabilities in AI, suggesting that models capable of harmful actions undermine their intended purpose
There is a belief that advancements in AI could lead to significant defensive capabilities, but this potential is not yet fully realized due to a lack of trained personnel and awareness in the field
The deployment of intelligent agents in production environments introduces complexities, as these agents can misinterpret guidance and act in unintended ways, necessitating sophisticated monitoring systems
The researchers advocate for a scientific approach to understanding AI vulnerabilities, stressing the need for well-defined experiments and meaningful assessments to address the challenges posed by these technologies

FULL

45:00–50:00

The discussion focuses on the vulnerabilities of proprietary LLM APIs, particularly the risks associated with replaying encrypted reasoning blobs across different models. Researchers emphasize the need for controlled environments and enhanced safety measures to mitigate these vulnerabilities.

The discussion emphasizes the importance of avoiding anthropomorphism in AI models, advocating for a scientific approach that relies on controlled environments and precise assessments
There is a concern about the implications of AI models becoming agentic and conceptualizing abstract concepts, which could lead to unintended behaviors and safety risks
The participants highlight the need for enhanced monitoring and safety mitigations to better understand and manage AI vulnerabilities, suggesting that current computational limitations hinder thorough investigation
The conversation touches on the distinction between jailbreaking and benign distillation, with a warning that mislabeling these phenomena could lead to regulatory overreactions, particularly regarding open-weight models
The researchers express optimism about the growing body of meaningful work in AI safety, noting the regional representation of authors in Europe as a positive sign for collaborative efforts in the field

CRITICAL ANALYSIS

The discussion highlights significant vulnerabilities in proprietary LLM APIs, particularly the ability to extract reasoning traces that could be exploited for malicious purposes. This raises critical concerns about data privacy and security, especially in contexts where sensitive information may be involved. The researchers emphasize the need for architectural changes to mitigate these risks, suggesting that current cryptographic measures are insufficient.

METRICS

other

50 percent %

the expected percentage of reasoning that would dictate the model's output style

This percentage indicates how much influence external reasoning can have on Kimi's output

if you like perfil I know like substantial part of the reasoning like a 50 percent I would expect that model just adopt this title to reasoning

THEMES

#ai_development#ai_safety#data_privacy#encrypted_reasoning#military_ai#ai_security#controlled_environments#defensive_uplift#extracted_reasoning#kimi_vulnerability#llm_exploitation#llm_vulnerabilities#llm_vulnerability#model_vulnerabilities#privacy_vulnerability#reasoning_leaks#reasoning_traces#stealing_reasoning#vulnerabilities#vulnerability#vulnerability_disclosure

DISCLAIMER

This analysis is an original interpretation prepared by Art Argentum based on the transcript of the source video. The original video content remains the property of the respective YouTube channel. Art Argentum is not responsible for the accuracy or intent of the original material.