InfluenceChat: What Failed While Building A Manipulation Dataset

Finding examples was the part of InfluenceChat that kept failing. We could retrieve conversations containing “guilt,” “pressure,” and “convince them.” Most of the results were fiction, ordinary persuasion, or people talking about their own feelings.

I wanted examples of a more specific behavior: someone asking an LLM to help manipulate another person. A request can be polite and still aim to hide information, manufacture authority, or make someone feel unable to refuse. Existing harassment or hate-speech labels don't necessarily capture that distinction.

These are the results and project status from June 3, 2026.

What we were trying to label

The working taxonomy had nine manipulation categories and a benign control:

The distinction I cared about was the other person's ability to understand and choose. Asking for a refund can be forceful without being manipulative. Inventing a competing offer or obscuring an opt-out changes the information the other person has available.

That distinction is clear in the examples above because the request states the intent. The public conversation data was much less cooperative.

Keywords found the vocabulary

We searched about 535,000 user turns from WildChat and LMSYS-Chat-1M using terms such as “make them feel,” “pressure,” “guilt,” and “convince them.” The search returned 7,891 rows.

Almost all of those candidates belonged to something else: roleplay, fiction, therapy-adjacent venting, coding questions, or ordinary persuasion. “Convince” found D&D scenes. “Guilt” found character writing. The words were present, but the behavior we wanted to study usually wasn't.

Those terms were useful leads, but poor selectors for the dataset we wanted.

Embeddings found related subjects

Sentence-transformer retrieval produced 1,686 candidates. A Qwen3:30b reviewer judged 94 of them manipulative, giving this retrieval run a precision of about 5.6 percent.

The recorded manual audit agreed with 98 of 99 reviewer judgments. That was encouraging for the small sample, but it left the larger labeling question open: we still needed independent review across the eventual dataset, including difficult benign cases.

We also wrote 90 exemplars, embedded them, and searched a 50,000-candidate sample. The neighboring results still included jailbreak discussion, harmless fiction, Excel VBA, vocabulary tests, and speech therapy prompts.

Those results made the problem more concrete. Semantic resemblance was enough to find a conversation about a related subject. It often wasn't enough to establish what the user intended to do to another person.

The context problem

“Help me write this message” tells us very little on its own. The message could be a reasonable complaint, an attempt to intimidate someone, or part of a fictional scene. The surrounding conversation may explain it, but sometimes the necessary facts aren't in the record at all.

My working explanation was that intent depended on context the retrieval query couldn't supply. The results supported spending more effort on that problem. They didn't give us a reliable estimate of how common manipulative requests were in the full corpus.

I stopped treating these retrieval runs as a practical primary source for the stream we were building. Ninety-four candidates were useful to inspect, but manually sorting that much unrelated material wasn't getting us close to the dataset we needed.

Why synthetic pairs were next

The next plan was to use the hand-written exemplars as context for generating candidates, screen those candidates, and independently audit the labels. Benign paired examples would be generated separately and reviewed alongside the manipulative ones.

I wanted pairs that stayed close in subject and wording. A model shouldn't be able to label a request manipulative just because every harmful example sounds theatrical and every benign one sounds like a customer-service template.

Synthetic data makes intent easier to specify. It also makes style shortcuts easier to introduce. Generation prompts, source provenance, held-out examples, and independent human review would therefore be part of the dataset work, rather than something to add after a model scored well.

Where the project stood

The next useful test was whether a reviewer could explain a label from the context actually supplied, then make the same distinction on examples from another source. Until that worked, generating more rows would mostly give us more material to argue about.