---
title: "Red Flags bei Rechts-KI-Anbietern: Die 45-Punkte-Checkliste"
url: https://www.haqq.ai/de/blog/45-red-flags-legal-ai-vendor-evaluation
locale: de
type: blog
description: "45 Red Flags in 8 Kriterien - Sicherheit, Datenschutz, Robustheit, Kosten - vor jeder Rechts-KI-Vertragsunterschrift zu prüfen. Aus dem ersten käufergeführten Framework."
category: "guides"
keywords: "legal AI red flags, AI vendor evaluation, legal tech procurement, legal AI checklist, vibe procurement, AI due diligence, legal AI security, AI data privacy legal, legal tech adoption, AI vendor risk, legal AI buyer guide, legal AI evaluation framework, legal benchmarks, legal AI robustness, AI tool evaluation criteria"
published: 2026-04-11
updated: 2026-06-11
english_version: https://www.haqq.ai/blog/45-red-flags-legal-ai-vendor-evaluation
source: src/data/blogPosts.ts
generator: scripts/generate-page-markdown.ts
---

# Red Flags bei Rechts-KI-Anbietern: Die 45-Punkte-Checkliste

45 Red Flags in 8 Kriterien - Sicherheit, Datenschutz, Robustheit, Kosten - vor jeder Rechts-KI-Vertragsunterschrift zu prüfen. Aus dem ersten käufergeführten Framework.

> **Note:** From the Legal AI Evaluation Framework by Legal Benchmarks. 45 concrete warning signs across 8 evaluation criteria: the signals your legal team should spot in websites, demos, docs, and contracts before you buy any AI tool.

"Vibe procurement" is the legal tech industry's worst-kept secret. A polished demo, a few buzzwords, and a charismatic sales rep, and suddenly your firm has committed to a six-figure contract for an AI tool that nobody actually evaluated properly.

## Key facts

- 45 concrete red flags across 8 evaluation criteria, extracted from the legal industry's first buyer-led AI evaluation framework (legalbenchmarks.ai).
- Rule of thumb: more than 10 red flags in a vendor = a problem; more than 20 = vibe procurement territory.

We recently helped build the legal industry's first buyer-led framework and toolkit for evaluating AI tools for legal teams. From that work, we extracted 45 concrete red flags: the warning signs your team should watch for across 8 core evaluation criteria. Each one is a practical signal you can spot during a demo, in vendor documentation, or in the contract itself.

If you recognise more than a handful of these in a vendor you're evaluating, it might be time to ask harder questions, or walk away.

## 1. Strategic Fit

Strategic fit is where most evaluations go wrong first. You're not just asking "does this tool do AI?" You're asking whether it was built for organisations like yours, works with your systems, and serves your jurisdictions.

### 1.1 Fit to your priority legal work

- 🚩 Website does not show customers similar to your organisation profile, team size, or industry.
- 🚩 Case studies and use cases focus on a different buyer profile than yours.

### 1.2 Fit with your systems and operating model

- 🚩 Integrations emphasised are not the ones relevant to your environment.
- 🚩 Ecosystem appears optimised for a different type of company (e.g. startup tools vs enterprise systems).

### 1.3 Fit with your jurisdictions, languages, and product direction

- 🚩 Defaults to US or English-only assumptions with weak regional coverage.
- 🚩 Recently announced product direction or market focus does not match your legal team's likely needs.

## 2. Functionality

AI demos always look incredible. The real test is what happens when a lawyer uses it on a Monday morning with a 200-page contract scanned from a fax machine in 2019.

### 2.1 Usable by lawyers with minimal friction

- 🚩 Interface is cluttered or core actions are hard to find.
- 🚩 Too many clicks, dropdowns, or manual steps for common workflows.
- 🚩 Product is not intuitive for a lawyer using it in practice.

### 2.2 Handles real-world input conditions

- 🚩 Performs poorly on low-quality scans or dense, heavily formatted agreements.
- 🚩 Struggles with longer or more complex documents, or multi-document review.
- 🚩 Cannot reliably handle messy, real-world legal inputs.

## 3. Robustness

This is the category where the gap between marketing and reality is widest. Robustness is not about whether the AI can produce an answer. It's about whether you can trust it.

### 3.1 Accurate, complete, and faithful outputs

- 🚩 Hallucinated content, missed provisions, or silent truncation of outputs.
- 🚩 Overly confident or sycophantic outputs that do not flag uncertainty or risk.

### 3.2 Verifiable and independently validated

- 🚩 Accuracy claims are entirely self-reported with no willingness to support independent verification.
- 🚩 No published methodology behind performance claims; numbers lack context or test conditions.

### 3.3 Stable performance in realistic conditions

- 🚩 Model loses context, contradicts itself, or agrees with clearly wrong assumptions.
- 🚩 Defaults to the wrong legal context or misses obvious material risks.

## 4. Security

Security is not a checkbox. It's an architecture question. Any vendor can claim they're "secure." What matters is whether they can explain how, in detail, and back it up with evidence.

### 4.1 Transparent architecture and data flow

- 🚩 Explanations are vague, inconsistent, or rely on generic diagrams with no operational detail.
- 🚩 No evidence of an information-security programme: no ISO 27001 or SOC 2 report, no documented control set built to those criteria, or a vendor relying mainly on subprocessor certifications rather than its own.

### 4.2 Strong access control, isolation, and retrieval boundaries

- 🚩 Restricted content can leak across users, matters, or workspaces.
- 🚩 Vendor cannot clearly explain how customer data is separated.

### 4.3 Safe behaviour under misuse and failure conditions

- 🚩 Prompt injection succeeds or actions can be triggered without proper approval.
- 🚩 Vendor lacks credible evidence of security testing and response preparedness.

## 5. Data Privacy

Data privacy in legal AI is not about GDPR compliance badges on a website. It's about whether the vendor's actual data practices match what they promise, and whether your clients' privileged information is truly protected.

### 5.1 Contractual limits on data use

- 🚩 "No training on customer data" appears only in marketing, not in contractual commitments.
- 🚩 Broad "service improvement" wording still permits use of anonymised or aggregated data.
- 🚩 Human review rights are unclear or insufficiently limited.

### 5.2 Deletion and lifecycle control

- 🚩 No clear mechanism to confirm data has been fully deleted on request.
- 🚩 Retention periods are indefinite, undefined, or not contractually committed.

### 5.3 Processing, localisation, and derived-data governance

- 🚩 Vendor cannot clearly state where data is processed or stored.
- 🚩 Embeddings and derived data are treated as outside the customer's protection framework.

## 6. Vendor Risk

Vendor risk goes beyond financial stability. It's about whether you can leave, what happens to your data if the vendor fails, and whether their commitments are enforceable.

### 6.1 Clear contractual and security commitments

- 🚩 Rights over outputs are unclear or important safeguards sit only in changeable web documentation.
- 🚩 Continuity protections are weak or ambiguous.

### 6.2 Real exit, portability, and accountability

- 🚩 Playbooks, clause rules, negotiation guidance, or approval logic cannot be extracted if you leave.
- 🚩 Configuration and institutional knowledge are trapped inside the tool.

### 6.3 Credible vendor conduct and resilience

- 🚩 Vendor is opaque, over-claims maturity, or has no credible reference base.
- 🚩 Cannot show how it would handle disruption or failure.

## 7. Adoption Support

The best AI tool in the world is worthless if nobody uses it. Adoption support is where you find out whether the vendor is invested in your success, or just in closing the deal.

### 7.1 Training and onboarding that work for legal users

- 🚩 Training materials are generic, thin, or outdated (e.g. not updated in 6+ months).
- 🚩 Limited self-serve resources; onboarding does not reflect legal workflows.

### 7.2 Responsive support and workable feedback loops

- 🚩 Support is slow, vague, or sales-led with unclear escalation paths.
- 🚩 No support coverage in your region, slowing response times and contextual understanding.

### 7.3 Documentation, change communication, and usage visibility

- 🚩 Model or system changes happen silently with no advance notice.
- 🚩 Reporting is too weak to support governance, renewal, or adoption decisions.

## 8. Cost & Resourcing

Legal AI vendors have learned that the demo sells and the invoice surprises. Cost transparency is not optional, and you need to model total lifecycle cost, not just license fees.

### 8.1 Transparent pricing that scales sensibly

- 🚩 Pricing is opaque, cost growth is hard to model, or there are steep step-changes between pilot and rollout.

### 8.2 Full lifecycle cost is understood

- 🚩 Important costs appear late: connectors, implementation support, migration, or exit charges.

## What to Do Next

If you counted more than 10 red flags in a vendor you're currently evaluating, you have a problem. If you counted more than 20, you may be in "vibe procurement" territory: buying based on enthusiasm rather than evidence.

The good news: every red flag on this list is observable before you sign. You can spot them in demos, in documentation, in contracts, and in the vendor's responses to direct questions. The framework these red flags come from (the Legal AI Evaluation Framework by Legal Benchmarks) provides structured scoring templates and evaluation toolkits to run a proper assessment.

> **Note:** Download the full Legal AI Evaluation Framework and Toolkit at legalbenchmarks.ai/framework: the legal industry's first buyer-led, vendor-neutral evaluation system for AI tools.

## How HAQQ Addresses These Red Flags

We built HAQQ specifically to pass this kind of scrutiny. Multi-jurisdictional coverage across 9 languages. Compliant with SOC 2 and ISO 27001, which is alignment rather than a completed third-party certification. Full data isolation per workspace. No training on customer data: contractually committed. Transparent architecture documentation. And a legal AI engine (Justinian) purpose-built for the evidentiary demands of legal practice.

We welcome buyer-led evaluation. If your firm is running a structured AI procurement process, we'll participate in any framework-based assessment, including the one these red flags come from.

## Related reading

- [evaluating AI contract review platforms](https://www.haqq.ai/blog/ai-contract-review-lawyers-guide)
- [independently validated, jurisdiction-aware benchmarks](https://www.haqq.ai/blog/mena-legal-ai-benchmark-scope-and-method)
- [how prompt injection attacks legal AI](https://www.haqq.ai/blog/prompt-injection-scanner-legal-ai)
- [why legal tech adoption fails](https://www.haqq.ai/blog/why-legal-tech-keeps-failing)

## Häufig gestellte Fragen

### Was ist Vibe Procurement im Legal-Tech-Bereich?

„Eine glänzende Demo, ein paar Schlagworte und ein charismatischer Vertriebler, und plötzlich hat sich Ihre Kanzlei auf einen sechsstelligen Vertrag festgelegt.“ Der Artikel nennt Vibe Procurement das schlecht gehütete Geheimnis der Legal-Tech-Branche: Kaufen aus Begeisterung statt aufgrund von Belegen.

### Wie bewertet man einen Anbieter juristischer KI?

Nutzen Sie die 8 Kriterien des käufergeführten Rahmenwerks: strategische Passung, Funktionalität, Robustheit, Sicherheit, Datenschutz, Anbieterrisiko, Unterstützung bei der Einführung sowie Kosten und Ressourcen, mit 45 konkreten Warnsignalen, die „in Demos, in der Dokumentation, in Verträgen und in den Antworten des Anbieters auf direkte Fragen“ erkennbar sind, bevor Sie unterschreiben.

### Wie viele Warnsignale sind zu viele?

Laut Artikel: „Wenn Sie bei einem Anbieter, den Sie gerade prüfen, mehr als 10 Warnsignale gezählt haben, haben Sie ein Problem. Wenn Sie mehr als 20 gezählt haben, sind Sie womöglich im Bereich des Vibe Procurement.“

### Was sind die größten Sicherheits-Warnsignale bei juristischen KI-Tools?

Das Sicherheitskriterium umfasst drei Bereiche: undurchsichtige Architektur und Datenflüsse („jeder Anbieter kann behaupten, sicher zu sein, entscheidend ist, ob er im Detail erklären kann, wie“), schwache Zugriffskontrolle, Isolation und Abrufgrenzen sowie unsicheres Verhalten bei Missbrauch und Fehlerbedingungen.

---

Canonical page: https://www.haqq.ai/de/blog/45-red-flags-legal-ai-vendor-evaluation
Whole-site context for agents: https://www.haqq.ai/llms.txt · https://www.haqq.ai/agents.md

Generated at build time from the site's own content data. Do not edit by hand.
