Custom Software Development Evaluation Criteria: Scoring Vendor Architecture, Code Quality, and Velocity

The Procurement Trap: Why Case Studies Fail Engineering Directors

Most vendor evaluations fail because non-technical procurement teams run them. They score agencies on polished decks, generic testimonials, and client logo wall boards. An agency displays logos of Fortune 500 companies where they once supplied three staff augmentation contractors to fix CSS bugs, claiming credit for the client’s core application.

Engineering Managers and VPs of Engineering cannot rely on sales presentations. A portfolio tells you that an agency can hire a frontend designer; it tells you nothing about how their engineers handle database migrations, thread safety, or distributed locking under load.

When you invest $150,000 or $400,000 into a custom build, your primary risk is not that the software will fail to launch. The risk is that it launches on time, but costs $30,000 a month in emergency bug fixes because the code is an untestable, coupled mess.

Validating a vendor requires shifting from subjective trust to objective technical evaluation. You need to inspect their actual artifacts: anonymized pull requests, infrastructure code, deployment logs, and sprint retrospectives.

Criteria 1: Architectural Durability and Boundary Isolation

Architectural durability determines how your system withstands changing requirements over a three-to-five-year window. Bad vendors build "distributed monoliths"—applications split across multiple microservices that share a single Postgres database and rely on synchronous HTTP calls. This gives you the complexity of microservices with the failure modes of a monolith.

Evaluate the vendor’s approach to domain boundaries and state management. Ask their principal architect to explain how they handle read/write contention, schema evolution, and async task execution.

Look for three specific architectural signals during the evaluation process:

• Strict data domain isolation: Services must own their database schemas. If Vendor A proposes three node services pointing directly to the same public schema tables, drop them immediately.
• Declarative infrastructure: Every resource—S3 buckets, IAM roles, ECS tasks, database clusters—must be defined in Terraform, AWS CDK, or Pulumi. Manual console clicks ("ClickOps") represent fatal technical debt.
• Pragmatic technology choices: Beware vendors pushing experimental frameworks to train their developers on your dime. Standardize on stable, mainstream runtimes like TypeScript (Node/React), Go, Python, or Java/Kotlin unless your domain demands specialized tools (e.g., Rust for embedded performance).

Criteria 2: Test Automation, CI/CD Rigor, and Static Analysis

A vendor that promises "high quality" without showing you a passing CI build log is lying. Code quality is not a matter of developer pride; it is an automated pipeline mechanism that blocks bad commits from reaching main branches.

Require the vendor to walk you through a live, non-proprietary GitHub Actions or GitLab CI pipeline from a recent client engagement.

Assess their testing strategy using these three strict benchmarks:

• Automated unit and integration test coverage: Demand a minimum of 80% line coverage on core business logic. UI components can sit lower, but data transformations, payment pipelines, and API integrations must have deterministic tests.
• Hermetic test environments: Tests must run against isolated local database containers (e.g., via Testcontainers or Docker Compose), not a shared, mutable staging environment that introduces flake.
• Mandatory static analysis gatekeepers: The pipeline must fail if static analysis checks fail. Look for automated linting (ESLint, golangci-lint), security vulnerability scanning (Snyk, Trivy), and dependency license checks on every pull request.

Ask the vendor: "What happens when a critical security patch drops for a core dependency?" If their answer involves manual quarterly audits rather than automated dependabot PRs and automated test runs, pass on them.

Criteria 3: Velocity Metrics and Pull Request Throughput

Agencies love to quote "sprint velocity" using story points. Story points are meaningless external metrics; vendors regularly inflate point estimates to make their burn down charts look impressive.

To evaluate true engineering velocity, look at concrete Git operational metrics from actual historical projects.

Score the vendor across these three operational markers:

• PR cycle time under 24 hours: The time elapsed between opening a Pull Request and receiving a peer review must average less than one business day. Long-lived feature branches breed massive merge conflicts and hidden bugs.
• Small atomic commits: High-velocity teams ship PRs that touch fewer than 300 lines of code across 3 to 5 files. Massive 3,000-line PRs indicate poor task decomposition and result in rubber-stamp code reviews.
• Deployment frequency: The vendor should deploy to a staging or preview environment automatically on every merged PR, and deploy to production multiple times per week behind feature flags.

Request anonymized Git metrics. A vendor reluctant to share average PR size or review turnaround times usually suffers from internal bottlenecks and unvetted junior developers.

Criteria 4: Staffing Math, Seniority Ratios, and Cost Realities

Vendor pricing models vary wildly across the US software market. An engagement quoted at $150,000 by one agency might sit at $450,000 at another. The variance rarely comes from profit margins; it comes from staffing ratios and developer seniority.

The classic agency margin trick is the "bait-and-switch": bidding the project using a brilliant Staff Engineer during pre-sales, then handing the daily commits to three offshore junior devs making $18/hour while billing them out at $125/hour.

Compare standard US market rates against our Engineer Cost Index 2026 to sanity-check your quotes:

A healthy development pod for a $250,000 build usually looks like: 1 Lead Architect (20%), 2 Senior Engineers (100%), 1 Mid-Level Engineer (100%), and 1 QA Automation Engineer (50%).

If a vendor offers a team of six developers at a blended rate under $75/hour, you are paying for an unmonitored junior coding farm. Your internal engineers will spend the next two years rewriting their work.

The Complete Vendor Evaluation Scoring Matrix

Use this weighted scoring rubric during vendor technical interviews and code review sessions. Assign scores from 1 (Unsatisfactory) to 5 (Exemplary) across each category.

Total the weighted score. Do not award contracts to vendors scoring below 3.8 out of 5.0, regardless of how low their price quote comes in.

Contract Mechanics: IP Ownership, SOW Milestones, and SLAs

Contract terms determine whether you maintain leverage during a complex build. Never sign a generic Time & Materials (T&M) agreement without explicit milestone gates and code delivery standard requirements.

Protect your downside risk by structuring the Statement of Work (SOW) around three hard contractual mechanics:

1. Continuous IP Assignment: Ensure IP transfers to your firm continuously upon payment for each bi-weekly billing cycle. Never accept clauses where intellectual property stays with the vendor until the final invoice of a year-long project is settled.
2. Paid Discovery Milestones: Run a $15,000 to $30,000 paid discovery sprint before signing a $300,000 main build contract. The output must be concrete: technical specs, schema models, sequence diagrams, and a working proof-of-concept repository. Review our past project deliverables in our Proof repository to see what a production-ready technical discovery artifact looks like.
3. Code Acceptance Criteria in SLAs: Tie invoice payments to objective code quality metrics. Specify in the SOW that delivered code must pass automated CI checks, maintain agreed test coverage thresholds, and deploy cleanly to a sandboxed environment before an invoice is considered payable.

If a vendor refuses to tie contract acceptance to automated pipeline checks, they do not trust their own engineering quality.

What This Means for Your Team

Procuring custom software is a technical risk management exercise. When you fail to score vendor code quality, architecture, and CI/CD pipelines during procurement, you are deferring technical debt payments to your future self.

Take these three immediate steps for your upcoming vendor selection process:

1. Stop reading marketing decks: Demand a live walkthrough of a non-proprietary codebase and a real CI/CD pipeline from an active engineering team.
2. Apply the weighted evaluation matrix: Score every candidate on the 1-5 rubric detailed above. Eliminate any vendor that hides their Git cycle time metrics or relies on manual deployment strategies.
3. Enforce technical acceptance terms in the SOW: Require continuous IP assignment and tie billing milestones directly to automated build passes and test coverage thresholds.

If you are planning an application modernization or new product build between $120,000 and $500,000 and want a senior, US-based engineering team that ships clean code behind transparent pipelines, get in touch with our engineering leadership.