How Smarsh Automated AWS Well-Architected Reviews for Agentic AI Workloads Using Custom Claude Skills

In the fast-evolving landscape of enterprise artificial intelligence, regulated industries face a fundamental tension between the industry-standard ethos of rapid prototyping and the uncompromising demands of corporate compliance. For organizations operating in highly supervised sectors—such as financial services, legal, and healthcare—there is virtually no tolerance for vulnerabilities, particularly when autonomous AI agents are tasked with managing sensitive, mission-critical customer communications data. Navigating this delicate balance requires rigorous governance frameworks that can keep pace with rapid technological development without sacrificing security or operational resilience.

Addressing this exact challenge, Smarsh, a recognized global leader in electronic communications archiving and digital communications governance, recently developed an innovative methodology to evaluate its pioneering multi-agent AI workload. Utilizing the AWS Well-Architected Framework’s newly introduced Agentic AI Lens, Smarsh engineered a custom automated review system powered by Anthropic’s Claude. This breakthrough transformed a traditionally tedious, multi-day manual auditing process into a streamlined, evidence-backed workflow capable of evaluating complex software architectures in a matter of hours.

The Background and Evolution of Digital Governance at Smarsh

Founded over two decades ago, Smarsh has established itself as an indispensable pillar of compliance infrastructure for more than 7,000 organizations globally, ranging from boutique advisory firms to massive multinational financial institutions. The multinational software-as-a-service (SaaS) provider enables its clients to capture, retain, monitor, and analyze electronic communications across more than 100 digital channels. These channels include traditional vectors like email and mobile messaging, as well as complex modern modalities such as voice, video, chat applications, and enterprise AI assistants.

As the volume and diversity of communications data have exploded, Smarsh has continually modernized its technological backbone. The company’s core enterprise archive architecture operates primarily on Amazon Web Services (AWS), providing the immense scalability and reliability required to handle petabytes of sensitive enterprise data. Building upon this foundation, Smarsh has begun deploying its next-generation suite of artificial intelligence capabilities utilizing Amazon Bedrock AgentCore—a comprehensive platform designed to build, connect, and optimize autonomous AI agents at scale across any framework or model. By leveraging AgentCore’s robust runtime, gateway, registry, and identity capabilities, Smarsh is able to deploy sophisticated agentic solutions that meet the stringent security expectations of its client base. The depth of this technological alignment was underscored when Smarsh was named a 2025 AWS Partner of the Year Award winner, cementing its status as a premier collaborator within the cloud ecosystem.

The Regulatory Dilemma: A Lens You Can Read But Not Run

Despite these advanced technological capabilities, deploying autonomous agents into production within regulated sectors demands strict adherence to architectural best practices. For Smarsh, every newly developed agentic solution must undergo a comprehensive Well-Architected review before it is cleared to interact with end customers or process live enterprise data. When a lead architect at Smarsh prepared to evaluate the company’s inaugural multi-agent workload—an intelligent compliance alert review system built on Amazon Bedrock AgentCore—the logical choice was the AWS Well-Architected Framework’s Agentic AI Lens.

However, the team encountered a significant operational hurdle. While established diagnostic tools like the Serverless Applications Lens can be integrated directly into the native AWS Well-Architected Tool console, the Agentic AI Lens initially existed solely as static published documentation. Comprising 150 detailed best practices spread across 41 distinct questions and six architectural pillars—each featuring specific risk levels and desired outcomes—the lens lacked native console execution support at the time of the review.

This limitation presented engineering leadership with an unappealing binary choice: either commit to a multi-day manual assessment where individual reviewers would inevitably structure evaluations inconsistently, or wait indefinitely for native console tooling to catch up with published guidance. Rejecting both options, Smarsh chose an ambitious third path: to make the published lens executable programmatically, creating a reusable asset that could govern all future agentic workloads across the enterprise.

Architectural Innovation: The Lens as Data and the Reviewer as a Skill

To operationalize the review process, Smarsh packaged its compliance auditing methodology into a specialized, versioned Claude skill. Designed for maximum efficiency, the skill consists of a meticulously structured bundle of instructions, reference datasets, and executable scripts that Claude discovers dynamically via frontmatter descriptions, loading them only on demand. This progressive disclosure mechanism ensures that token utilization remains negligible until a comprehensive review is explicitly requested.

The underlying architecture of the custom skill relies on three core components that bridge raw repository data with the structured requirements of the AWS Well-Architected Framework. Central to this architecture is a standardized assessment contract defined within a single JSON document, which maps technical findings directly to individual lens criteria. For instance, an evaluation of an input validation mechanism might yield a structured record indicating partial compliance, citing specific technical evidence such as two-stage JSON schema validation and payload caps, while simultaneously highlighting architectural gaps and prescribing concrete remediation steps.

Initiating an audit is remarkably straightforward, requiring just a single-line prompt directing Claude to examine a target GitHub repository and execute a Well-Architected review. From there, the automated workflow initiates a multi-step sequence:

1. Cloning and indexing the target repository to map out infrastructure-as-code definitions, application logic, and testing suites.
2. Querying the codebase against all 150 individual best practices outlined in the Agentic AI Lens documentation.
3. Gathering empirical evidence directly from code files, configuration scripts, and documentation records.
4. Evaluating compliance statuses—ranging from fully compliant to high-risk non-compliant—based on verifiable code artifacts rather than anecdotal developer claims.
5. Identifying architectural gaps and correlating them with specific risk tiers.
6. Generating precise, actionable remediation recommendations tailored to the workload’s specific technology stack.
7. Compiling all findings into a comprehensive, standardized review workbook ready for engineering triage.

Rigorous Validation: Testing the Tester

Recognizing the risks associated with automated compliance tooling, the Smarsh engineering team instituted a rigorous validation protocol to evaluate the reliability and accuracy of the custom skill before deploying it against production workloads. The team constructed two synthetic fixture repositories designed to mimic real-world development scenarios: a moderately mature, LangGraph-style multi-agent application, and a deliberately flawed proof-of-concept repository laden with hardcoded credentials and a historical record of runaway-cost incidents.

The team executed multiple review runs both with and without the custom skill enabled, grading each output against eight strict, verifiable criteria per fixture. These criteria mandated that all 150 lens items be present, that no status fields remain blank, that definitive statuses include explicit evidentiary citations, and that the tool maintain honest calibration regarding identified risks. Furthermore, evaluations were tested on their ability to flag fixture-specific vulnerabilities, such as the intentionally embedded hardcoded credentials.

The results of the evaluation phase strongly validated the architectural approach. Review runs utilizing the custom skill successfully passed 16 out of 16 defined expectations, demonstrating flawless traceability back to the Agentic AI Lens. Conversely, baseline runs conducted without the skill passed only 8 out of 16 expectations, yielding unstructured, ad-hoc checklists containing merely 18 to 39 items with zero traceability to the Well-Architected Framework. To ensure ongoing quality, the evaluation set was permanently integrated into the skill’s repository, establishing a continuous regression testing pipeline as the tool evolves.

Key Findings from the Inaugural Production Review

Armed with a validated assessment tool, the Smarsh architecture team deployed the skill against their live multi-agent workload. In a matter of hours, the system ingested and analyzed a diverse array of artifacts, including architecture documentation, authentication specifications, AWS Cloud Development Kit (AWS CDK) infrastructure definitions, GitHub Actions CI/CD workflows, registry and gateway design records, and comprehensive test suites.

The resulting workbook revealed a realistic, highly credible profile of the workload’s architectural posture, highlighting a balanced mix of robust security controls and areas requiring targeted remediation. On the positive side, the audit verified strong foundational controls: per-agent AWS Identity and Access Management (IAM) execution roles strictly scoped to named runtime Amazon Resource Names (ARNs), SigV4-signed Model Context Protocol (MCP) communications routed securely through the Amazon Bedrock AgentCore Gateway, schema-validated inputs equipped with typed error-handling contracts, exported decision audit trails, and continuous behavioral quality monitoring via AgentCore online evaluations.

Conversely, the automated review successfully uncovered critical high-risk gaps that might have otherwise escaped detection during a manual review. Notable findings included a gateway ingress authorizer operating with authentication temporarily disabled—a calculated risk accepted during early development phases—alongside an absence of dedicated prompt-injection screening on message content influenced by external actors. Additionally, the audit flagged rate-limiting mechanisms that had been relegated to unaddressed code comments and a lack of automated cost-control safeguards. Each identified vulnerability was explicitly mapped to its corresponding lens item, supported by direct code evidence and accompanied by an engineered remediation path.

Following the automated generation of the workbook, the Smarsh architect convened the core engineering team to conduct a collaborative triage session. Developers and infrastructure specialists validated the findings against their firsthand knowledge of the system, accepting, challenging, or refining results directly within the shared document. This human-in-the-loop collaboration significantly enhanced the accuracy of the review; for instance, multiple isolated findings were logically consolidated into a single, comprehensive anomaly-detection epic, while another finding was upgraded in severity after the team pointed to auxiliary evidence that the automated scan had initially overlooked, such as pre-existing dependency scanning active on every pull request.

Elevating AWS Engagement Through Precision Data

A particularly transformative outcome of the automated review process was its ability to bridge the gap between internal software engineering and overarching cloud architecture. While codebase audits can effectively evaluate internal logic and configuration scripts, they cannot resolve system-semantics questions inherent to underlying cloud services. Rather than scheduling a generalized, unfocused architecture review with their cloud provider, the Smarsh engineering team utilized the workbook to construct a highly targeted, data-driven agenda for their dedicated AWS Solutions Architects.

Mapping every discussion point directly to specific checklist items slated for closure, the team engaged in a series of deep-dive working sessions with clear definitions of done. The agenda focused on optimizing the interplay between Amazon Bedrock AgentCore’s runtime, gateway, registry, and identity features, ensuring that Smarsh’s implementation adhered to the most rigorous standards of cloud-native security and scalability. This precise, evidence-based engagement model dramatically accelerated problem resolution, transforming routine consultations into high-value engineering collaborations.

Implications and Broader Industry Impact

The methodology pioneered by Smarsh offers a compelling blueprint for organizations navigating the complexities of agentic AI governance in regulated markets. As artificial intelligence architectures evolve at a breakneck pace, cloud providers and standards bodies frequently publish prescriptive guidance faster than native console tooling can absorb and operationalize it. By adopting a "lens as data" philosophy and encoding evaluation discipline into versioned, testable AI skills, enterprises can bridge the governance gap independently.

Furthermore, the integration of automated pre-screening with expert human judgment ensures that velocity does not compromise security. For Smarsh, the ability to execute rigorous Well-Architected reviews in hours rather than days dramatically reduces compliance friction, accelerating time-to-market for innovative AI solutions while maintaining the uncompromising reliability expected by institutional clients in the financial sector and beyond.

Looking toward the future, Smarsh plans to expand the capabilities of its review framework. Planned enhancements include scaling the custom skill to evaluate a broader portfolio of agentic workloads currently under development, integrating automated remediation ticket generation directly into enterprise project management systems, and contributing continuous feedback to the broader developer community regarding the practical application of emerging AI governance frameworks.

As enterprises increasingly transition experimental AI prototypes into production-grade autonomous agents, the challenge of maintaining architectural integrity will only intensify. Smarsh’s successful deployment of an automated, skill-driven Well-Architected review demonstrates that with the right combination of custom automation, rigorous self-evaluation, and strategic cloud partnerships, organizations can innovate rapidly while upholding the highest standards of digital trust and compliance.