Assessing AI-Assisted Search Answer Integrity on Digital Banking Websites: A Comparative Audit of US and Pakistani Banks

Abstract

AI assistants are now used to answer everyday questions, including about banking, yet their accuracy in this domain is largely untested. A wrong fee or interest rate is not academic; it can cost the consumer real money. Whether accuracy holds across markets, or whether phrasing changes the answer, is also unknown. The design is a cross sectional comparative audit of the top five banks by total assets in the United States and Pakistan. Each was queried about four consumer-banking facts (two fees, a savings rate and a 24/7 helpline) under a structured source-anchored prompt and a short naturalistic one, each repeated twice for 160 LLM responses. Ground truth was built manually from the official websites, and each response was classified into five categories by an LLM-as-judge validated against two human annotators. Even with web browsing on, the correctness rate across the 160 responses is 51.3% and the hallucination rate is 31.9%. The country difference is significant (χ2 = 16.31, p = 0.0026, Cramér’s V = 0.32): U.S. banks are Correct two thirds of the time, Pakistani banks just over one-third. The naturalistic prompt scores higher than the structured one (57.5% vs. 45.0%), but a paired McNemar’s on 40 pairs is not significant, so the claim is directional. Run-to-run consistency is 83-85%. The study contributes a replicable framework, the five-category taxonomy, hybrid automated/human verification and paired prompt-formality design, for evaluating AI accuracy wherever a verifiable reference is available, including fields beyond banking. It also sheds light on how the way banks construct information shapes what AI assistants report.