Scaling AI: The Foundational Pillars for IT Leaders

The relentless march of artificial intelligence, characterized by increasingly sophisticated capabilities and the rise of autonomous agentic systems, presents both immense opportunities and significant challenges for organizations. As AI applications expand across diverse use cases, IT leaders grapple with the crucial task of making strategic investments that will remain valuable in a rapidly evolving technological landscape, all while mitigating inherent risks. The key to navigating this complexity lies in returning to the foundational elements of AI architecture – the robust structural framework essential for deploying and managing integrated, reliable AI systems at scale. This approach empowers technology leaders to make informed decisions today that will support a future where AI agents can seamlessly retrieve information, make intelligent decisions, and execute complex workflows across various enterprise systems. A cornerstone of any effective AI architecture is data quality. Models are inherently limited by the data they access; poor quality data inevitably leads to AI hallucinations, biases, and unreliable outputs, undermining user confidence and business value. Many enterprises struggle with fragmented data ownership, inconsistent structures, and incomplete datasets residing in legacy systems, which severely impedes AI scalability. As Adnan Adil, CIO of Elastic, emphasizes, “The data is a durable part of AI architecture because without it, these models won’t run, won’t provide the right context, or won’t give the right level of services that we’re looking to implement.” Industry surveys consistently highlight data quality as a primary barrier to AI success, with Gartner predicting that 60% of AI projects will fail by 2026 if not supported by AI-ready data. An effective strategy demands connecting, organizing, governing, and ensuring real-time accessibility of data from the outset. Beyond raw data, context engineering is critical for ensuring AI models draw upon the most pertinent information for each query, producing accurate and efficient answers. Unlike prompt engineering, which focuses on query wording, context engineering designs the entire information environment around the model. This involves intelligently retrieving the right data and presenting it in a structured, machine-readable format. Organizations are increasingly recognizing that reliable AI depends as much on the quality of context as it does on the model’s inherent strength. This discipline relies on a modernized, unified data foundation, coupled with advanced retrieval and memory systems like Retrieval Augmented Generation (RAG) and vector databases. Adil stresses that “Minimum context, correct and current data, and machine-readable information are critical to effective context engineering,” cautioning against overwhelming models with excessive context that can dilute relevance and increase costs. Strong governance and security are indispensable for maintaining control over how AI systems utilize data, monitoring performance, and preemptively identifying issues. Without clear controls over data retrieval, workflows, and model usage, AI systems often process unnecessary information, leading to inefficiencies, higher computing costs, and increased token consumption. Furthermore, AI expands an organization’s attack surface, introducing new risks such as prompt-based data leakage, model vulnerabilities, and adversarial inputs. Robust access controls, continuous monitoring, and vigilant oversight are paramount for protecting sensitive information. Adil points out that essential controls — including those related to security, granular cost management, and data architecture — are frequently insufficient, underscoring the need to embed governance structures into architecture and decision-making processes from the very beginning, rather than as an afterthought. Finally, LLM observability complements governance by providing critical insights into how AI applications perform in real-world scenarios. When governance is established from the start, it enables robust observability, allowing teams to assess accuracy, utility, and adoption patterns over time. This visibility is crucial for building trust, identifying failure points, and continuously refining systems as requirements evolve. Observability is also key to demonstrating the return on investment (ROI) for AI initiatives, as the business value often depends heavily on how systems are adopted and used. Real-time visibility into AI behavior allows organizations to measure performance against expectations, bridge gaps between intent and reality, and ensure continuous improvement. An Elastic report from 2026 indicates that 85% of IT decision-makers plan to enable LLM observability for their internal generative AI applications, highlighting its growing importance. As Adil concludes, “Observability is actually huge. We can use observability data for cost control, decision-making, and engineering efficiency.”