Nikkei revamps search infrastructure of over 200 million records with Elastic Cloud, forming the foundation for new generative AI services

About Acroquest Technology (Integration Partner)

Three-time winner of Japan’s No. 1 ranking for "Most Rewarding Companies to Work For." An IT venture characterized by an open corporate culture where all employees participate in decision-making, including salary determinations. The company provides end-to-end support for the implementation, design, and operation of the Elastic Stack and has a proven track record of deploying Elasticsearch in large-scale search systems, including those for the Nikkei Inc. It is Japan's first Elastic Consulting Partner and boasts the largest number of Elastic-certified professionals in the country.

As a company that has accumulated economic news records for over 150 years, Nikkei Inc. and its database services are currently undergoing a rapid evolution. The company recently incorporated state-of-the-art technology in its search engine service, including the release of NIKKEI KAI, which goes beyond traditional article search services by using generative AI to extract key points from retrieved articles and directly provide answers to user questions. At the heart of these technologies is the search engine Elasticsearch.

Building on its 150-year history in the media business, Nikkei Inc.'s major revenue pillar today is its corporate-geared database business, including services such as NIKKEI TELECOM and Nikkei NEEDS, which leverage the company's massive stock of articles. Notably, NIKKEI TELECOM is utilized by over 70% of domestic listed companies, marking it an essential service for businesspersons in Japan.

The company's article database was previously operated on-premises. Taisuke Hinata, search engineer for the Service Infrastructure First Group of the Technical Strategy Unit, reflects on the past setup, "One of NIKKEI TELECOM's primary features is its abundant search function. We developed the service so that users can finely configure features such as aggregation of article search results and highlighting of searched keywords within articles. Back when we outsourced database management, we had to rewrite specifications each time requirements changed and outsource development, incurring not only outsourcing costs, but time as well."

The service also faced the challenge of its database scalability.

"The strength of our company's database service is its capability to search a broad range of external media records all at once. As such, we are constantly pursuing partnership opportunities with new media outlets — in fact, more media partnerships have shown to drive user growth. However, since operations were conducted on-premises, we had to increase our server capacity in line with partnership expansion, which led to a unilateral increase in operational burdens," explains Hinata.

With roughly 200 servers in operation, the company was at a crossroads — it was time to fundamentally review the labor and costs required with scaling its database on-premises. In 2018, the company decided to migrate its database to the cloud and revamp its search engine. And in line with this transition, it was also determined to fully internalize the development and operations of its database system.

The company's requirements for the new search engine were stringent.

"With the cloud migration of NIKKEI TELECOM, adopting an engine that could accommodate its rich search features was absolutely essential. Elasticsearch fulfilled this requirement," says Hinata.

The post-migration effects were notable. By transitioning the search engine to Elasticsearch, the company succeeded in reducing infrastructure costs by 22.5%. Additionally, it also realized the other existing challenge of internalizing operations.

"As a company that is also a newspaper business, we don't have as many residing engineers, but Elasticsearch distributes images that makes it easy to virtualize. This presented the significant merit of our small team being able to scale services themselves while operating a large-scale system," Hinata describes.

Elasticsearch's fault tolerance and high scalability are also recognized as positive effects. Even if a specific node goes down, the service can conduct automatic recovery from pre-prepared replicas. Additionally, version upgrades — which were difficult back when NIKKEI TELECOM was operated on-premises — could now be handled entirely in-house. These changes have successfully established a system that constantly incorporates the latest features.

Migrating the database and search infrastructure to the cloud has enabled smooth response to the ever-growing volume of data, which has reached roughly 210 million documents. These documents trace all the way back to PDFs of prints of the weekly newspaper Chugai Bukka Shinpo — the predecessor of The Nikkei first issued in 1876 — to all article data up to the present.

Yasufumi Mizoguchi, who works alongside Taisuke Hinata at the Service Infrastructure First Group, notes, "Even with new media partnerships, as long as we add nodes and set up indices, we can scale search targets without sacrificing performance. This high level of scalability is a crucial element of our company's database."

As Nikkei’s database business steadily grew following its migration to a cloud-based infrastructure, the company turned its attention to reducing the operational overhead of managing infrastructure on Amazon Web Services (AWS). In 2024, the company implemented Elastic Cloud, Elasticsearch’s managed service.

"Previously, when system errors occurred for unknown reasons, it took us a substantial amount of time to identify the cause. However, we now have Elastic to lean on as responders, which has significantly reduced burdens on our end," Mizoguchi explains.

Taisuke Hinata and Yasufumi Mizoguchi’s teams have also been credited for establishing a system that can be sustainably managed by a small team, and they were awarded an internal corporate award for their accomplishment.

Hinata notes, "With the emergence of generative AI, our database business is undergoing a major transformation. By replacing our search engine — the heart of our database — with Elastic Cloud, I believe we’ve thoroughly prepared our service to thrive in the next generation."

Nikkei recently introduced a new service: a next-generation search model called NIKKEI KAI. Departing from the former service of simply searching for articles based on user searches, NIKKEI KAI presents a completely new experience and value of providing AI responses to user questions based on article search results.

On the importance of NIKKEI KAI, Hinata describes, "It represents a major transition from a service that merely provides documents that match search results to one that extracts information that users are looking for — what is known as 'zero-click' searches."

From a technical standpoint, the company implemented RAG through Elasticsearch, adopting a hybrid search system that combines vector search with full-text search.

On the service's mechanism, Mizoguchi illustrates, "Once a user enters a question, the system internally breaks down its contents and executes search queries from various perspectives. The results are then aggregated and passed to the generative AI. The service is formed upon a complex underlying logic."

What is particularly notable is the development speed behind NIKKEI KAI. Following the launch of its development in early 2024, its database schema was constructed by August, allowing for the operation of the service's beta version. After running and verifying the data processing batches, NIKKEI KAI went live in the production environment in March 2025.

On how the team succeeded in commercializing a new service developed in-house in less than one year, Hinata reflects, "Not only had we already implemented a search infrastructure through Elasticsearch, but we also had an operational system in place thanks to Elastic Cloud."

The two also praise Elastic's support services. "Since the conception phase of NIKKEI KAI, we received advice from Elastic on questions like what kind of queries we should compose given RAG-based system development, or what we should expect in terms of performance. While testing features ahead of a higher-tier license rollout, we received accurate guidance on configurations possible with the new features. I also believe Elasticsearch's latest feature adapted to RAG contributed to the construction of NIKKEI KAI ahead of other companies."

Mizoguchi also credits Elastic's support in the development and operation processes, saying, "Through tools like chat tools, Elastic provided precise answers that grasped our intentions, even in response to questions that were highly abstract."

Moving forward, the company is considering creating indices of articles that are divided into several sections per piece called "chunks" to enhance the accuracy and speed of search results, as well as introducing a new form of data expression called the "sparse vector model."

"The vector data we are currently using for search functions is extremely heavy and demanding, so it's not realistic from a cost perspective to assign it to every record in our database of over 200 million documents. However, we believe that if we adopt a sparse vector model like ELSER, we can significantly compress information to provide an intuitive search experience for every piece of data we possess that goes beyond simple keyword searches," explains Hinata.

Nikkei's search infrastructure has continued to adopt Elastic's latest technology, steadily paving the path to maximizing the value of article information to the next level.