Elastic and Tencent Cloud AI search technology make Dunhuang Digital Scripture Cave accessible to everyone

Ancient books in the Dunhuang Digital Scripture Cave have now become accessible to everyone online. The Dunhuang Research Institute officially released the Digital Scripture Cave database, including more than 9,900 volumes of Dunhuang documents and more than 60,700 images that contain Buddhist scriptures, legal codes, contracts, silk paintings, and more.

With the help of Elastic and Tencent Cloud AI search and large language model (LLM) technology, traditional Chinese and rare characters in ancient books have been awakened. They’ve become searchable and translatable instead of obscure and difficult to understand.

The Dunhuang Digital Scripture Cave has brought civilization deep into the caves of everyone's eyes.

In the past, approaching a volume of Dunhuang ancient books required flipping rubbings and reading annotations. Now, these books can be opened in the Dunhuang Digital Scripture Cave with the help of AI.

But with the support of AI technology, obscure Buddhist scriptures become clear and readable. The Diamond Sutra homepage automatically extracts the summary of the vernacular version:

“The Buddha told Subhuti that all conditioned dharmas are like dreams and bubbles, and should not be attached to form. Bodhisattvas should have no self-image, human image, sentient beings image, or life image, and practice charity with a non-dwelling mind to achieve enlightenment. The true Dharma cannot be described in words. Only by transcending concepts and distinctions can the essence of Tathagata be seen.”

AI intelligently summarizes the background knowledge of the Diamond Sutra, introducing the historical status, core ideas, structural characteristics, and cultural influences.

The platform also supports translation between Chinese, English, French, Japanese, and other languages. In the AI assistant, users can input any four languages, and it will communicate with you in the same language — switching between Chinese, English, French, and Japanese without barriers. One more language means one more way for the world to understand Dunhuang.

The transformation from thousand-year-old fragments to digital interpretation is based on a set of AI search technologies tailored for ancient books. Tencent uses Tencent Cloud Elasticsearch Service (ES) and LLM to build a combination of retrieval augmented generation (RAG) capabilities to provide fast and accurate responses to the needs of sorting and searching massive ancient book information.

Whenever a user enters a question, the system will find all the literature information related to this question and then use the powerful generation capabilities of the LLM to provide users with accurate answers.

The Tencent Cloud ES team addressed this challenge by developing the ancient Chinese tokenizer. In collaboration with Dunhuang experts, they refined tokenization rules for complex terminology, creating a specialized system tailored for Dunhuang manuscripts. This system transforms obscure texts into machine-readable tokens optimized for AI-driven semantic search.

Phrases like "佛在舍卫国祗树给孤独园 (Buddha in Savatthi Jetavana Grove)" are now accurately tokenized as “佛 / 在 / 舍卫国 / 祗树 / 给孤独园 ( Buddha / in / Savatthi / Jetavana Grove),” preserving contextual meaning.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use. 

Elastic, Elasticsearch, and associated marks are trademarks, logos, or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.