Engineering a new Kibana dashboard layout to support collapsible sections & more

We are developing collapsible sections to hide and show panels in your Kibana dashboards to help organize content and improve performance. It’s a classic software development tale: sometimes to go forward you have to go…down? Read about how building an in-demand feature that seemed straightforward can sometimes lead you to bigger simplifications than you ever intended! 😅

Collapsible sections

A little bit of background: Dashboards in Kibana can contain many visualizations that Site Reliability Engineers (SREs) use to keep systems running or Security Analysts use in their investigations. These dashboards can be lengthy and slow to load. Users want to better organize dashboard content to avoid performance pitfalls and make them easier to scan. Today, the best way to accomplish this is to split dashboard content into multiple dashboards and then link them using the dashboard links panel to facilitate navigation. This unfortunately doesn’t let you see things side-by-side and makes updates and dashboard maintenance require a lot of effort from the dashboard author. To solve this need, we are developing collapsible sections to hide and show panels in your Kibana dashboards–these sections help organize content and don’t load content that is collapsed to improve performance.

These new sections will allow you to group dashboard panels and data visualizations that are thematically related making it easier to find the information you are looking for. Most importantly, you can easily hide and expand these sections allowing you to load only the data that you need. This will help you create side-by-side comparisons for your charts and streamline dashboard performance.

Dashboards in Kibana Collapsible sections

Planning the engineering approach

At the onset when looking at what our customers wanted the feature seemed like a business-as-usual-sized engineering effort: Dashboards contain panels (more about those in a moment) and they are to be organized into sections and the product requirements ask that we only render them when a section is open. There’s also a drag and drop system to lay out a dashboard and it needs to account for these sections and handle a variety of moving-things-between-sections sort of use cases. Seems well in hand as an enhancement to existing code, right?

Well unfortunately after a short proof of concept we found the answer is no. It’s not that simple. Kibana uses an “embeddable” framework and this framework lacks the qualities needed to not render certain embedded objects on a dashboard. Let's take a look at why…

What is an “embeddable”?

Even though "embeddable" does not appear in the navigation menu alongside "Discover" and "Dashboard", you interact with embeddables throughout Kibana. The histogram in Discover, each panel in a Dashboard, a panel’s context menu, a Lens chart in Observability, or a Map in Security - all made possible with embeddables.

Embeddables are React components that provide an API to deeply integrate with Kibana. This API allows them to be persisted and restored by any page, gives them access to the current search context, allows them to define editing UI, and is extensible so engineers can define how components interact with one another. They live in a registry, which separates their behaviours from where the code is written. Because of this, many engineers can work on different embeddables at the same time without getting in each other’s way.

The need for a new embeddable system

The legacy embeddable system we were working on at the time dates back to 2018. embeddable functionality is exposed through a custom user experience component abstraction. At the time, Kibana was transitioning from Angular 1 to React, so the embeddable system was designed to be framework agnostic which could smooth a theoretical transition away from React. While the architecture was required at the time, Kibana has changed a lot since then, and a move away from React is unlikely. Now, the inflexible and agnostic Embeddable architecture is a growing point of friction. Some pain points are:

Complex state management: All state in an embeddable goes through one of two observables (input, output) in order to be inherited, set, or read. This requires consumers to set up complex two-way sync pipes.
Limited inheritance: embeddables can have exactly one parent, limiting inheritance to a single level of hierarchy. Additionally, embeddable state flows from the parent to child, with child state overriding parent state if defined.
Manual rendering: embeddables need a cumbersome manual render process and a compatibility layer between the rest of Kibana, which renders via React.

Collapsible sections are not possible with a single level of hierarchy. Collapsible sections require multiple levels of hierarchy to allow panels to belong to the dashboard and a collapsible section. Otherwise, you wouldn’t be able to place panels into a collapsible section.

New embeddable system

So, to deliver this feature, we actually had to go “down” to the embeddable system itself and modernize how we manage embeddables: We had to design a new embeddable system. Fun! But also…..scope!

The new embeddable functionality is exposed through plain old JavaScript objects and can compose their functionality by implementing interfaces. For example, an embeddable can communicate data loading by implementing the PublishesDataLoading interface. This offers the following benefits:

Clean state management: Each piece of state is exposed as a read-only observable. Setter methods can be exposed for mutable state.
Flexible inheritance: embeddables can have a chain of parents, allowing for as many levels of hierarchy as required. Each layer retains its own state so that the decision of which state to use can be determined at the time of consumption.

With a system that tolerates the inheritance we need, collapsible sections can now be built.

However, like any good refactor there’s a bit of a catch: embeddables are everywhere in Kibana and to implement this change without causing regressions we needed to migrate to the new embeddable system across Elastic’s full experience–from the Alerts page in Elastic Security to the Service Inventory in Elastic Observability and nearly everything in between. This has taken us some time but allows for some exciting new possibilities.

New layout engine

The driving force behind any Dashboard is the layout engine, which is the thing that allows panels to be dragged around and resized — without it, Dashboards would be entirely static (and boring)!

Currently, Kibana uses the external react-grid-layout package to drive our Dashboards, which is an open-source layout engine managed by a small group of volunteers. This layout engine has worked great for our Dashboards up to this point; however, it is unfortunately missing critical features that would make collapsible sections possible out-of-the-box: either “panels within panels” or dragging panels across two separate instances of a layout.

Due to the small team behind react-grid-layout, updates to the package are infrequent — this means that, even if we started contributing directly to react-grid-layout in order to add the features we need, incorporating these changes into Kibana Dashboards would be slow and unreliable. While we briefly considered making a Kibana-specific branch of react-grid-layout in order to get updates published at a pace that matched our development, the maintenance costs and inflexibility of this ultimately led us to discard this idea.

After researching alternative layout engine packages, we decided that the best path forward would be to develop our own, internal layout engine — one that was built specifically with the Kibana Dashboard use case in mind! Work on this new layout engine, which we are calling kbn-grid-layout, has already started. To our knowledge, this is the first layout engine available that makes use of the native CSS grid in order to position its panels — all other layout engines that we found in our research relied on pixel-level transforms or absolute positioning. This makes it a lot easier to understand how panels are placed on a dashboard.

kbn-grid-layout uses passive event handlers for all dragging and resizing events, with an emphasis on reducing the number of re-renders to a minimum during these actions to improve performance. Because we are in control of these event handlers, this allows us to focus on the user experience much more than we previously could, and we’ve added features such as auto-scrolling when dragging near the top or bottom of the screen, and locking the height of the grid during resize events to prevent unexpected behavior that could result from the browser responding to height changes before the resize event was complete.

We are currently working on refining the implementation, which includes improving the management of collapsible sections, adding keyboard support for dragging and resizing (which is not currently supported by Kibana dashboards), and much more. Not only will this new layout engine unlock the ability to add collapsible sections, it is being built with accessibility and efficiency at the forefront — which means the entire Dashboard experience should be improved once we make the final layout engine swap from react-grid-layout to kbn-grid-layout!

react-grid-layout

<ResponsiveReactGridLayout
  cols={{
    lg: 48,
    sm: 1,
  }}
  layouts={layouts}
  className={classes}
  width={viewportWidth}
  breakpoints={breakpoints}
  onLayoutChange={
    onLayoutChangeCallback
  }
  isResizable={
    !expandedPanel && !focusedPanel
  }
  isDraggable={
    !expandedPanel && !focusedPanel
  }
  rowHeight={20}
  margin={useMargins ? [8, 8] : [0, 0]}
  draggableHandle={'.embPanel--dragHandle'}
  useCSSTransforms={false}
>
  {panelComponents}
</ResponsiveReactGridLayout>

kbn-grid-layout

<GridLayout
  layout={currentLayout}
  gridSettings={{
    gutterSize: useMargins ? 8 : 0,
    rowHeight: 20,
    columnCount: 48,
  }}
  renderPanelContents={
    renderPanelCallback
  }
  onLayoutChange={
    onLayoutChangeCallback
  }
  expandedPanelId={expandedPanel}
  accessMode={accessMode}
/>

Check it out before the release

We’re nearly out of the embeddable woods and ready to enjoy the fruits of our labors with all of our customers from weekly-releasing Elastic Serverless to our selfhosted users. Our customers will be able to design a single dashboard with many sections that can be collapsed by default allowing an investigation to only load panel content that’s needed while keeping lengthy dashboards tidy. If you want to provide us feedback or sign up for early testing please let us know! We will announce when this feature is ready to be used in the next few months. Stay tuned!

Ready to try this out on your own? Start a free trial.

Want to get Elastic certified? Find out when the next Elasticsearch Engineer training is running!