在同一場景結合 Gaussian Splats 與 BIM：整合過程遇到的問題

Our BIM viewer originally dealt with walls, beams, equipment, and pipes. These objects had geometry, but also element IDs, properties, and a hierarchy. Adding Gaussian splatting introduced a very different kind of data: it could show what a site looked like, without automatically becoming a collection of engineering components whose properties we could query.

Putting the two together was appealing. A reconstruction from photographs could provide the surroundings and a sense of the space, while BIM retained the component information in the design model. Once we started integrating them, though, the work quickly went beyond loading another file format.

An earlier article covered stalls and blur in Gaussian rendering. This one looks back at the initial integration into our native WebGPU viewer: how we aligned coordinates, established occlusion, framed the whole scene, and decided which existing component operations still applied.

Figure 1. The kind of data we were adding: the public Knock Community Hall sample rendered in our viewer, retaining 1,935,275 Gaussians. The model was published by scbenoit (SteveB) under CC BY 4.0. We decoded SOG to PLY before rendering it. This image shows only the Gaussian input; the later images use a separate synthetic scene to test occlusion between Gaussians and BIM.

Align the coordinate conventions first

A BIM file and a Gaussian file can each display correctly on their own and still fail to line up when placed in the same scene.

Gaussian data often uses local coordinates established during reconstruction. An engineering model has its own origin, orientation, and units. Sharing a camera cannot remove those differences. We first need to establish which direction each axis represents and how much physical distance one unit corresponds to.

Our integration path first converts the source Gaussian coordinates to the viewer's Z-up convention, then applies the scene member's placement matrix. Translation, rotation, and scale are expressed at this layer, so the original Gaussian file can remain unchanged.

The integration layer therefore places the data using a supplied transform. The registration between the two models still has to be determined beforehand. Where the same wall corner lies in each dataset, and whether their physical scales match, are questions that do not disappear just because both files have loaded.

For engineering scenes with large world coordinates, we also subtract the same local origin in both rendering paths, letting GPU calculations operate on smaller relative coordinates. If each path chooses its own origin without carrying that difference through the pipeline, the two datasets still will not occupy the same space.

Figure 2. The main relationships in a mixed frame. Opaque depth produced by BIM participates in Gaussian rendering, and the colors from the two paths are composited.

Sharing a camera also means sharing depth

The next task was to make the front-to-back relationships work.

Suppose a group of Gaussians sits behind an opaque wall. The wall should hide it. When the camera moves to the other side and the Gaussians are in front, they should become visible again. Simply stacking two rendered images in a fixed order cannot guarantee that relationship.

We let BIM write the depth of opaque objects first. Gaussian rendering then uses that depth to determine whether a fragment lies behind geometry already drawn. Both paths need the same camera, canvas dimensions, and depth convention for those comparisons to mean the same thing.

We checked this with a deliberately simple synthetic scene: a gray wall as the BIM occluder, with red Gaussians on one side. Simple shapes made it harder for complex textures to hide an error.

Figure 3. A screenshot from the running test. The red region consists of Gaussians; the gray region is opaque BIM test geometry. From this side, the Gaussians are visible.

Figure 4. Looking from the other side hides the Gaussians. Hiding the gray occluder in the test makes the red Gaussians reappear. Figures 3 and 4 both show the synthetic validation scene.

The Gaussians also have their own rendering requirements. 3D Gaussian Splatting represents a scene using Gaussians with opacity, whose projected contributions must be blended in order. We put the visible Gaussian members into one sorting process and draw them from back to front, so the result does not depend on the order in which models were loaded.

The Gaussians participate in depth testing, but do not write to that opaque depth buffer. If a translucent Gaussian with a soft edge claimed depth as though it were a solid wall, it could incorrectly hide later contributions.

The key steps of a frame can be reduced to this pseudocode:

const camera = scene.camera;
const { color: bimColor, depth } = drawOpaqueBim(camera);
const ordered = sortBackToFront(visibleSplats, camera);
const splatColor = drawSplats(ordered, {
  camera,
  depthAttachment: depth,
  depthLoadOp: "load",
  depthCompare: "less-equal",
  depthWriteEnabled: false,
  blend: "premultiplied-alpha",
});
compositeOver(bimColor, splatColor);

This leaves out GPU resource creation and antialiasing configuration. It keeps the three relationships that matter here: a shared camera, existing depth, and alpha blending.

At this point, opaque BIM could correctly occlude the Gaussians. Transparent components such as glass needed separate handling. They also participate in alpha blending, and sharing opaque depth alone cannot resolve their ordering with Gaussians. That integration did not yet implement a global transparent sort covering both glass and Gaussians.

Framing the whole scene must include the Gaussians

Once the images could be combined, some previously routine features became worth checking again.

One example was the command to frame the entire scene. With BIM alone, computing the camera position from the BIM bounds was sufficient. After adding Gaussian members, the scene bounds needed to include them as well. Otherwise, both datasets could be loaded while the camera continued to frame only one of them.

We built a test that placed the Gaussians far from the BIM geometry and then checked the full scene bounds. This forced us to include Gaussian bounds in camera framing and near/far clipping calculations, rather than merely checking that they displayed correctly near the origin.

Figure 5. A diagram of the combined bounds, not a rendering screenshot. The green dashed rectangle encloses both types of data. Framing only the BIM on the left could leave the Gaussians on the right outside the view.

The combined bounds also have a specific purpose. The camera needs to know the size of the whole scene, while BIM culling and shadow calculations retain their own bounds. Adding distant Gaussian data does not mean that every BIM calculation should expand to cover it.

Rendering together does not make the interaction capabilities identical

At the interaction layer, it was easy to make another assumption: if everything on screen is 3D, should clicking it open properties, allow hiding by floor, or support distance measurements?

In our integration path, the answer depends on the data type.

A BIM element ID can connect geometry to properties and a hierarchy. Ordinary Gaussian data contains rendering information such as position, shape, color, and opacity. It does not inherently carry a business identity such as "this is a beam" or "this belongs to the third floor." A wall that looks continuous in the image is not necessarily a geometric surface available for calculations.

We therefore manage Gaussians as independent scene members. They can have an initial placement, be shown or hidden as a whole, and participate in scene framing. Element picking, property queries, and hierarchy operations continue to come from BIM data. Measurement and sectioning of Gaussian surfaces did not automatically arrive with the ability to display them.

The API needs to express that difference too. The application configures its actions from the capabilities returned by each member, rather than enabling every button whenever it sees 3D content:

function actionsFor(member) {
  const caps = member.capabilities;
  return {
    toggleVisible: caps.visibility.supported,
    pickElement: caps.picking.supported,
    showProperties: caps.properties.supported,
    browseHierarchy: caps.hierarchy.supported,
  };
}

For the Gaussian members described here, whole-member visibility is supported, while the other three component operations are not. A capability flag says that the interface supports an operation; it does not guarantee that a particular BIM file contains complete properties or hierarchy data.

This gives the two data types a clear division of work: Gaussians help people understand the site's appearance, while BIM anchors operations to identifiable components. Connecting a region in the captured site to a specific piece of equipment will require a data correspondence, beyond further changes to rendering code.

Scene switching is part of a complete integration

The final part of the work often stays outside the screenshot.

Loading and refining Gaussians creates asynchronous tasks. A user may enter a scene and immediately leave it, or destroy the entire viewer while new data is being uploaded. Requests already in flight, decoded results, and GPU resources all need a clear owner.

We tie loading, decoding, and rendering resources to the scene session's lifecycle. Disposing the scene cancels its tasks. Late results must be released even if they have already created GPU resources, and must not re-enter a scene the user has left. If initialization fails partway through, buffers allocated earlier also need to be reclaimed. Member visibility is handled separately: hiding is not disposal.

During refinement, we keep the current drawable resources until the next set is fully ready, then replace them. The content already on screen can keep rendering while preparation continues. If building the new resources fails, the current image and statistics should still correspond to the batch that was successfully committed.

The following example expands one replacement. Each scene session builds new resources serially, while the old resources continue rendering:

async function replacePass(data, session) {
  let candidate = null;
  try {
    candidate = await buildPass(data, session.signal);
    if (session.disposed || session.signal.aborted) return;
    candidate.setVisibility(session.memberVisibility);
    const previous = session.pass;
    session.pass = candidate;
    session.committedCount = data.count;
    candidate = null;
    previous?.dispose();
  } finally {
    candidate?.dispose();
  }
}

If buildPass fails internally, it must release the resources it has already allocated. If the build succeeds after the scene has been left, finally releases the unused candidate. Ownership of that candidate passes to the scene only when it successfully replaces the old resources.

To check the integration, we moved the camera to the other side of the test wall, hid the wall, and then switched the entire scene. Those ordinary operations checked spatial relationships, occlusion, and resource management together. Once Gaussians enter a BIM viewer, they need to withstand those operations alongside the original models.