Use cases from Paper, a DOM-based design tool

[StephenHaney]

Hello! We make Paper, which is a browser-based design tool, a bit like Sketch or Figma. We use DOM to render, which lets designers do cool things like design with shaders, import their actual code components, and design with anything else HTML can do (like video embeds).

I'm pretty excited about html-in-canvas because it might help us solve three of our bigger challenges:

1. Our eye dropper is very limited to the EyeDropper API. We can't control the color format of the preview. And we can't update anything as the user hovers, because we don't find out about the color until they click.

2. Exporting a design reliably as a PNG is currently impossible (without an electron wrapper). The best solution at the moment is dom-to-image-more, but all SVG/canvas based approaches will miss some CSS (like backdrop-filter). The methods of taking a screenshot with browser APIs or extension APIs all have flaws that make them not good enough for professional designer use.

3. This is already in your design, but applying our WebGL filters to underlying HTML content (e.g., "liquid glass") would be a major unlock. Right now our shaders let users upload images, but obviously can't filter HTML content underneath them (without expensive and imperfect workarounds like dom-to-image-more)

My hope would be that we can reliably render our DOM into a canvas element so that we can sample pixel data exactly as the designer sees it. Both for eye dropper and PNG file export use cases.

I hope it's helpful to hear about our use cases. Great API idea, we're excited to try it out in many ways.

(screenshot of Paper for context)

[utopyin]

That explains why the eye dropper is not available on the web version of Framer! Tools architected around the DOM would benefit so much from html-in-canvas, I can’t think of a proposal I’ve been more excited about.

[schenney-chromium]

Thanks for the use cases. We really appreciate it because it helps us motivate browser vendors to support the proposal.

I think the proposal as-is will broadly meet your use cases, with the only caveat being restrictions to prevent privacy leaks and reduce fingerprinting risks. See the Privacy Preserving Painting doc for more info on how that might impact you.