Scapers

As users post messages containing links in Weavy, we attach simple previews, adding context to the conversations. When a link is spotted, Weavy crawls it and provides a simple preview. We call this process unfurling.

Unfurled previews in Weavy are handy because they give everyone a glimpse of what someone linked to. A preview saves lots of time throughout your day, as you decide whether or not something is worth reading, or better skipped over. They also help with search and your own recollection of what you’ve seen in Weavy.

Unfurling

When we crawl a link in Weavy, we "scrape" up information from the headers of the web page in order to create a simple preview. This is very similar to the way link unfurling works in applications such as Slack, Twitter, Facebook, etc.

When you paste a URL into a Weavy message, Weavy will fetch the URL, and start going through the headers of the page. Weavy reads them in this order:

  1. oEmbed
  2. Twitter card tags / Facebook Open Graph tags
  3. HTML meta tags

oEmbed

oEmbed is a standard for sharing metadata with other sites and services. You basically run your own standalone JSON server that responds to requests by giving out up to about a dozen parameters describing your content. It works best for multimedia sites (photos work best, audio and video to a lesser extent), and was designed specifically to furnish custom previews for other services.

Twitter card

Twitter developed a system of embedding metadata to customize how sites look when previewed within tweets. The full tag reference gives a good overview of all the different card types and options.

Facebook Open Graph

Similar to Twitter, Facebook developed Open Graph as a system to help other sites customize how their content is previewed on Facebook’s site and apps. Their basic markup reference covers everything you’d need for customizing your own unfurl (as well as how your sites will appear on Facebook).

HTML meta tags

Finally, we have the HTML meta tag. In absence of any other options, Weavy will read the description attribute and display that in an unfurl.

Custom scraper

To further customize the unfurling process preview, you can register a custom scraper by implementing the IScraper interface.