Neptune's
rkestra

jump to main content

A Newsletter Domain-Specific-Language with Org-Mode

Attention Conservation Notice

An attempt at creating the building blocks necessary for a simple newsletter generating domain-specific-language in org-mode. Not very useful unless you are building a newsletter of sufficient complexity (or are sufficiently stubborn to turn everything into plain text instead of using a GUI).

Preamble

I have been writing a quarterly newsletter for the past couple of quarters. To be more communal, I made the mistake of using $DOMINANT_OS_DISTRIBUTOR's $OFFICE_SOFTWARE newsletter maker $ONE_WHO_PUBLISHES since the folks I write the newsletter with don't code.

Difficulties with $ONE_WHO_PUBLISHES

The program provides an area with which to drag and drop different elements. In (my) practice this amounted to dragging squares all over the place, many of which overlap and obscure each other. When using more than 3+ replicas of a single component style, making changes became cumbersome. An update on one component would mean having to manually update the others. Since all the components share a single workspace, changing one item causes shifts in the placement of other items. With 10+ matching components, this became very time consuming. I'd estimate that I spent more time fiddling with components, moving things out of the way, and moving them back than I did making/adding content. The resulting newsletter ended up being both time-consuming to make and inconsistent in presentation.

A new hope

Being the libre-loving code-monkey that I am, my first instinct was to:

  • never do that again
  • write out something that can be version-controlled and will be consistent
  • escape the clutches of a software ecosystem that may not be relevant in \(n\) years (as technology release velocity increases, \(n \to 0\)) in favor of plain-text and open formats

Goals for the End Product

  • It needs to be usable by non-programmers
  • The build system should not require any special skills or arcane terminal usage
  • The language itself should be pretty straightforward
  • extending and adding functionality should be simple

Final Result (Visual)

Final Result (Code + Stylesheet)

This is the code that is written by the newsletter writer

Click for Newsletter Code
  #+BEGIN_PROPERTIES
  #+TITLE: @@html:<p class="title-p"><span class="title-left">Q3</span><span class="title-right">Newsletter Update</span></p>@@
  #+SUBTITLE: @@html:<p class="subtitle-p"><span class="vol-issue-left">Vol. 1, Issue 3</span><span class="vol-issue-right">30Sep2022</span></p><hr>@@
  #+DATE: <2022-07-17 Sun>
  #+OPTIONS: num:nil author:nil date:nil html-postamble:nil
  #+HTML_HEAD: <link rel="stylesheet" type="text/css" href="newsletter_styles.css"/>
  #+MACRO: split-sentence @@html: <span class="date-str-date"><code>$1</code></span>&emsp;|&emsp;<span class="date-str-str">$2</span><br>@@
  #+MACRO: h3 @@html: <span class="in-column-h3">$1</span><br>@@
  #+MACRO: hr #+HTML: <hr>
  #+FOOTER:
  #+END_PROPERTIES

  * Welcome!
{{{hr}}}

*Here is some text welcoming you!*

Lorem Ipsum is simply dummy text of the printing and typesetting
industry. Lorem Ipsum has been the industry's standard dummy text ever
since the 1500s, when an unknown printer took a galley of type and
scrambled it to make a type specimen book. It has survived not only
five centuries, but also the leap into electronic typesetting,
remaining essentially unchanged. It was popularised in the 1960s with
the release of Letraset sheets containing Lorem Ipsum passages, and
more recently with desktop publishing software like Aldus PageMaker
including versions of Lorem Ipsum.


  * Our Main News Story
{{{hr}}}

  #+begin_banner
./images/nscrop.png
  #+end_banner

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam
mattis orci ultrices enim gravida, vitae venenatis libero
faucibus. Donec at pulvinar sem, ut lobortis elit. Mauris eget pretium
nunc, pretium consequat tellus. Ut justo augue, iaculis id efficitur
ullamcorper, suscipit sit amet sapien. In pulvinar id mauris vel
viverra. Duis vitae nunc consequat, iaculis eros id, cursus nisi. Cras
vitae sem metus. Duis lobortis porta dui, vel volutpat turpis pretium
vitae. Cras aliquet lectus eu felis lacinia euismod. Maecenas eros
velit, pellentesque lacinia ipsum id, imperdiet cursus leo. Nullam
pharetra blandit quam, ut ornare risus gravida quis. Nullam hendrerit
et erat et faucibus. Aenean vitae sagittis mi. Nunc ornare congue
fringilla. Aliquam luctus nisi bibendum orci dignissim, et consectetur
tortor dapibus. Vivamus lacinia accumsan pellentesque.

Fusce nisi neque, auctor eu tellus in, ullamcorper aliquet ipsum. Duis
dui nisi, aliquet consectetur tincidunt non, ultricies id
metus. Maecenas tristique laoreet egestas. Nunc lacinia, tortor id
sagittis scelerisque, mi est viverra quam, at porttitor justo lorem
vel ipsum. Praesent aliquet enim magna. Sed luctus neque nibh, vel
egestas eros ultrices in. Nulla efficitur accumsan nisi, et venenatis
velit fringilla at. Nam vitae magna id neque elementum pellentesque ut
eu lacus. Vivamus sapien libero, elementum aliquam leo id, luctus
pellentesque augue. Donec ac ligula sit amet lacus consectetur
hendrerit. Aenean vitae tempor nulla, at dapibus tortor. In eu congue
quam.

Nulla facilisi. Nam massa metus, suscipit a cursus et, posuere sit
amet mauris. Sed id mi sem. Phasellus placerat augue nisl, sit amet
eleifend elit semper ut. Maecenas eget risus ultricies, feugiat diam
et, auctor est. Integer maximus orci vitae lectus commodo
scelerisque. Fusce a elementum mauris. Maecenas in cursus
ipsum. Pellentesque accumsan tempor sem hendrerit auctor. Mauris
placerat est volutpat sapien luctus congue. Quisque pharetra mauris
quis purus cursus dapibus.

Maecenas molestie vehicula lacus aliquet gravida. Morbi euismod
lacinia enim, at volutpat ex lacinia eget. Quisque in ligula ut metus
mattis luctus. Praesent ut vehicula mi. Vestibulum velit odio,
convallis eu suscipit blandit, convallis quis ex. Nullam sagittis est
vitae diam tristique imperdiet. Fusce vel urna auctor, feugiat libero
vel, bibendum lectus. Curabitur aliquam dolor non faucibus porta.


  * A look at some cats
{{{hr}}}

  #+begin_center
*hello, world!*

All of the cat images in this were generated by https://thiscatdoesnotexist.com/
  #+end_center


  #+begin_twocol
*Fusce nisi neque*, auctor eu tellus in, /ullamcorper aliquet
 ipsum/. _Duis dui nisi_, aliquet consectetur =tincidunt non=,
 ultricies id metus. Maecenas tristique laoreet egestas. Nunc lacinia,
 tortor id sagittis scelerisque, mi est viverra quam, at porttitor
 justo lorem vel ipsum. Praesent aliquet enim magna. Sed luctus neque
 nibh, vel egestas eros ultrices in. Nulla efficitur accumsan nisi, et
 venenatis velit fringilla at. Nam vitae magna id neque elementum
 pellentesque ut eu lacus. Vivamus sapien libero, elementum aliquam
 leo id, luctus pellentesque augue. Donec ac ligula sit amet lacus
 consectetur hendrerit. Aenean vitae tempor nulla, at dapibus
 tortor. In eu congue quam. Maecenas molestie vehicula lacus aliquet
 gravida. Morbi euismod lacinia enim, at volutpat ex lacinia
 eget. Quisque in ligula ut metus mattis luctus.

./images/c1.jpeg
  #+end_twocol


  #+begin_twocol
./images/c2.jpeg

*Mauris efficitur nisi ut ex efficitur, id ultrices felis dignissim?*
Nam fringilla arcu eu mi accumsan, in rhoncus velit auctor.

*Aenean eget lacus dapibus, venenatis arcu sed, egestas nisl?*
Cras quis elit porta, imperdiet turpis ac, luctus dui.

*Sed non ligula porta purus sodales accumsan?*
Integer vestibulum tortor sit amet maximus facilisis.

*Vestibulum pulvinar urna a varius lacinia?*
Nullam porta dolor quis condimentum aliquet.
  #+end_twocol


  ** Pet Trio!
{{{hr}}}

  #+begin_threecol
  #+CAPTION: Choco
./images/c3.jpeg

  #+CAPTION: Latte
./images/c4.jpeg

  #+CAPTION: Chip
./images/c5.jpeg
  #+end_threecol


  * Events
{{{hr}}}

  ** Fitness

| Event Name                  | Date             | Notes         | Emoji |
|-----------------------------+------------------+---------------+-------|
| Doing Yoga with Goats       | <2022-07-18 Mon> | very fun!     | 😺    |
| Weightlifting with Gorillas | <2022-07-25 Mon> | very intense  | 😸    |
| Calisthenics with Ants      | <2022-08-18 Thu> | not even fair | 😹    |

  ** Non-Fitness

| Event Name                 | Date             | Notes      | Emoji |
|----------------------------+------------------+------------+-------|
| Sitting Around with Sloths | <2022-07-26 Tue> | *yawn*       | 😻    |
| Not Doing Much with Bats   | <2022-07-27 Wed> | snooze     | 😼    |
| Cat Naps with Dogs         | <2022-07-28 Thu> | good times | 🙀    |




  * Celebrations
{{{hr}}}

  #+begin_twocol
{{{h3(Set of Chaotic Maps 1)}}}
{{{split-sentence(10 coolness, 3-Cells CNN System)}}}
{{{split-sentence(11 coolness, 2D Lorenz System)}}}
{{{split-sentence(12 coolness, 2D Rational Chaotic Map)}}}
{{{split-sentence(13 coolness, ACT Chaotic Attractor)}}}
{{{split-sentence(14 coolness, Aizawa Chaotic Attractor)}}}
{{{split-sentence(15 coolness, Arneodo Chaotic System)}}}

{{{h3(Set of Chaotic Maps 2)}}}
{{{split-sentence(16 coolness, Arnold's Cat Map)}}}
{{{split-sentence(17 coolness, Baker's Map)}}}
{{{split-sentence(18 coolness, Basin Chaotic Map)}}}
{{{split-sentence(19 coolness, Beta Chaotic Map)}}}
{{{split-sentence(20 coolness, Bogdanov Map)}}}
{{{split-sentence(21 coolness, Brusselator)}}}

{{{h3(Set of Chaotic Maps 3)}}}
{{{split-sentence(22 coolness, Burke-Shaw Chaotic Attractor)}}}
{{{split-sentence(23 coolness, Chen Chaotic Attractor)}}}
{{{split-sentence(24 coolness, Chen-Celikovsky System)}}}
{{{split-sentence(25 coolness, Chen-LU System)}}}
{{{split-sentence(26 coolness, Chen-Lee System)}}}
{{{split-sentence(27 coolness, Chossat-Golubitsky Symmetry Map)}}}

  #+CAPTION: This is artwork made by a GAN
./images/a1.jpeg
  #+end_twocol


  #+begin_footer
This newsletter was built with *Org-Mode*
  #+end_footer

This is the stylesheet that isn't touched by the newsletter writer

Click for Template Stylesheet
@import url(https://fonts.googleapis.com/css?family=Montserrat:400,500,80);

:root {
    --c1: #F2F1E8;
    --c2: #A8DADC;
    --c3: #050533;
    --c4: #1D3557;
    --c5: #E34234;
    --c6: #403D39;
}

body {
    width: 100vw;
    background-color: var(--c1);
    color: var(--c6);
    line-height: 1.4;
    font-size: 18px;
    font-family: 'Montserrat', sans-serif;
}

h1, h2 {
    color: var(--c5);
    font-weight: bold;
}


h3 {
    color: var(--c3);
}


p {
    padding: 20px;
}

.twocol {
    background-color: var(--c2);
    column-rule: dotted var(--c5);
    column-rule-width: thin;
    column-count: 2;
    padding-top: 25px;
    padding-bottom: 25px;
    margin-top: 10px;
    margin-bottom: 10px;
    border-radius: 10px;
}

.twocol img {
    max-width: 100%;
    max-height: 100%;
    border-radius: 10px;
}


.threecol {
    background-color: var(--c2);
    column-rule: dotted var(--c5);
    column-rule-width: thin;
    column-count: 3;
    padding-top: 25px;
    padding-bottom: 25px;
    border-radius: 10px;
}

.threecol img {
    max-width: 100%;
    max-height: 100%;
    border-radius: 10px;
}

.banner {
    padding: 0px;
    margin: 0px;
}

.banner img {
    width: auto;
    padding: 0px;
    margin: 0px;
}

.banner .figure {
    padding: 0;
}

.banner .figure p {
    padding: 0;
}

.timestamp {
    font-size: 18px;
    color: var(--c5);
    font-weight: thin;
}

.top-banner {
    float: right;
}

.title-left {
    float: left;
    color: var(--c2);
    font-size: 64px;
    font-family: 'Montserrat', sans-serif;
    font-weight: bold;
}

.title-right {
    float: right;
    color: var(--c5);
    font-size: 64px;
    font-family: 'Montserrat', sans-serif;
    font-weight: bold;
}

.vol-issue-left {
    float: left;
    color: var(--c3);
    font-weight: thin;
    font-family: 'Montserrat', sans-serif;
}

.vol-issue-right {
    float: right;
    color: var(--c3);
    font-weight: thin;
    font-family: 'Montserrat', sans-serif;
}

.title-p {
    padding-top: 0px;
    padding-bottom: 1px;
}

.subtitle-p {
    padding: 8px;
}

hr {
    border-top: 1px dotted;
    border-bottom: 0px;
    border-color: var(--c5);
}

a {
    color: var(--c3);
    text-decoration: none;
}

a:hover {
    color: var(--c4);
}

.date-str-date {
    font-weight: thin;
    color: var(--c5)
}

.date-str-str {
    color: var(--c4)
}

.footer {
    float: right;
}

.in-column-h1 {
    font-weight: bold;
    font-size: 36px;
}

.in-column-h2 {
    font-weight: bold;
    font-size: 28px;
}

.in-column-h3 {
    font-weight: bold;
    font-size: 20px;
}

.figure-number {
    display: none;
}

table {
  border-collapse: separate;
  border-spacing: 0;
}

table tr th,
table tr td {
  border-right: 1px solid #bbb;
  border-bottom: 1px solid #bbb;
  box-shadow: 2px 2px 1px #e5dfcc;
}

table tr th:first-child,
table tr td:first-child {
  border-left: 1px solid #bbb;
}

table tr th {
  border-top: 1px solid #bbb;
}

/* top-left border-radius */
table tr:first-child th:first-child {
  border-top-left-radius: 0px;
}

/* top-right border-radius */
table tr:first-child th:last-child {
  border-top-right-radius: 0px;
}

/* bottom-left border-radius */
table tr:last-child td:first-child {
  border-bottom-left-radius: 6px;
}

/* bottom-right border-radius */
table tr:last-child td:last-child {
  border-bottom-right-radius: 6px;
}

th, td {
    padding: 8px 20px;
}

th {
    background: var(--c2);
    color: var(--c4);
}

td {
    background: var(--c1);
}

The Secret Sauce

Below you will find an overview of the different components and the related bits of org-markdown-language created. Before we get into that jazz, let's zoom out and ask:

What is the secret sauce? How does it all work?

3 big things allow this to work:

  1. Org-mode has a ton of functionality already built into it, including different types of markup and export options
  2. Special blocks can be created on the fly, giving the user a container object to interact with instead of HTML/CSS/Javascript. These can then be styled with CSS
  3. If functionality needs to be added to a component, org lets the user create macros that expand at export time. These can inject HTML, CSS, or emacs-lisp into the export

Org-mode functionality

Org comes with a lot of functionality out of the box. You'd be well served to look at the org syntax specification

The most important part for the end-user (of the newsletter generator) is to know the very basics:

# Headers
  * This is a heading
  ** This is a subheading
  *** etc etc

# Lists
 1. item 1
 2. item 2
 3. item 3

 - item 5
 - item 6
 - item 7

# Inline styles
  *bold*
  /italic/
  _underline_
  =monospace=
  ~code~
  +strike-through+

Special Blocks

For every component I want to add to the newsletter, I simply make a corresponding org-mode block like so:

#+begin_component-name
some content here
#+end_component-name

At export time, this turns into:

<div class="component-name">
  <p>some content here</p>
</div>

Here is the relevant section from the org-manual:

When special blocks do not have a corresponding HTML5 element, the HTML exporter reverts to standard translation (see org-html-html5-elements). For example, #+BEGIN_lederhosen exports to <div class="lederhosen">

Since org export turns things it doesn't recognize as HTML tags into a div with a class, I can then style the class with CSS in the style.css file.

Extra details

This also works particularly nice when you use a tag that is found in html5. As an example,

#+BEGIN_aside
  Lorem ipsum
#+END_aside

exports to:

<aside>
  <p>Lorem ipsum</p>
</aside>

Functionality with Macros

If I want to add functionality to something, I can use a macro that creates HTML tags (or css, elisp, etc) to implement the functionality.

Macros allow org to replace text during export. Here is an example from the org-manual:

#+MACRO: poem Rose is $1, violet's $2. Life's ordered: Org assists you.
{{{poem(red,blue)}}}

becomes

Rose is red, violet's blue.  Life's ordered: Org assists you.

It even allows the use of emacs-lisp snippets, but I haven't needed to use that functionality here (yet).

For the newsletter I use the following 3 macros (defined in the BEGIN_PROPERTIES block):

#+MACRO: split-sentence @@html: <span class="date-str-date"><code>$1</code></span>&emsp;|&emsp;<span class="date-str-str">$2</span><br>@@

#+MACRO: h1 @@html:<span class="in-column-h1">$1</span><br>@@

#+MACRO: hr #+HTML: <hr>

This syntax:

@@html: stuff here@@
#+HTML: tag

allows org to directly place html into the exported output (instead of running it through its preprocessor).

These are all used throughout the newsletter using the following syntax:

# example
#+MACRO: something <h1>Write $1 Write $2</h1>
{{{something(arg1, arg2)}}}
<h1>Write arg1 Write arg2

# actual usage
{{{split-sentence(left hand side, right hand side)}}}
{{{h1(sentence here)}}}
{{{hr}}}

where something is the macro name and the args are replacing the $1 and $2 anchors.

Build Tool

The build tool is very simple. The idea is the following:

  1. Write the newsletter as newsletter.org
  2. Build the newsletter with emacs through an executable script

This way the end user only has to do 3 things to get going:

  1. Write the file in emacs
  2. Build the file by clicking a shortcut
  3. Inspect the results in the browser

On Windows

Since emacs is very portable on Windows (you can download a self-contained program in a folder here) each user can get access to it without much trouble.

To create an executable

Then the build tool can be put together using a little bit of elisp and powershell

Here is the file create-html.el

#!/home/neptune/.guix-profile/bin/emacs --script

(progn (find-file "newsletter.org") (org-html-export-to-html) (kill-buffer))

(browse-url--browser "newsletter.html")

Then the user can make the file executable and run the command

./create-html.el

Additional Notes

Why HTML? Why not pdf?

While there are ways of creating PDF content with org-mode, getting all the styling and spacing to line up seemed like an extra hurdle for little gain. I find it much easier to parse and tinker with HTML pages than PDF pages.

The images in the newsletter

The images in the newsletter were all generated by various Generative Adversarial Networks.


Components Overview

main banner

banner

Syntax

To get the banner to show up, I needed to build it out of a <span> in the title of the org document. When org-export as html is called, org-mode writes the content to both the header <title> tag as well as a standalone <h1> tag at the top of the page.

As a result, we can inject html into the title (and subtitle) like so:

#+TITLE: @@html:<p class="title-p"><span class="title-left">Q3</span><span class="title-right">Newsletter Update</span></p>@@
#+SUBTITLE: @@html:<p class="subtitle-p"><span class="vol-issue-left">Vol. 1, Issue 3</span><span

The caveat here is that this code is also copied into the <title> in the header, meaning that our page titlebar will be @@html...

I'm still uncertain about how to fix this.

table of contents stylized

yes

Syntax

This comes for free with org-export to html. As a default org export also creates unique anchor points for each of the headers and subheaders. This can be customized to use the verbatim text as an anchor point as well. In our case, it doesn't matter too much what the anchor wording is.

If you want to get more readable link anchors, you can use the following emacs lisp snippet that I took from this excellent blog post:

(defun my/ensure-headline-ids (&rest _)
"All non-alphanumeric characters are cleverly replaced with ‘-’.
If multiple trees end-up with the same id property, issue a
message and undo any property insertion thus far.
"
  (interactive)
  (let ((ids))
    (org-map-entries
     (lambda ()
       (org-with-point-at (point)
         (let ((id (org-entry-get nil "CUSTOM_ID")))
           (unless id
             (thread-last (nth 4 (org-heading-components))
               (s-replace-regexp "[^[:alnum:]']" "-")
               (s-replace-regexp "-+" "-")
               (s-chop-prefix "-")
               (s-chop-suffix "-")
               (setq id))
             (if (not (member id ids))
                 (push id ids)
               (message-box "Oh no, a repeated id!\n\n\t%s" id)
               (undo)
               (setq quit-flag t))
             (org-entry-put nil "CUSTOM_ID" id))))))))

;; Whenever html & md export happens, ensure we have headline ids.
(advice-add 'org-html-export-to-html   :before 'my/ensure-headline-ids)
(advice-add 'org-md-export-to-markdown :before 'my/ensure-headline-ids)

section banner

section banner

Syntax

This is the first case of using a special code block to achieve our desired formatting. In this case, it is important to note that banner is not a part of the default org syntax

#+begin_banner
./images/nscrop.png
#+end_banner

What is happening here is detailed in the special sauce section. Even though org doesn't have a banner tag, org makes a div with class "banner".

Here is the relevant CSS for the banner image:

.banner img {
    width: auto;
    padding: 0px;
    margin: 0px;
}

prefix | sentence

pref<sub>sent</sub>

Syntax

This is a simple macro that expands to html at export time.

#+MACRO: split-sentence @@html: <span class="date-str-date"><code>$1</code></span>&emsp;|&emsp;<span class="date-str-str">$2</span><br>@@

This provides syntax like the following:

{{{split-sentence(first argument, second argument)}}}

which expands to the following at export time

<span class="date-str-date"><code>first argument</code></span>&emsp;|&emsp;<span class="date-str-str">second argument</span><br>

We can add a class to style it as well:

.date-str-date {
    font-weight: thin;
    color: var(--c5)
}

.date-str-str {
    color: var(--c4)
}

left-image

left-img

Syntax

This creates a div with the class id twocol. Where the magic happens is in the css file. With css, I can split the div into n columns and style the intersecting border:

.twocol {
    background-color: var(--c2);
    column-rule: dotted var(--c5);
    column-rule-width: thin;
    column-count: 2;
    padding-top: 25px;
    padding-bottom: 25px;
    margin-top: 10px;
    margin-bottom: 10px;
    border-radius: 10px;
}

.twocol img {
    max-width: 100%;
    max-height: 100%;
    border-radius: 10px;
}

Then, when I enter text into a twocol block, it will split the content into the first set of content and the second set.

  #+begin_twocol
an image goes here

some text accompanying it goes here
  #+end_twocol

This is also generalized in the right image and triple-image components

right-image

right-img

Syntax

This is the same thing as the left image component, except we add the image to the block second:

  #+begin_twocol
some text accompanying the image goes here

an image goes here
  #+end_twocol

triple-image

triple-img

Syntax

This is the same as left-image, except we use css to create 3 images.

.threecol {
    background-color: var(--c2);
    column-rule: dotted var(--c5);
    column-rule-width: thin;
    column-count: 3;
    padding-top: 25px;
    padding-bottom: 25px;
    border-radius: 10px;
}

.threecol img {
    max-width: 100%;
    max-height: 100%;
    border-radius: 10px;
}

This could also house content like image | text | image or any 3-tuple of image and text inputs. In the future, if needed, I could add any n-tuple.

headers inside of columns

inside-header

I wanted to keep the functionality sufficiently separated from the component itself, i.e. don't hardcode the header transformation into the underlying component. To do this, I used an org-macro (see The Secret Sauce > Functionality with Macros) and allow the user to invoke it when desired.

Syntax

#+MACRO: h3 @@html:<span class="in-column-h3">$1</span><br>@@

{{{h3(Set of Chaotic Maps 1)}}}

image captions

img-caption

This comes for free with org-mode. Simply add the #+CAPTION tag above an image

Syntax

#+CAPTION: This is artwork made by a GAN
,./images/a1.jpeg

tables

tables

Tables are super easy to use in org-mode. I usually just write \(n\) | characters in a row and |- like so for a 3-column table:

||||
|-

and then hit <tab> and the table is created like so:

|   |   |   |
|---+---+---|
|   |   |   |



▽ Comments