Skip to main content

Chapter 2 · Programming

How to Design a Programming Language from Scratch

Design a small statically typed C-style programming language from scratch, including its syntax, types and grammar, then build its compiler in C++ with CMake and Docker.

By Updated 6 min Intermediate

Illustrated compiler pipeline showing source code passing through analysis and code-generation stages before becoming machine code.
On this page

Before building a compiler in C++, we need a precise programming-language design. In this chapter, we will define the syntax and type system of Si, write its grammar and prepare a repeatable development environment.

What You Will Learn

By the end of this chapter, you will be able to:

  • Define the syntax and type system for the Si programming language.
  • Write a context-free grammar for functions, statements and expressions.
  • Set up a C++ compiler project with CMake and Docker.
  • Create a stable language specification for later compiler stages.

Designing a Programming Language: Si

We are designing a small, statically typed, C-style language called Si, a name inspired by “C” and “simple”. The following specification will guide every later compiler stage, so the syntax and semantics must remain consistent.

Programming Language Structure

Every Si program must have a main function, which serves as the program’s entry point. The compiler and runtime will eventually arrange for execution to reach main when the program starts.

fn main(): int {
    // Code goes here
}

Designing the Type System

To keep our Lexer (the part that reads our text) simple, we will use the let keyword for all declarations. This ensures the compiler always knows a new variable is coming the moment it sees that keyword.

  • Types: int (32-bit integer), bool (true/false).
  • Syntax: let <name>: <type> = <expression>;

Writing the Context-Free Grammar

For the computer to understand our language, we define it using a Context-Free Grammar. Don’t be intimidated by the notation below; it’s just a formal way of saying “a function is made of a name, followed by its parameters, a return type, and curly braces containing the logic.”

<program> ::= <function>+
<function> ::= "fn" <id> "(" <params>? ")" (":" <type>)? <block>
<params> ::= <param> ("," <param>)*
<param> ::= <id> ":" <type>
<type> ::= "int" | "bool" | "void"
<block> ::= "{" <statement>* "}"
<statement> ::= <declaration>
              | <assignment>
              | <if_stmt>
              | <while_stmt>
              | <return_stmt>
              | <expr_stmt>
<declaration> ::= "let" <id> ":" <type> "=" <expression> ";"
<assignment> ::= <id> "=" <expression> ";"
<if_stmt> ::= "if" "(" <expression> ")" <block> ("else" (<if_stmt> | <block>))?
<while_stmt> ::= "while" "(" <expression> ")" <block>
<return_stmt> ::= "return" <expression>? ";"
<expr_stmt> ::= <expression> ";"
<expression> ::= <comparison>
<comparison> ::= <additive> (("==" | "!=" | "<" | "<=" | ">" | ">=") <additive>)*
<additive> ::= <multiplicative> (("+" | "-") <multiplicative>)*
<multiplicative> ::= <unary> (("*" | "/") <unary>)*
<unary> ::= ("!" | "-") <unary> | <primary>
<primary> ::= <number>
            | "true"
            | "false"
            | <id>
            | <func_call>
            | "(" <expression> ")"
<func_call> ::= <id> "(" (<expression> ("," <expression>)*)? ")"

Operators and Expressions

  • Arithmetic: +, -, \*, /
  • Logic: ==, !=, <, >, <=, >=
  • Assignment: =

To keep our first version lean, we will implement one “magic” function: out(value). This will map to a system call that prints an integer to the standard output. It’s our way of seeing that our compiler actually worked!

A Complete Si Program

Here is what a complete Si program looks like. It calculates the sum of all numbers from 1 to 10:

fn add(a: int, b: int): int {
    return (a + b);
}

fn main(): int {
    let limit: int = 10;
    let sum: int = 0;
    let i: int = 1;
    let remove: bool = false;

    while (i <= limit) {
        sum = add(sum, i);
        i = i + 1;
    }

    out(sum); // Should output 55

    return 1;
}

Setting Up the C++ Compiler Project

To build a compiler, we need a reliable workshop. We require a C++ Compiler (to build our tool), NASM (to assemble our output), and GDB (to debug the assembly when things inevitably go wrong).

Prerequisites

  • Docker: To ensure our build and execution environment is identical regardless of your OS.
  • VS Code (Optional): Our recommended editor for this series.

Project Structure

Create a dedicated directory for our project (I’m naming mine gsisi). To keep our workspace organised, I will follow a standard C++ project layout. Create these folders inside your root directory:

  • gsisi/
    • src/ - Where our C++ compiler code lives.
    • include/ - For our header (.hpp) files.
    • tests/ - For our .si test programs.
    • CMakeLists.txt - Our build manager.
    • Dockerfile - Our environment.

Docker Development Environment

Inside our project root, create a Dockerfile:

FROM ubuntu:24.04

ARG DEBIAN_FRONTEND=noninteractive

\# Install the C++ and x86-64 assembly development tools
RUN apt-get update && \\
    apt-get install -y --no-install-recommends \\
        build-essential \\
        cmake \\
        gdb \\
        nasm \\
        sudo && \\
    apt-get clean && \\
    rm -rf /var/lib/apt/lists/\*

\# Create a non-root development user
RUN useradd --create-home --shell /bin/bash dev && \\
    echo "dev ALL=(ALL) NOPASSWD\:ALL" > /etc/sudoers.d/dev && \\
    chmod 0440 /etc/sudoers.d/dev

USER dev
WORKDIR /home/dev

CMake Build System

Rather than compiling each C++ source file manually, we will use CMake to configure the project, collect sources and apply the required C++ standard. Create a CMakeLists.txt file in your root folder:

cmake_minimum_required(VERSION 3.10)

\# Name of your project
project(gsisi LANGUAGES CXX)

\# Use C++20
set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)

file(GLOB SOURCES "src/\*.cpp")
list(REMOVE_ITEM SOURCES "${CMAKE_CURRENT_SOURCE_DIR}/src/main.cpp")

add_library(gsisi_frontend ${SOURCES})

target_include_directories(gsisi_frontend
    PUBLIC
        ${CMAKE_CURRENT_SOURCE_DIR}/include
)

add_executable(si_compiler src/main.cpp)

target_link_libraries(si_compiler
    PRIVATE
        gsisi_frontend
)

Docker Helper Script

Inside our project root, create docker.sh:

\#!/bin/bash

IMAGE_NAME="dev-env-gsisi"
CONTAINER_NAME="dev-env-gsisi-container"
CONTAINER_ENTRY="/usr/bin/bash"
WORK_DIR="/home/dev/gsisi"

PROJECT_ROOT="$(pwd)"

function container_stop() {
    sudo docker stop $CONTAINER_NAME || echo "container already stopped"
}

function clean() {
    container_stop
    sudo docker rm $CONTAINER_NAME || echo "container already removed"
    sudo docker image rm $IMAGE_NAME  || echo "image already removed"
}

function container_start() {
    if [ -z "$(sudo docker ps -a | grep $CONTAINER_NAME)" ]; then
        echo ">> creating dev container"

        sudo docker run -w $WORK_DIR --hostname dev -it --name $CONTAINER_NAME \\
            --net=host \\
            --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \\
            -v $PROJECT_ROOT:$WORK_DIR\:z \\
            $IMAGE_NAME $CONTAINER_ENTRY
    else
        echo ">> found existing dev container"
        sudo docker start $CONTAINER_NAME
        sudo docker exec -it $CONTAINER_NAME $CONTAINER_ENTRY
    fi
}

function image_build() {
    if [ -z "$(sudo docker images | grep $IMAGE_NAME)" ]; then
        echo ">> building image: $IMAGE_NAME"
        sudo docker build -t $IMAGE_NAME ./
    fi
}

function print_usage() {
    echo "Usage: ./docker.sh [container-start | container-stop | image-build | clean]"
}

function main() {
    if ! command -v docker >/dev/null 2>&1; then
        echo "Could not find docker. Please install it https\://docs.docker.com/engine/install/"
        exit 1
    fi

    case "$1" in
      container-start) container_start ;;
      container-stop)  container_stop ;;
      image-build)     image_build ;;
      clean)           clean ;;
      \*)               print_usage ;;
    esac
}

main $@

How to use Linux/macOS:

sh docker.sh

How to use Windows PowerShell:

bash docker.sh

Building the Project

Once you are inside your Docker container, building the project is a simple three-step process:

  1. Configure & Create a build directory: cmake -S . -B build
  2. Compile: cmake --build build

Note: Running it now will give an error, becuase we do not yet have source files

Optional: VS Code Setup

If you’re using VS Code, the following extensions can make development easier:

  • ms-vscode.cpptools-extension-pack: Essential for C++ IntelliSense.
  • ms-azuretools.vscode-docker: To manage your container easily.
  • sonarsource.sonarlint-vscode: To catch C++ bugs before you even compile.

Chapter Summary

  • Si now has a defined syntax, type system, grammar and required main entry point.
  • The project uses a clear C++ directory structure with CMake as its build system.
  • Docker provides a repeatable environment with the compiler, NASM and debugging tools.
  • This language specification gives later compiler stages a stable contract to implement.

Next Chapter

In Chapter 3, we will build the lexer in C++ and convert Si source text into a stream of tokens.

Subscribe / Stdin

Get the next part

New parts of From Source to Machine Code land in your inbox the day they publish. No spam, unsubscribe in one click.

We send your address to Kit, our email provider, and ask you to confirm by email before anything is sent. See our Privacy Statement.