Plain text files are the likely outcome of your work, but should you default to Notepad to create them? While sufficient for basic tasks, there’s a world of enhanced functionality beyond Notepad’s capabilities.

Syntax highlighting and automatic formatting are merely the beginning. Consider the benefits of linting, code completion, and semi-automatic refactoring – essential tools for daily development. A deeper understanding of their workings can significantly enhance your workflow.

This Language Server Protocol tutorial explores the mechanisms behind these features, demystifying the inner workings of text editors. As a practical exercise, we’ll create a basic language server and implement example clients for VSCode, Sublime Text 3, and Vim.

Compilers vs. Language Services

Let’s set aside syntax highlighting and formatting, both handled by static analysis (a topic worthy of its own exploration), and focus on the primary feedback provided by these tools, categorized as compilers and language services.

Compilers process your source code and generate a modified form, flagging errors if the code deviates from language rules. While familiar, this process is often slow and limited in scope. Wouldn’t it be advantageous to receive assistance during the code creation phase?

Language services address this need by offering real-time insights into your codebase during development, typically faster than full project compilation.

These services vary in complexity, ranging from simple tasks like listing project symbols to more involved operations like suggesting code refactoring steps. They represent the core reason behind using advanced code editors, going beyond basic compilation and error detection to provide rapid and insightful feedback.

The Case for Editor Agnosticism in Programming

You’ll notice that we haven’t specified any particular text editors yet, and there’s a reason for this approach, best illustrated with an example.

Imagine developing a new programming language called Lapine, known for its elegance and Elm-like error messages. You’ve incorporated features like code completion, references, refactoring help, and diagnostics.

Which code editor do you prioritize support for? And what about subsequent editors? With user adoption being paramount, choosing the wrong editor could alienate potential users. Ideally, you want to remain editor-agnostic, focusing on your language and its features rather than editor-specific integrations.

Language Servers

This is where language servers come into play. These tools interact with language clients, providing the insights mentioned earlier, remaining independent of specific text editors for the reasons just discussed.

As is often the case, an abstraction layer offers a solution. Language servers decouple language tools from code editors, allowing language creators to encapsulate their features within a single server and enabling code editors to become clients through simple extensions. This benefits everyone involved. However, it necessitates a standardized communication protocol between clients and servers.

Fortunately, this isn’t a hypothetical concept. Microsoft has already laid the groundwork by defining the Language Server Protocol.

As with many groundbreaking ideas, this stemmed from necessity rather than foresight. Numerous code editors had begun incorporating language features, some outsourced to external tools and others developed internally. As scalability became an issue, Microsoft took the lead in separating these functionalities. Rather than confining these advancements within VSCode, Microsoft chose to promote openness, allowing users greater flexibility in their choice of editors.

Language Server Protocol

Introduced in 2016, the Language Server Protocol (LSP) aims to decouple language tools from text editors. While bearing traces of its VSCode origins, LSP represents a significant step towards editor agnosticism. Let’s delve into the protocol itself.

Clients and servers (representing code editors and language tools, respectively) communicate using simple text messages. These messages consist of HTTP-like headers, JSON-RPC content, and can originate from either the client or server. The JSON-RPC protocol defines requests, responses, notifications, and governs their interactions. Crucially, it’s designed for asynchronous operation, enabling clients and servers to handle messages out of order and in parallel.

In essence, JSON-RPC allows a client to request method execution on another program with specific parameters, receiving a result or error in return. Building upon this, LSP defines available methods, expected data structures, and additional rules for these transactions. For example, it includes a handshake process during client startup.

The server maintains state and is intended to handle a single client at a time. While there are no explicit limitations on communication, a language server could technically run on a separate machine from the client, though this would introduce noticeable latency for real-time feedback. In practice, language servers and clients operate on the same files and engage in frequent communication.

The LSP offers a wealth of documentation once you become familiar with its structure. As previously mentioned, much of the documentation is presented within the context of VSCode, even though the underlying principles have wider applicability. For instance, the protocol specification is written entirely in TypeScript. To assist those unfamiliar with VSCode and TypeScript, we’ll provide a brief primer.

LSP Message Types

The Language Server Protocol defines numerous message categories, broadly classified as “admin” and “language features.” Admin messages encompass those used during client/server handshakes, file operations, and feature negotiation, where clients and servers declare their supported functionalities. Different languages and tools naturally offer diverse feature sets, and this mechanism allows for gradual adoption. Langserver.org outlines several key features recommended for client and server support, with at least one being mandatory.

Language features constitute our primary interest. Among these, diagnostic messages deserve special attention. Diagnostics play a crucial role. Upon opening a file, an implicit assumption is that the file will be analyzed for potential issues. Your editor should alert you to any problems encountered. LSP facilitates this through the following steps:

1. The client opens the file and transmits a textDocument/didOpen notification to the server.
2. The server analyzes the file and responds with a textDocument/publishDiagnostics notification containing the results.
3. The client interprets the results and displays corresponding error indicators within the editor.

This represents a passive approach to obtaining insights from your language services. A more active example involves identifying all references to a symbol under your cursor. This process unfolds as follows:

1. The client sends a textDocument/references request to the server, specifying a location within a file.
2. The server determines the symbol, locates references within the current and potentially other files, and responds with a list of references.
3. The client presents the references to the user.

A Blacklist Tool

While we could delve into the intricacies of the Language Server Protocol, let’s leave that for client implementers. To solidify the concept of separating editors from language tools, we’ll step into the shoes of a tool creator.

Keeping things straightforward, we’ll focus on diagnostics instead of creating a new language with complex features. Diagnostics, essentially warnings about file content, align well with our goal. Linters, for instance, return diagnostics. We’ll create a simplified version.

Our tool will identify and flag undesirable words within text. We’ll then integrate this functionality into a couple of different text editors.

The Language Server

Let’s begin with the tool itself, embedding it directly within a language server. For simplicity, we’ll use a Node.js app, though any technology capable of handling input and output streams would suffice.

The core logic is as follows: given a text string, this function returns an array containing blacklisted words and their corresponding indices within the text.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

const getBlacklisted = (text) => {
  const blacklist = [
    'foo',
    'bar',
    'baz',
  ]
  const regex = new RegExp(`\\b(${blacklist.join('|')})\\b`, 'gi')
  const results = []
  while ((matches = regex.exec(text)) && results.length < 100) {
    results.push({
      value: matches[0],
      index: matches.index,
    })
  }
  return results
}

Next, let’s transform this into a server.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

const {
  TextDocuments,
  createConnection,
} = require('vscode-languageserver')
const {TextDocument} = require('vscode-languageserver-textdocument')

const connection = createConnection()
const documents = new TextDocuments(TextDocument)

connection.onInitialize(() => ({
  capabilities: {
    textDocumentSync: documents.syncKind,
  },
}))

documents.listen(connection)
connection.listen()

We’re utilizing the vscode-languageserver package, despite its potentially misleading name. While originating from VSCode, it functions perfectly well outside of that environment. This exemplifies the lingering influence of LSP’s origins. The package handles the low-level protocol details, allowing you to focus on the specific use case. This snippet establishes a connection and associates it with a document manager. When a client connects to the server, it signals its interest in receiving notifications about opened text documents.

We could stop here, having implemented a fully functional, albeit rudimentary, LSP server. Instead, let’s enhance it to respond to document changes by providing diagnostic information.

1
2
3
4
5
6

documents.onDidChangeContent(change => {
  connection.sendDiagnostics({
    uri: change.document.uri,
    diagnostics: getDiagnostics(change.document),
  })
})

Finally, we connect the pieces: the modified document, our logic, and the diagnostic response.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

const getDiagnostics = (textDocument) =>
  getBlacklisted(textDocument.getText())
    .map(blacklistToDiagnostic(textDocument))

const {
  DiagnosticSeverity,
} = require('vscode-languageserver')

const blacklistToDiagnostic = (textDocument) => ({ index, value }) => ({
  severity: DiagnosticSeverity.Warning,
  range: {
    start: textDocument.positionAt(index),
    end: textDocument.positionAt(index + value.length),
  },
  message: `${value} is blacklisted.`,
  source: 'Blacklister',
})

The diagnostic payload is generated by processing the document’s text through our function and mapping the output to the format expected by the client.

This script automates the entire process for you.

1

curl -o- https://raw.githubusercontent.com/reergymerej/lsp-article-resources/revision-for-6.0.0/blacklist-server-install.sh | bash

Note: If you’re uncomfortable executing scripts from unknown sources, please check the source. It creates the project, downloads the index.js file, and establishes an npm link for you.

Complete Server Source

The complete source code for the blacklist-server is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

#!/usr/bin/env node

const {
  DiagnosticSeverity,
  TextDocuments,
  createConnection,
} = require('vscode-languageserver')

const {TextDocument} = require('vscode-languageserver-textdocument')

const getBlacklisted = (text) => {
  const blacklist = [
    'foo',
    'bar',
    'baz',
  ]
  const regex = new RegExp(`\\b(${blacklist.join('|')})\\b`, 'gi')
  const results = []
  while ((matches = regex.exec(text)) && results.length < 100) {
    results.push({
      value: matches[0],
      index: matches.index,
    })
  }
  return results
}

const blacklistToDiagnostic = (textDocument) => ({ index, value }) => ({
  severity: DiagnosticSeverity.Warning,
  range: {
    start: textDocument.positionAt(index),
    end: textDocument.positionAt(index + value.length),
  },
  message: `${value} is blacklisted.`,
  source: 'Blacklister',
})

const getDiagnostics = (textDocument) =>
  getBlacklisted(textDocument.getText())
    .map(blacklistToDiagnostic(textDocument))

const connection = createConnection()
const documents = new TextDocuments(TextDocument)

connection.onInitialize(() => ({
  capabilities: {
    textDocumentSync: documents.syncKind,
  },
}))

documents.onDidChangeContent(change => {
  connection.sendDiagnostics({
    uri: change.document.uri,
    diagnostics: getDiagnostics(change.document),
  })
})

documents.listen(connection)
connection.listen()

Language Server Protocol Tutorial: Time for a Test Drive

After linking the project, run the server with stdio as the transport mechanism:

1

blacklist-server --stdio

The server is now listening for LSP messages on stdio. We could manually send these messages, but let’s create a client instead.

Language Client: VSCode

Given that this technology originated in VSCode, it’s fitting to start there. We’ll develop an extension that creates an LSP client and connects it to our server.

There are a number of ways for creating VSCode extensions, including Yeoman with the generator-code generator. However, for the sake of simplicity, we’ll create a barebones example.

Let’s clone the boilerplate and install its dependencies:

1
2
3

git clone git@github.com:reergymerej/standalone-vscode-ext.git blacklist-vscode
cd blacklist-vscode
npm i # or yarn

Open the blacklist-vscode directory in VSCode.

Press F5 to launch a new VSCode instance, debugging the extension.

In the debug console of the initial VSCode instance, you should see the message: “Look, ma. An extension!”

We now have a basic VSCode extension up and running. Let’s transform it into an LSP client. Close both VSCode instances and execute the following command within the blacklist-vscode directory:

1

npm i vscode-languageclient

Replace extension.js with:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

const { LanguageClient } = require('vscode-languageclient')

module.exports = {
  activate(context) {
    const executable = {
      command: 'blacklist-server',
      args: ['--stdio'],
    }

    const serverOptions = {
      run: executable,
      debug: executable,
    }

    const clientOptions = {
      documentSelector: [{
        scheme: 'file',
        language: 'plaintext',
      }],
    }

    const client = new LanguageClient(
      'blacklist-extension-id',
      'Blacklister',
      serverOptions,
      clientOptions
    )

    context.subscriptions.push(client.start())
  },
}

This utilizes the vscode-languageclient package to create an LSP client within VSCode. Unlike vscode-languageserver, this package is tightly coupled to VSCode. In short, we’re instructing the extension to create a client and connect to the server we defined earlier. Setting aside the VSCode extension specifics, we’re essentially configuring the extension to use this LSP client for plain text files.

To test it out, open the blacklist-vscode directory in VSCode and press F5 to launch a new instance for debugging.

In this new instance, create a plain text file and save it. Type the words “foo” or “bar” and observe. You should see warnings indicating that these words are blacklisted.

And that’s it! We didn’t have to replicate any of our logic, merely establish communication between the client and server.

Let’s repeat this process for another editor, this time Sublime Text 3. The steps will be similar and even more straightforward.

Language Client: Sublime Text 3

First, open ST3 and access the command palette. We need a framework to turn the editor into an LSP client. Type “Package Control: Install Package” and press enter. Locate the “LSP” package and install it. This provides us with the capability to define LSP clients. While presets exist, we’ll configure our own.

Open the command palette again and search for “Preferences: LSP Settings.” This opens the LSP package’s configuration file, LSP.sublime-settings. Add the following configuration to define a custom client.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

{
  "clients": {
    "blacklister": {
      "command": [
        "blacklist-server",
        "--stdio"
      ],
      "enabled": true,
      "languages": [
      {
        "syntaxes": [
          "Plain text"
        ]
      }
      ]
    }
  },
  "log_debug": true
}

This should look familiar from the VSCode extension. We’ve defined a client, specified plain text files as its scope, and provided the language server’s location.

Save the settings and create a new plain text file. Type “foo” or “bar” and observe. Once again, you’ll see warnings for the blacklisted words. The presentation, meaning how messages are displayed, might differ, but the underlying functionality remains identical. We barely had to lift a finger to add support for this editor.

Language “Client”: Vim

If you’re still not convinced of the benefits of separating concerns and sharing features across editors, here’s how to add the same functionality to Vim using Coc.

Open Vim and type :CocConfig. Then, add the following:

1
2
3
4
5
6
7

"languageserver": {
  "blacklister": {
    "command": "blacklist-server",
    "args": ["--stdio"],
    "filetypes": ["text"]
  }
}

And that’s all it takes.

Client-server Separation Lets Languages and Language Services Thrive

The separation of language services from their host text editors offers clear advantages. It allows language developers to focus on their area of expertise and editor developers to do the same. While still relatively new, this approach is gaining traction.

Armed with this foundational knowledge, perhaps you can contribute to projects that further this concept. The editor wars may rage on, but that’s perfectly fine. As long as language capabilities can exist independently of specific editors, you remain free to use your editor of choice.

As a Microsoft Gold Partner, Toptal is your elite network of Microsoft experts. Build high-performing teams with the experts you need - anywhere and exactly when you need them!