blog
[!quote] A programmer should program the way s/he programs.
For now I am doing almost all work in my Obsidian vault, which is named Markcode.
It's a knowledge base, a research engine, also an IDE capable of full stack software engineering. Mostly importantly, it develops itself.
In this blog I will cover its essence: writing code in a decentralized way.
The Folder Structure
Markcode
- Notes
- Modules
- Scripts
- Others
- Template
Block Based Programming
From Markdown to Code Files
The system binds .md files with the code files extracted from code blocks in it. For example, if you both Python and Rust implementations (as two code blocks) in the note file euler_method.md , then when running the code execution command, euler_method.py, euler_method.rs will both be created in the same folder. There are some simple refactor mechanism - when you rename the .md file, relevant code files will be automatically renamed. And orphans will be automatically deleted.
Markers to Direct Execution Flow
Markcode employs specific markers that guide the execution flow within the vault. These markers act as directives, instructing the system on how to handle and execute the enclosed code blocks.
For example, the following marker is used to tell the code executor to only extract code blocks into code files without actually running the code.
<!-- nr -->
If you want to ignore other code blocks and run Haskell only, use this combination of markers.
<!-- nr --><!-- run haskell -->
By default, all code blocks in the vault are extracted, combined sequentially and executed together. If you want to run only some specific blocks, simply comment the first line with #! (The symbol to comment is dependent on the language)
Everything Can Be Templated
This is powered by the Templater plugin.
To accelerate development and maintain consistency, Markcode incorporates templates that can be invoked across various environments. These templates act as blueprints, providing standardized structures for new modules, scripts, or components.
Here's typical Template for C++ programming.
tR += `
\`\`\`cpp
#include <iostream>
extern "C" {
int main();
}
int main() {
std::cout << "Hello!" << std::endl;
return 0;
}
\`\`\`
\`\`\`python
import runcpp
if __name__ == "__main__":
lib = runcpp.load_dll("${tp.file.title}.dll")
lib.main()
\`\`\`
`
Test Flow Module
Testing is the bedrock of robust software development. The Test Flow Module in Markcode automates the testing pipeline. Here's an example to test rust code with a push script (the first block) to interact with the local project files.
```push fold D:\Codebase\rustplay\src\main.rs
```python fold
import testflow
testflow.rustplay()
fn main() {
println!("HW!");
}
A Note Can Be an App
In the traditional paradigm, applications are monolithic constructs with tightly coupled components. Markcode disrupts this notion by transforming individual notes within the Obsidian vault into self-contained applications. Each note encapsulates its functionality, dependencies, and execution logic, allowing for a highly modular and scalable development process.
Here's a LLM chat App written this way. It supports multi-round conversation, note referencing and web browsing. It reads the Question and writes the output.
# Question
# Response
----
```python fold
!
import requests import json import myapikeys import fileUtils from iohelper import *
API_KEY = myapikeys.anthropic
def call_claude_api(prompt): url = "https://api.anthropic.com/v1/messages" headers = { "Content-Type": "application/json", "X-Api-Key": API_KEY, "anthropic-version": "2023-06-01", } data = { "model": "claude-3-5-sonnet-20240620", "max_tokens": 3000, "messages": [{"role": "user", "content": prompt}] } response = requests.post(url, headers=headers, json=data)
if response.status_code == 200:
return response.json()['content'][0]['text']
else:
return f"Error: {response.status_code}, {response.text}"
def main(): user_input = inputh("Question") clearh("Response") printh("Thinking...", "Response")
# Replace links with file content
processed_input = fileUtils.replace_links_with_content(user_input)
if "<!-- web -->" in user_input:
import webCrawler
processed_input = webCrawler.replace_links_with_content(user_input)
# printh(processed_input)
response = call_claude_api(processed_input)
clearh("Response")
printh(response, "Response")
chat_history = processed_input + "\n\n**Claude:**\n" + response + "\n\n---\n"
with open("chat_history_temp.md", "w", encoding="utf-8") as f:
f.write(chat_history)
if name == "main": main()
## Implementing the Modules
### Obsidian Server
The `obsidian.md ` module is the linchpin that interacts with an Obsidian server, ensuring that the vault is initialized correctly and that the server remains responsive. It encapsulates functions to run JavaScript code within the Obsidian environment, manage file operations, and handle user interactions.
Here's a brief view of it.
```python
# vaultBuilding.md Module
import requests
import os
import time
note_path = os.environ.get('MD_FILE')
obs_vault = os.environ.get('OBS_VAULT')
if note_path:
note_title = os.path.splitext(os.path.basename(note_path))[0]
note_directory = os.path.dirname(note_path)
def initialize(max_attempts=30, delay=0.5):
"""
Wait for the server to launch and become responsive.
:param max_attempts: Maximum number of attempts to connect to the server
:param delay: Delay in seconds between each attempt
:return: True if server is responsive, False otherwise
"""
for attempt in range(max_attempts):
try:
response = runjs("return 200;")
if response == 200:
return True
except requests.RequestException:
pass
time.sleep(delay)
print(f"Server did not become responsive after {max_attempts} attempts.")
return False
def runjs(js_code, vault_path=obs_vault, server_url='http://localhost:3300'):
code_file = os.path.join(vault_path, 'code.js')
# Write the JavaScript code to the file
with open(code_file, 'w', encoding="utf-8") as f:
f.write(js_code)
# Send the request to the server
response = requests.post(f'{server_url}/run')
if response.status_code == 200:
try:
return response.json()
except Exception:
return ""
else:
raise Exception(f"Error: {response.status_code} - {response.text}")
# Additional functions omitted for brevity
This module not only ensures that the Obsidian server is up and running but also provides utility functions to execute JavaScript code, interact with the file system, and facilitate user interactions within the vault.
IO Helper Module
The iohelper.md module serves as the intermediary between Python scripts and the Obsidian environment. It provides functions to manipulate markdown files, prompt user inputs, display notifications, and handle cursor interactions within the editor.
# iohelper.md Module
import sys
import os
import obsidian
import time
note_path = os.environ.get('MD_FILE')
obs_vault = os.environ.get('OBS_VAULT')
note_title = os.path.splitext(os.path.basename(note_path))[0]
note_directory = os.path.dirname(note_path)
def printh(content: str, title_name: str = "", output_file: str = ""):
# Function implementation...
pass
def inputh(title_name: str) -> str:
# Function implementation...
pass
def clearh(title_name: str = "", target_file=None):
# Function implementation...
pass
def input_prompt(prompt_text: str) -> str:
# Function implementation...
pass
def notice(text: str):
# Function implementation...
pass
def get_selection():
# Function implementation...
pass
def append_cursor(text: str):
# Function implementation...
pass
def get_cursor_line():
# Function implementation...
pass
# Optional functions
def refreshh(title_name, refresh_function):
# Function implementation...
pass
Modularizing Other Languages
Here's the here.md module that implements IO operation for Haskell.
-- Here.md Module
module Here
( Showable(..)
, appendHere
, getOutputFilename
) where
import System.FilePath (takeBaseName, replaceExtension)
import System.Environment (getProgName)
class Showable a where
toString :: a -> String
instance Showable String where
toString = id
instance Showable Int where
toString = show
instance Showable Integer where
toString = show
instance Showable Double where
toString = show
instance Showable Bool where
toString = show
-- Add more instances as needed
appendHere :: Showable a => a -> IO FilePath
appendHere obj = do
outputFile <- getOutputFilename
appendFile outputFile ("\n" ++ toString obj)
return outputFile
getOutputFilename :: IO FilePath
getOutputFilename = do
mainScriptName <- getProgName
let baseName = takeBaseName mainScriptName
return $ replaceExtension baseName ".md"
main :: IO ()
main = do
putStrLn "Successfully extracted."
In this module, the Showable typeclass and its instances facilitate the conversion of various data types to strings, enabling seamless integration with markdown outputs. Functions like appendHere and getOutputFilename provide utility methods to manage file operations, ensuring that each module remains focused and maintainable.
Being Self-contained
- Bootstrapping The system is written by itself, and thus can improve itself iteratively.
- Literal Programming As a form of literal programming, each module documents itself by simply putting markdown content between code blocks.
More
Lots of things are not covered in this blog but also crucial for this vault: Version control system, AI integration, scripts to interact with the internet and local file system, project management system, academic resource management, code highlight, reference tracing, vim editor...