Websites, like any software product, need to be tested. They need to be tested for both usability and functionality, like any other software, but unlike most other software, particularly backend software, which is what I primarily work on, websites need to be tested for what users like. At least for now, this type of testing can’t be automated. LLMs may be able to assist you, and it’s likely a good start in building your frontend, but you don’t really have any way to verify the opinion of LLMs are represented in the demographic of your website visitors. The only way to know for sure is to try it.
That’s where A/B testing comes in. It allows you to run multiple experiments at once, which in combination with your analytics tools allow you to see which your users like more: option A, or option B.
There are a few ways to set up A/B testing for a website. You can do it purely using Javascript, on the client’s side, or you can serve completely different files based on the experiment you want to show the user, which is what this website does. I’ll cover both options, but I will focus more on the server side version, as I believe that it’s better.
The client-side way may be simpler to implement for most websites, but it does have some drawbacks: namely that the user has the ability to change what test they are seeing. If that isn’t something you mind, this may be the better option for you. Another downside, although not very common, is if the user has Javascript disabled, they won’t see any of your tests, and depending on how you configured your tests, the website may be completely missing some content. Keep this in mind when designing your experiments.
The Javascript version is pretty simple. In your HTML, all you need to do is put any data you want to hide or show for your experiments in a separate <div> and give it a class name to represent your experiment.
<h1> This text is shown to every visitor </h1>
<p> This text is also shown to everyone. </p>
<div class="test-text">
<h2> But only some users will see this one </h2>
</div>
And then you’ll need some CSS to hide the test when it’s not explicitly enabled.
.disable-experiment {
display: none;
}
And then add that class to your test divs.
The actual Javascript to enable the test is pretty simple too:
const body = document.getElementByTagName('body')[0];
const experiments = body.getAttribute("data-experiments").split(' ');
for ( const class of experiments ) {
const elements = document.getElementsByClassName(class);
for ( const ele of elements ) {
ele.classList.remove('disable-experiment');
}
}
This is pretty simple. It just loops through an attribute on the body tag of the page called data-experiments, which we will
inject from a server that we will write later. The data-experiments attribute will contain a space-separated list of experiments.
The names will match the class name assigned to the experiment div(s). We then remove the disable-experiment class from each of
the enabled experiments, which will make them visible to the visitor.
For the server we’ll use a Cloudflare Worker. If you would rather use Express.js or an AWS Lambda function, it will be basically the same. You could even write a server from scratch and host it yourself, if you prefer.
Helpfully, Cloudflare provides a Worker template for this exact use-case. If you have a Cloudflare account, you can log in and head over to the “Workers & Pages” tab, and select “Create application”. From there you should see the “A/B test script” template, which should look something like this:
const ORIGIN_URL = 'https://example.com';
const EXPERIMENTS = [
{ name: 'big-button', threshold: 0.5 }, // enable the Big Button experiment for 50% of users
{ name: 'new-brand', threshold: 0.1 }, // enable the New Brand experiment for 10% of users
{ name: 'new-layout', threshold: 0.02 }, // enable the New Layout experiment for 2% of users
];
export default {
async fetch(request, env, ctx) {
const fingerprint = [request.headers.get('cf-connecting-ip'), request.cf?.postalCode]; // add any values you want considered as a fingerprint
const activeExperiments = await getActiveExperiments(fingerprint, EXPERIMENTS);
// add a data-experiments attribute to the <body> tag
// which can be styled in CSS with a wildcard selector like [data-experiments*="big-button"]
const rewriter = new HTMLRewriter().on('body', {
element(element) {
element.setAttribute('data-experiments', activeExperiments.join(' '));
},
});
const res = await fetch(ORIGIN_URL, request);
return rewriter.transform(res);
},
};
// Get active experiments by hashing a fingerprint
async function getActiveExperiments(fingerprint, experiments) {
const fingerprintHash = await hash('SHA-1', JSON.stringify(fingerprint));
const MAX_UINT8 = 255;
const activeExperiments = experiments.filter((exp, i) => fingerprintHash[i] <= exp.threshold * MAX_UINT8);
return activeExperiments.map((exp) => exp.name);
}
// Hash a string using the Web Crypto API
async function hash(algorithm, message) {
const msgUint8 = new TextEncoder().encode(message); // encode as (utf-8) Uint8Array
const hashBuffer = await crypto.subtle.digest(algorithm, msgUint8); // hash the message
const hashArray = new Uint8Array(hashBuffer); // convert buffer to byte array
return hashArray;
}
The only changes you’ll need to make is at the top of the file. The ORIGIN_URL needs to point to where all your website files are
hosted, and they must be publicly accessible from here. EXPERIMENTS is simply a list of all the experiments you want to run.
After you make your changes, you’ll just publish your worker, and set up your domain name to point to the worker instead of your website.
Instead of using ORIGIN_URL and pulling the files from a different URL, you can set it up to pull the files from an R2 or S3 bucket
instead, which we’ll cover in the Server Side for server side tests section.
If you’d rather manage your tests in a way more hidden from users, this is probably the best option for you. Since our website is written in Jekyll (or just using the Liquid template engine), it is pretty simple to set up your tests.
Simply wrap any code that you only want enabled for tests in an if block. For example, this website has the following code for
the header bar:
<div class="trigger">
{%- for path in page_paths -%}
{%- assign my_page = site.pages | where: "path", path | first -%}
{%- if my_page.title -%}
{%- if site.portfolio == true and my_page.title == "About" -%}
<a class="page-link" href="{{ my_page.url | relative_url }}">Portfolio</a>
{%- else -%}
<a class="page-link" href="{{ my_page.url | relative_url }}">{{ my_page.title | escape }}</a>
{%- endif -%}
{%- endif -%}
{%- endfor -%}
</div>
The important line here is: {%- if site.portfolio == true and my_page.title == "About" -%}. We have a variable called
portfolio that we set in the site’s config file. That is what tells Jekyll that the test has been enabled, and when it is,
we’ll call the About page Portfolio. If you look at the header right now, and see Portfolio that means that that
variable has been set in the version of the site that you’re seeing.
Very helpfully, Jekyll lets us pass in multiple config files when we build our site. Any variable in a file later in the list will override the setting in previous files.
So, let’s make a config file for our new test. We’ll name it _config_test_portfolio.yml. It’s only one line:
protfolio: true
And then if we want to enable the test we can build the site with the following command:
bundle exec jekyll build -c _config.yml,_config_test_portfolio.yml
The server for this is very similar to the Javascript version. We’ll start with the template that we used last time, for the Clouldflare Worker:
const ORIGIN_URL = 'https://example.com';
const EXPERIMENTS = [
{ name: 'big-button', threshold: 0.5 }, // enable the Big Button experiment for 50% of users
{ name: 'new-brand', threshold: 0.1 }, // enable the New Brand experiment for 10% of users
{ name: 'new-layout', threshold: 0.02 }, // enable the New Layout experiment for 2% of users
];
export default {
async fetch(request, env, ctx) {
const fingerprint = [request.headers.get('cf-connecting-ip'), request.cf?.postalCode]; // add any values you want considered as a fingerprint
const activeExperiments = await getActiveExperiments(fingerprint, EXPERIMENTS);
// add a data-experiments attribute to the <body> tag
// which can be styled in CSS with a wildcard selector like [data-experiments*="big-button"]
const rewriter = new HTMLRewriter().on('body', {
element(element) {
element.setAttribute('data-experiments', activeExperiments.join(' '));
},
});
const res = await fetch(ORIGIN_URL, request);
return rewriter.transform(res);
},
};
// Get active experiments by hashing a fingerprint
async function getActiveExperiments(fingerprint, experiments) {
const fingerprintHash = await hash('SHA-1', JSON.stringify(fingerprint));
const MAX_UINT8 = 255;
const activeExperiments = experiments.filter((exp, i) => fingerprintHash[i] <= exp.threshold * MAX_UINT8);
return activeExperiments.map((exp) => exp.name);
}
// Hash a string using the Web Crypto API
async function hash(algorithm, message) {
const msgUint8 = new TextEncoder().encode(message); // encode as (utf-8) Uint8Array
const hashBuffer = await crypto.subtle.digest(algorithm, msgUint8); // hash the message
const hashArray = new Uint8Array(hashBuffer); // convert buffer to byte array
return hashArray;
}
We are going to serve the website files from an R2 bucket, so we can replace ORIGIN_URL with our bucket in the env parameter.
We also don’t mind having all our experiments being equally likely to be chosen, so instead of the EXPERIMENTS array we’ll
find out what experiments we have by just looking at the top level directories in our R2 bucket. This way we won’t need to update
the Worker every time we want to add or remove an experiment.
Since we have no EXPERIMENTS array, we’ll also completely rewrite the getActiveExperiments function. We’ll change the name too. We’ll
call it selectExperiment.
So, first we’ll change our fetch function.
async fetch(request, env, ctx) {
// Get the page that the caller has requested
const path = new URL(request.url).pathname;
const fingerprint = [ request.headers.get('cf-connecting-ip'), request.cf?.postalCode ];
// Get all top level directories in the bucket. These are all the possible test sites
const allFiles = await env.SITE_BUCKET.list();
// Filter to only top level directories
const activeExperiments = [];
for (const file of allFiles.objects) {
const parts = file.key.split('/');
if (parts.length === 2 && !activeExperiments.includes(parts[0])) {
activeExperiments.push(parts[0]);
}
}
// Select an experiment based on the fingerprint
const experiment = await selectExperiment(fingerprint, activeExperiments);
// Serve the file
let fileKey = path;
if (fileKey.split('.')[1] == null) {
fileKey = `${experiment}${fileKey}index.html`
} else {
fileKey = `${experiment}${fileKey}`
}
const file = await env.SITE_BUCKET.get(fileKey);
return new Response(file.body)
}
We first get the path the user requested. We use this so we know what file to send them later. Next we get their fingerprint, which is the same as the template.
After that we get a list of every file in the R2 bucket, and then filter that down to only the top level directories. This is a list of all our available experiments. It’s important that we only use this bucket for our website experiments, as any other folder in this bucket will be treated as an experiment.
After we have the list of all experiments, we call selectExperiment to pick which folder to pull the file from, and then we
return the file from the selected folder.
Our selectExperiment function works similarly to the getActiveExperiments function from the template:
async function selectExperiment(fingerprint, activeExperiments) {
const fingerprintHash = await hash('SHA-1', JSON.stringify(fingerprint));
const fingerprintHashString = fingerprintHash.join('');
const experimentIndex = parseInt(fingerprintHashString, 16) % activeExperiments.length;
return activeExperiments[experimentIndex];
}
Our hash function remains unchanged.
This is a trimmed down version of what this website uses. The only major difference is that this website’s Worker handles invalid page requests by returning a 404 page. You can see the Worker that this site uses here.
The SITE_BUCKET is configured in our wrangler.toml file:
[[r2_buckets]]
binding = 'SITE_BUCKET'
bucket_name = '<bucket name>'
And then we can deploy the function with wrangler, as normal with npx wrangler deploy.
Now you can open your web browser, navigate to the URL of your newly deployed worker, and you should see an error page because you forgot to upload your website to the R2 bucket.
You can do this manually with npx wrangler r2 object put <path in bucket> --file <file>, or you can upload them through the web interface, or you can write a script to do it automatically, which is what we’re going to do in the next section.
Before we get started on the script, we need to get all our different experiment website builds together. We’ll put them in a folder called _test_sites.
Let’s build all two versions of our site. One with the “portfolio” test enabled, and one without.
mkdir _test_site
bundle exec jekyll build -d _test_sites/default_site
bundle exec jekyll build -c _config.yml,_config_test_portfolio.yml -d _test_sites/portfolio_site
And we’ll put our script to upload the files in scripts/copy-to-bucket.
We’ll write the script in bash, so we can simply use the wrangler CLI to interact with the R2 bucket. Technically this can be done in any language, of course, but our script isn’t very complex, so bash will do just fine.
The script will be pretty simple. It only really needs to do two tasks:
First we’ll delete all the old files. There’s two ways we can do this. We can delete the entire bucket and recreate it, or we can just delete the files. Deleting the bucket is simpler. It’s just
npx wrangler r2 bucket delete <bucket name>
npx wrangler r2 bucket create <bucket name>
But that feels dirty to me for some reason, so I opted to instead delete every file.
If we look at the help menu for wrangler r2 object --help, you’ll see we have to delete each object individually, so we’ll need to get a list of all the objects.
npx wrangler r2 object get <bucket name>/
First, you’ll see that I added a / to the bucket name. This is important. If you don’t add that / wrangler will just throw an error and exit. Second, if you look at the end of the output you’ll notice that it’s truncated, and only shows 20 results per page. This is fine. We’ll
just have to use a loop.
We’ll get a list of the first 20 objects, delete them, and then get a list of the next 20, and repeat until we don’t have any objects left:
BUCKET_NAME=$1
while
# Get array of the files
BUCKET_OBJECTS=$( npx wrangler r2 object get ${BUCKET_NAME}/ 2> /dev/null | sed -n '/{/,$p}' -- | jq '.result[]' -c )
[[ "$BUCKET_OBJECTS" == "" ]] && break
for obj in $BUCKET_OBJECTS; do
npx wrangler r2 object delete ${BUCKET_NAME}/$( echo $obj | jq .key | tr -d '"' )
done
do true ; done
This might be kind of confusing if you’re not familiar with bash, so I’ll explain the line where BUCKET_OBJECTS gets assigned. The $() wrapping means “run this as a command”. We already know what the wrangler command does. The next thing after is 2> which tells bash to print
anything printed to stderr to the file following the 2> symbol instead of to stdout. /dev/null is a file that you can think of as a void. Anything printed there is discarded, never to be seen again. This means that any error messages from wrangler will be discarded.
This is important because error messages may cause the next parts to produce errors.
The | (pipe) symbol is very important in bash. It means to send the output of one command to whatever command comes after the pipe character. In this case that is a sed command. This particular sed command will output the first line we find that starts with the { character.
This is the start of the JSON output. We pass that to a program called jq, which we’ll use to parse the JSON. the .result[] means that we only want the array found in the JSON string with the key “result”. If you add echo $BUCKET_OBJECTS you can see exactly what the final
output looks like.
After that we check if BUCKET_OBJECTS is empty, and if it is, we break out of the loop. If not, we use a for loop to loop through each object and delete it from the bucket.
Now that we’ve cleaned out our bucket, we can upload the new sites to it. We know where our website files are, relative to our script, so let’s set a variable to point to it our script, and from there we know how to navigate to our websites.
SCRIPT_PATH="$( cd -- "$(dirname "$0")" > /dev/null 2>&1; pwd -P )"
WEBSITE_PATH="$SCRIPT_PATH/../../_test_sites"
The slightly complicated command to get SCRIPT_PATH is simply because the script can be called from any directory in the system, so we have to account for the fact that the caller will possibly not be in the same folder as the script.
Now we can loop through all the sites we’ve built.
for website in "$WEBSITE_PATH"/*; do
WEBSITE_NAME=${website##*/}
echo $WEBSITE_NAME
done
This will print out the name of each of our website folders. To upload the files we’ll have to upload each file in each subdirectory of the website. It’s probably easiest to do this recursively, so that’s what we’ll do. We’ll start in the website’s root folder, and loop through everything in that folder. If it’s a file we upload it, if it’s a folder we enter that and loop through every file in that folder, and repeat until we stop finding folders.
function upload_site () {
# Function paramteres in bash are weird. They're $1, $2, $3, etc. (same as command line arguments outside of the functions)
LOCAL_WEBSITE_PATH=$1
WEBSITE_NAME=$2
# Upload each file
for file in "$LOCAL_WEBSITE_PATH"/*; do
# If $file is a regular file
if [[ -f "$file" ]]; then
# Remove "$WEBSITE_PATH" from the filename
CLEANED_FILENAME=${file#$WEBSITE_PATH}
# Upload file
npx wranger r2 object put ${BUCKET_NAME}/${CLEANED_FILENAME} --file $file
elif [[ -d "$file" ]]; then # If $file is a directory
upload_site $file $WEBSITE_NAME
fi
done
}
and we’ll update our for loop to call this function
for website in "$WEBSITE_PATH"/*; do
WEBSITE_NAME=${website##*/}
upload_site $website $WEBSITE_NAME
done
The full script is available here.
This website also utilizes a Python script to automatically build each and every combination of experiments.
If there are some experiments that are mutually exclusive, you can simply disable the other one in the config file for the one you want. Since the config files are read in order, and later ones override the previous ones, this will result in only the last test in the list being used. Or, you can do what I did and have every test work correctly with every other test. Regardless, let’s start writing our build script.
So, what exactly does the script need to do? It needs to navigate to the folder where all our website source is located (or at least know where to find the files), remove the previous website builds, get a list of every experiment we have,
and then run bundle exec jekyll build -c <configs> for every possible combination of config files.
We’ll put our script in the folder scripts/gen-test-sites/src, so we can get the path to the website source with the following function:
def get_working_directory() -> str:
return os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
This returns the path to the folder three folders up from the script that we’re running. This works regardless of how you call the script.
The next step is to write a function to generate all the possible experiment groups. For this website, order doesn’t matter, but if you have different behaviour depending on the order (ie. mutually exclusive experiments), you’ll have to modify this function slightly.
To get every test group, we simply search the website directory for every YAML file starting with _config_test_. Then we can simply use itertools.combinations to construct our experiment groups.
def get_test_groups(directory: str) -> list:
# Get a list of each experiment
test_group_files = [f for f in os.listdir(directory) if f.startswith("_config_test_") and f.endswith(".yml")]
# Get every unique combination of tests (order doesn't matter)
test_groups = []
for i in range(1, len(test_group_files) + 1):
test_groups += list(itertools.combinations(test_group_files, i))
# Add an empty test group to build the base site
test_groups.append(())
return test_groups
And now all that’s left to do is build the sites. We can do this by simply calling bundle exec jekyll build as we would on the command line, through os.system. We need a bit of setup before that though. We need to clean up old sites, and
make sure we have a folder to put the new ones, which we’ll call _test_sites to match our upload script.
def gen_test_sites(directory: str, test_groups: list) -> list:
site_paths = []
old_dir = os.getcwd()
# Move to the same folder as the website
os.chdir(directory)
# Cleanup
# Create _test_sites folder if it doesn't exist. Otherwise delete and remake
if not os.path.exists("_test_sites"):
os.mkdir("_test_sites")
else:
os.system("rm -rf _test_sites")
os.mkdir("_test_sites")
# Loop through each group and build
for group in test_groups:
# Group name is a combination of all the test group names without the file extensions
# and without _config_test_, and with _site appended to the end
group_name = "_".join([g.split(".")[0].slpit("_")[-1] for g in group])
# Call the jekyll build function
os.system(f"bundle exec jekyll build --config _config.yml,{','.join(group)}")
# Rename build directory to group_name
os.rename("_site", group_name)
# Move to desired folder
os.system(f"mv {group_name} _test_sites")
# Add path to the site to the list
site_path.append(os.path.join(directory + "/_test_sites", group_name))
# Change back to the original directory
ow.chdir(old_dir)
return site_paths
And there you have it. A rather simple script, that should save you a lot of time manually typing out build commands. The only thing left to do is call all those functions.
if __name__ == "__main__":
working_dir = get_working_directory()
test_groups = get_test_groups(working_dir)
site_paths = gen_test_sites(working_dir, test_groups)
# Make site paths a valid JSON string
site_paths = str(site_paths).replace("'", '"')
print(site_paths)
The full script is available here.
If you have any questions or comments, please feel free to email me at [email protected].
]]>I have a lot of experience in video games. Both playing and creating. I’ve played hundreds of games, and developed several, both commercial and just as a hobby. Most of the hobby ones aren’t open source, and none of the commercial ones are, but a game I worked on for Global Game Jam 2019 is open source, and you can find it on GitHub here, if you’re interested. However, the repo for the game engine I used, Angel2D, is archived, and the exact branch seems to no longer exist, so it may be difficult to build the game, if you want to actually play it.
You can find more information about my commercial projects on my about page.
I especially like the more backend aspect of game development. I enjoy the less visual aspects of programming in general. In that theme, this guide is not about how to make a video game. This guide is about how to make a video game replicate.
So, what is replication? Simply, in the context of video games, it means cloning the game state of one player to that of another player. So, if player A and player B are playing a game together, and player A kills player B’s character, it is replication that allows player B to see in his game that he has been killed by player A. Replication is what makes a multiplayer game multiplayer.
As this guide is not about how to make a game, you will need to provide your own if you want to follow along with this tutorial. I would recommend a game written in Javascript or Typescript (admittedly, a relatively small subset of open source games), but this guide will be easily applicable in any programming language.
In this guide, we’re going to be using the game from a tutorial by jslegend, which can be found here, but you can feel free to use any game you like.
Since we already have our game, we’ll start by making the game server to connect to. This will be what is called an “authoritative server”. That means that the server is treated as the authority of truth in the game world. Whatever the server says happens is what happened, and any rumours to the contrary are ignored and discarded by the game.
There are a few protocols we can use to make this, but we’re going to use one called Websockets in our example. Websockets is effectively a message protocol. Similar to a phone call between you and your mom, the game client opens a “call” (called a “channel”) with the server, and they both remain on the call for the duration of the play session. They use the channel to send messages to each other. These messages will contain information about the game state. They can be things like the client informing the server that the user is trying to move up, or the server informing all the connected clients that another player has moved to the left.
Depending on the requirements to prevent cheating, and minimum acceptable latency, and things of that nature, the server verifies all actions the client claims to perform before announcing that action to everyone else who needs to know. Since this is just a demo, we don’t really care if anyone cheats, so our server will have very minimal checks on client actions, but we will discuss what could be added at the end.
You don’t have to write the server in the same language as the game, like we will be doing in this tutorial, however it is often easier to write the game and server in the same language, so you can share code between the two.
Ok, so let’s start writing our server. With Javascript, and many other languages, we don’t have to write the entire server on our own. We’ll use the npm package ws to help us.
npm install ws
And then we can write our first iteration of the server.
import WebSocket, { WebSocketServer } from 'ws';
const wss = new WebSocketServer({port: 8080});
wss.on('connection', (ws: WebSocket) => {
ws.on('message', (message: string) => {
console.log(`Received message => ${message}`);
});
ws.on('error', console.error);
ws.on('close', console.log);
ws.send('something');
});
This is pretty basic, but it’s a good start. You can connect on port 8080. When the client connects, the server starts checking for any messages from the client. If we get a message, we print it out.
You can connect to this server with your preferred Websocket client and interact with it to see it in action, but it will be more useful to update our game to interact with the server.
I added a new file to the game, that I called socket.js which is used to open the connection to the game server. It effectively mirrors what the server does: connects to the server, and passes
messages that it receives to another function to handle.
// Connect to the websocket server
const socket = new WebSocket('ws://localhost:3000')
socket.onopen = () => {
alert('Connected to the websocket server')
}
socket.onclose = (event) => {
if (event.wasClean) {
alert('Connection closed cleanly')
} else {
alert('Connection died')
}
}
socket.onerror = (error) => {
alert(`Error: ${error.message}`)
}
window.setupHandler = (func) => {
socket.onmessage = (event) => {
func(event)
}
}
window.socket = socket;
This is good, but doesn’t really do anything. Let’s send messages to the server. The simplest thing would probably be to tell the server where the player has moved to, so let’s start with that.
In the file scene/world.js, which is the file that manages player input and world events for the game I’m using, I added a function called updatePlayer which takes the coordinates of the player, and
uses window.socket.send to send the server those coordinates.
function updatePlayer({x, y}) {
window.socket.send([x, y])
}
and then we’ll just add a call to the updatePlayer function everywhere that handles player movement.
If we did everything correctly, you should see your server printing out the players coordinates as you walk around the map in game. Congratulations, you have half of a replicating game. Now we need the server to tell the client where everyone else is.
To do this, we will need to add a function the game to receive messages from the server. This is pretty simple:
function handleMessage(message) {
console.log(message);
}
window.setupHandler((event) => {
handleMessage(event.data)
})
As you have probably guessed, this will print out any message sent to the server. You’ll notice that we pass event.data instead of simply event. The event object returned from the websocket has more information
than we need for this, so we’re only interested in the data portion, which is the exact message that we get from the server.
Just printing out the message isn’t very useful. Let’s move the character based on the server messages. It would be better to use some sort of binary encoded messages, but for ease of reading, we will just use plaintext. We can indicate
that a message adjusts the players position by using a prefix, and a separator, followed by the information in the expected format. For the prefix we’ll use position. We can use a colon for the separator, as
we don’t expect that that will be in any of the data we want to send. The data part can simply be formatted as x,y.
Now let’s update our handleMessage function
function handleMessage(message) {
if(!message) return false;
if(message.split(':')[0] === 'position') {
const [x, y] = message.split(':')[1].split(',');
player.moveTo(parseInt(x, 10), parseInt(y, 10));
}
console.log('handled!', player.pos)
return true;
}
Now we need to change the server to send messages to the client. We’ll start simple. We’ll have the server expect only one client to connect, and track where the player is, and then inform the same client of where the player has moved to.
First we need to store the player’s location, so let’s add a variable for that.
const lastLocation = {x: 0, y: 0};
We’ll define this within the wss.on('connection', ...) block, so that the variable is unique per client.
Now let’s update our ws.on('message', ...) block.
ws.on('message', (message: string[]) => {
console.log(`Recieved message => ${message}`);
const messagearr = message.toString().split(',');
// messagearr now contains the x coord in the first element, and y coord in the second
const parsedMessage = [parseInt(messagearr[0], 10), parseInt(messagearr[1], 10)];
console.log(parsedMessage)
lastLocation.x = parsedMessage[0];
lastLocation.y = parsedMessage[1];
// Send location to client
ws.send(`position: ${parsedMessage[0]},${parsedMessage[1]}`);
});
The full code, as we’ve written it so far, is available here for the client, and here for the server. However, if you’ve never done this before, I would recommend that you type out the code yourself.
If you’ve done everything correctly, you should see your character gliding around the screen. Or, maybe the game you’re building on even still animates the character. If it doesn’t, that’s normal. The game we’re using only animates the character when it is moved through normal means, but with our new code it just plops the character at the new location. We’ll fix this a little later.
Since we’re making a replicating game, the server should support multiple clients. If you open another client and move around, you should notice something: it’s exactly the same as the first client you opened. You can’t see any other players. Let’s fix that next.
Now that we have a game sending information between the server and multiple clients, but with a separate game state for each client, it should be pretty simple to make the game multiplayer. We just need to create a game state shared between all clients, instead of being unique to each client. So, what do we need to do to do that?
We need some way to broadcast a message to all clients, to tell them the game state has updated. Let’s add a function to do that.
function updateAllClients(message: string) {
console.log(`Broadcasting message: ${message}`);
}
It’s a start, but not very useful. We need to loop through each client and send them each message, so we’ll need to store each client. We can use an array for this. We’ll make it a global variable so that it will
be the same for each connection:
const clients = [];
and then our broadcast function can be
function updateAllClients(message: string) {
for (const client of clients) {
client.send(message)
}
}
and then in our wss.on('connection', ...) we’ll save each connection to the client array
clients.push(ws)
What about when a client disconnects? We’ll need to remove it from the array. Finding the exact spot in the array and re-arranging it every time a client disconnects is not very simple though, computationally, and it’ll be hard to maintain the same order of clients on the server and on each client, so let’s store the clients like this instead
const clients = {};
And we’ll give each client a unique key, that we’ll call clientId. So, to add a new client to the object
const clientId = Math.random().toString(36).substring(2, 15) + Math.random().toString(36).substring(2, 15);
clients[clientId] = ws;
and our broadcast function body will be changed to
for (const clientId in clients) {
clients[clientId].send(message)
}
And when a client disconnects we simply
delete clients[clientId]
Ok, awesome! Now our server can keep track of multiple clients at once, and send messages to each client. We’re still missing something though. The clients don’t know how many other clients
there are, or where to put each of the other player characters. We’ll use our broadcast function for this. We need to broadcast whenever a client connects, disconnect, or when the player on the client
changes location. We’ll add two new messages for this: connected and disconnected. We also need some way to tell each client exactly which client has changed positions, so let’s modify our position
message slight too. Instead of just position:x,y, we’ll include the client ID as well: clientId:position:x,y, and, last but not least, we need a message to tell each client what their own ID is. We’ll call this id.
We will also need to tell the newly connected client about every other client, and then tell every other client about it. Each client also needs to know their own client ID, so they know which player is their own.
ws.send(`id:${clientId}`);
for (const otherClientId in clients) {
if (otherClientId in clients) {
ws.send(`connected:${otherClientId}`)
}
}
// Inform all clients of newly connected one
updateAllClients(`connected:${clientId}`);
Ok, so after all those changes, our wss.on('connection', ...) now looks like this
wss.on('connection', (ws: WebSocket) => {
// generate client ID
const clientId = Math.random().toString(36).substring(2, 15) + Math.random().toString(36).substring(2, 15);
const lastLocation = {x: 0, y: 0}
// add client to clients object
clients[clientId] = ws;
ws.on('message', (message: string[]) => {
const messagearr = message.toString().split(',');
const parsedMessage = [parseFloat(messagearr[0]), parseFloat(messagearr[1])];
lastLocation.x = parsedMessage[0];
lastLocation.y = parsedMessage[1];
updateAllClients(`${clientId}:position:${parsedMessage[0]},${parsedMessage[1]}`);
})
ws.on('error', (error: Error) => {
console.log(error);
delete clients[clientId];
// Notify all clients of disconnect
updateAllClients(`disconnected:${clientId}`);
})
ws.on('close', () => {
delete clients[clientId];
// Notify all clients of disconnect
updateAllClients(`disconnected:${clientId}`);
})
ws.send(`id:${clientId}`);
for (const otherClientId in clients) {
if (otherClientId in clients) {
ws.send(`connected:${otherClientId}`)
}
}
// Inform all clients of newly connected one
updateAllClients(`connected:${clientId}`);
})
You can see all the changes here.
Now if you open the client, if it still works with the updated server, you’ll notice it’s still the same as before.
The changes are pretty simple. We essentially just need to handle the new messages.
We’ll start with the new id message, since it’s the simplest. This message informs the client of their own ID.
So, we’ll start by making a global variable called clientId that will store our ID.
let clientId = '';
and then we’ll add some code to handle the message in our handleMessage function
if (message.split(':')[0] === 'id') {
clientId = message.split(':')[1];
return true;
}
Next we’ll change our position handling to handle the position of other players. We will handle our own movement slightly differently, but you can handle both together, if you prefer, or if that is easier in the
project that you’re working on.
We’ll need some way to store the locations of the other players. We’ll store it by their client IDs.
cons otherPlayers = {};
and then we can handle the message as follows
if (message.split(':')[0] !== clientId && message.split(':')[1] === 'position') {
const id = message.split(':')[0];
const [x, y] = message.split(':')[2].split(',');
// Move player
otherPlayers[id].moveTo(parseFloat(x), parseFloat(y));
return true;
}
Handling movement of our own player is almost the same
if (clientId && message.split(':')[0] === clientId && message.split(':')[1] === 'position') {
const [x, y] = message.split(':')[2].split(',');
player.moveTo(parseFloat(x), parseFloat(y));
return true;
}
The only messages left are connected and disconnected. When a player connects they’ll need a new player entity on each client,
in order to represent that player to every other player. And when they disconnect we need to destroy that entity.
// Handle connections
if (message.split(':')[0] === 'connected') {
const id = message.split(':')[1];
// Ignore our own connection message
if (id === clientId) return false;
// `add` is the function to add new entites to the world in kaboom.js
otherPlayers[id] = add([
sprite('player-down'),
pos(500, 700),
scale(4),
area(),
body(),
{
currentSprite: 'player-down',
speed: 300,
isInDialogue: false
}
]);
alert(`${id} has connected!`);
return true
}
// Handle disconnections
if (message.split(':')[0] === 'disconnected') {
const id = message.split(':')[1];
destroy(otherPlayers[id]);
delete otherPlayers[id];
alert(`${id} has discnnected!`);
return true;
}
Now if we open our client, we should be able to play with a single player as normal, and then, if everything is working, if you open another game client and connect to the same server, you should see a message on the original client saying that the new one has connected, and you should see a new player on your map. If you move one player, you should see it move, almost immediately, on both clients.
You can find the client diff here.
That’s the basics of it. You’ve now made a singleplayer game into a multiplayer one.
If you notice bugs, that’s expected. You probably don’t have working animations, and it’s unlikely that the players start in the correct location on other player’s screens. There’s also no verification on the server side. If the client says it can move outside of the map, then the server just takes its word for it.
Things like these are very important for multiplayer games, but outside the scope of this guide, as the implementation of those is far too specific to the exact game.
Also of note, in this guide we used an authoritative server (although, its authority is questionable), but you could make a peer to peer multiplayer game with many of the same principles you learned in this guide.
A peer to peer setup has many benefits, but also downsides. In a peer to peer setup, there is no central server, which save a lot on cost to the developer.
Each client simply connects to other clients. Essentially, each client acts as a server. In fact,
you could simply copy the server code to the client, and you will have effectively made a peer to peer multiplayer game. If you want more than two players, or any sort of player discovery system, that will
be more difficult to implement for a peer to peer game, but it is possible. Also, worth noting that it is much easier to cheat in a peer to peer game. Since one (or more) of the peers is the host (basically
treated the same as an authoritative server) the player using the host can change the game rules to do whatever they want. If allowing this is a dealbreaker for your game, peer to peer is not for you.
If you’re interested you can see my client and server with many of the bugs fixed here and here respectively.
If you have any questions or comments, please feel free to email me at [email protected].
]]>If you’re reading this, you probably know, but a chess engine is simply a computer program that can calculate chess moves. It can calculate very bad moves, like worstfish or the chess engine we will make in this guide, or it can be very good like Stockfish or AlphaZero.
There are many chess engines in the world, and many programmers who make chess engines, so there are a lot of resources on how to make them. One of the most useful is probably chessprogramming.org, which is essentially Wikipedia for chess programming. It has pages explaining pretty much everything you’ll need to make a chess engine, most importantly, it explains the various techniques and algorithms for move searching and evaluation.
I also made use of this site, as it provides a very good reference for UCI, which stands for “Universal Chess Interface”, which is a standardized protocol (one of a few) that chess engines use to talk to chess clients. Without UCI (and the other chess communication protocols) each chess engine would need to make its own client, and any other programs that want to use the chess engine would need to make a custom implementation for each engine they want to use, which is obviously unacceptable. With this, as long as the client and the engine support UCI, they’ll be compatible. The developer of the client doesn’t need to have ever even heard about the engine and vice-versa.
I found it to be annoying to write a UCI implementation each time I try my hand a new chess engine, and I was unable to find a pre-built implementation, so I built my own. You can write your own too. It is fairly simple to implement, but if you don’t want to, my implementation is available here.
For this guide, we’re going to write a chess engine in Python. There’s not really any reason for that, aside from that I want to use Python for this, and the UCI implementation I wrote is written in Python, and I don’t feel like writing another one.
Ok, so let’s start by making a program for two humans to play chess. We won’t make a GUI, and we’ll ignore UCI for now. Our chess player will just be a text interface that displays an ASCII representation of the chess board, takes text input as a move, draws the new board, and then waits for text input from the other player, until the game is over.
So, how do we do that?
The first thing the program needs to do is draw a chess board, so lets start with that. We’ll just use the same letters to represent the pieces as is used in algebraic chess notation: K for King, Q for
Queen, N for kNight, R for Rook, and B for Bishop. In algebraic notation, you don’t put any symbol for pawns, so we’ll use P for pawn. Upper case
letters will represent a white piece, and a lowercase letter will be a black piece. A . will be an empty space. So a chess board will look like this:
r n b q k b n r
p p p p p p p p
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
P P P P P P P P
R N B Q K B N R
We’ll assign each square a number which we’ll use to keep track of what piece is on which square. We can use an 8x8 matrix (2D list/array) to represent this, but it’s simpler to just use a single list and store 64 values. So let’s do that.
board = ['R', 'N', 'B', 'K', 'Q', 'B', 'N', 'R'] + ['P' for _ in range(8)] + ['.' for _ in range(8*4)] + ['p' for _ in range(8)] + ['r','n','b','k','q','b','n','r']
and then we can print it out with
count = 0
for sq in board[::-1]:
if count % 8 == 0:
print()
print(sq, end=" ")
count += 1
You’ve probably noticed that the list is reversed. That’s because we want the square a1 to be at index 0. We’ll come back to this soon.
Ok, so now let’s play a move. We can get input with something as simple as
move = input()
But what do we do with that? Let’s say that the first move is the very common e5. That’s great, but how do we move the pawn? We know it’s a pawn move because the piece that is moved isn’t specified, and we know where the pawn needs to go, but we aren’t told which pawn to move. We know it must be the pawn on e2 because that’s the only pawn that can move to e5, but how do we tell our program
that? We could write some code to figure that out. It would be pretty simple, actually, but it gets more complicated when there are more move options. So, instead of dealing with that, let’s just
make the user tell us. Instead of the normal algebraic notation used for chess, we’ll use a more specific notation. The user will tell us what square the piece they want to move is on and what square they
want it to go to.
So, instead of just typing e5 the user would need to type e2e5.
This is also much easier to parse. The first two characters will always be the coordinates of the square to move the piece from, and the second two characters will be the coordinates of the square to move the piece to.
from_square = move[0:2]
to_square = move[2:]
Cool, but we can’t index a list using just e2 and e5. As mentioned earlier, a1 is 0, so b1 will be 1, c1 is 2, and so on. a in ASCII is 97, so we can get the index with this:
from_square = (ord(move[0:1].lower()) - 97) + ((int(move[1:2]) - 1) * 8)
to_square = (ord(move[2:3].lower()) - 97) + ((int(move[3:4]) - 1) * 8)
And then to move the piece to that square, we can simply do
board[to_square] = board[from_square]
board[from_square] = '.'
This is just one option to use to store the chess board. It does have some downsides. It uses a list of 64 strings, which are generally harder to work with than numbers. If we want to know where every pawn is on the board,
we need to loop through the entire list and check if the value is a P or a p. There is a better way, that’s more common, called a bitboard.
Bitboards are essentially just lists, but smaller. As the name implies, a bitboard is a board made up of bits. An entire chess board shown as a bitboard can be represented as
1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1
This is just a binary number that I wrote out in a fancy way. It’s 18446462598732906495 in decimal.
A 1 means that there is something there, and a 0 means that there is not. This on its own isn’t very useful, except to know which squares have pieces and which don’t. So instead of just one bitboard,
we use at least 8. One for each of pawn, knight, bishop, rook, queen, and king, and then two more to denote the colour of that piece.
As you can see, these bitboards are effectively just lists of bits. However, even though you need more of them, you only need to store 8 numbers, instead of 64 strings, so much smaller. You can also get information through bitwise operations on the bitboards, which is very useful.
You should note that this works perfectly, because a chessboard has 64 squares, which is the same number of bits that an integer uses in Python, and in general on 64-bit systems.
We can make moves in the same way too.
import sys
# Construct bitboards
bishops = 2594073385365405732
rooks = 9295429630892703873
knights = 4755801206503243842
pawns = 71776119061282560
kings = 1152921504606846992
queens = 576460752303423496
bitboards = [bishops, rooks, knights, pawns, kings, queens]
move = input()
from_square = (ord(move[0:1].lower()) - 97) + ((int(move[1:2]) - 1) * 8)
to_square = (ord(move[2:3].lower()) - 97) + ((int(move[3:4]) - 1) * 8)
# Check what piece is being moved
index = 0
max_index = len(bitboards)
while index < max_index
board = bitboards[index]
if (1 << from_square) & board:
# This is the board we want to move on
# Change the bit of the from_square to 0
board &= sys.maxsize ^ (1 << from_square)
# Change the bit of the to_square to 1
board |= 1 << to_square
bitboards[index] = board
break
index += 1
You’ve (hopefully) noticed a lot of problems with this example, but this is the basic way that piece movement works on bitboards. First thing that you’ll notice is that there is no bitboard for white and black pieces, so we have no way of knowing what piece is which colour. This isn’t needed in our example, as we don’t bother to check if a move is legal. Of course, if you were to use this for your chess engine, you would need to implement move validation, otherwise you can move any piece to any square, regardless of the piece, multiple pieces can be on the same square (as long as they’re of different types), and you can capture your own pieces.
In this section I’ll explain the bitwise operations. They’re not relevant to the rest of this guide, so if you already know how they work, or you don’t care, feel free to skip to the next section.
I’ll explain from top to bottom.
if (1 << from_square) & board:
This checks if the piece exists on the board. In chess only one piece can be on a given square, so we can assume that the first instance of a piece we find on that square is what we’re looking for (even though that’s not necessarily true with the above implementation).
You can think of (1 << from_square) & board as the equivalent to board[from_square] != '.'. So, the condition is True iff there is a piece on the square. Why (1 << from_square) & board behaves that way is
more apparent if you look at the binary representation of each part.
>>> bin(1 << from_square)
'0b1000000000000'
>>> bin(pawns)
'0b11111111000000000000000000000000000000001111111100000000'
If you count the number of 0s in the first result, you’ll also see that the 1 at the end aligns with a 1 in bin(pawns), so the bitwise AND of both values will contain a 1 bit at some point. It doesn’t matter where. If there is at least a single non-zero bit, the condition will be True. Because there can only be one piece on the square, the same operation with every other bitboard will be 0, so the condition would be False.
Next is
board &= sys.maxsize ^ (1 << from_square)
# Which is equivalent to
board = board & (sys.maxsize & (1 << from_square))
This is more complicated, but still pretty simple. sys.maxsize is simply the largest possible 64 bit value. That means the bitboard it makes is simply all 1 bits. We want to set the bit to zero though, not one, so we need whatever bit represents the piece we’re moving to be zero, and every other bit to be one, so it doesn’t remove any other pieces. To do this we do a bitwise XOR with ^. Any bit in 1 << from_square that is one will be flipped to zero in the result of the XOR operation. We then do a bitwise AND with the bitboard. This will change the from_square bit to 0 and leave every other bit unchanged. This is the equivalent to the list operation we did before:
board[from_square] = '.'
And last but not least, we have the bitwise operation
board |= 1 << to_square
# Which is equivalent to
board = board | (1 << to_square)
This one is pretty simple. It sets the bit on to_square to 1, indicating that the piece is now there.
I mentioned before that the example is missing a lot, notably checking if the move is even legal. Thankfully, we don’t need to write that, as there are many people who already have. Despite this, it is still very useful to know how they work. For this guide we’ll be using the aptly named chess.
As it says on the PyPi page, you can use it like this:
import chess
# Create a new board
board = chess.Board()
# List all legal moves
print(board.legal_moves)
# Make a move using standard algebraic notation
board.push_san("e5")
# Make a move using the move notation we've been using in this guide (called UCI notation)
board.push("e2e5")
# Display the current board
print(board)
As you can see, this is much simpler than writing this all ourselves.
So, let’s finally make the two player chess program. With this library, it’s pretty simple.
import chess
board = chess.Board()
while not board.is_game_over():
print(board)
move = None
while not move:
print("Legal moves: ", board.legal_moves)
if board.turn:
move = input("White move: ")
else:
move = input("Black move: ")
try:
board.push_san(move)
except chess.IllegalMoveError:
move = None
print(chess.outcome())
That’s all you need for a full two player chess game. Really, you don’t even need that much. You could do it with less code if you don’t print out legal moves, or bother to check for errors when pushing the move to the board.
The only thing that might not be straight forward or immediately clear is board.turn. This, as the name suggests, denotes who’s turn it is. It is True if it is white’s turn, and False if it
is black’s turn.
Also potentially of note: with board.push_san you can provide the move in either of the two move formats we’ve discussed.
We’re not here to just make a two player chess game though. We’re here to make a chess engine. So, let’s replace one of the two players with a bot.
import chess
import random
def gen_best_move(board: chess.Board, time_limit: int, depth: int) -> chess.Move:
# We'll just pick a random move for now
return random.choice(list(board.legal_moves))
board = chess.Board()
while not board.is_game_over():
print(board)
move = None
while not move:
print("Legal moves: ", board.legal_moves)
if board.turn:
move = input("White move: ")
else:
move = gen_best_move(board, 5000, 2)
try:
board.push_san(move)
except chess.IllegalMoveError:
move = None
print(chess.outcome())
Now you can play against a bot. A very bad bot, most likely, but perhaps by some act of God it’ll choose the best move every time. Probably not. This bot probably sucks.
You’ll notice it takes three arguments: board, time_limit, and depth. board and time_limit are pretty self-explanatory, but I’m going to explain anyway. board contains the game board.
time_limit is the maximum amount of time that the move selection should take, specified in milliseconds. This is a requirement for supporting UCI. depth may not be as
obvious. It is also required for proper UCI support, but more importantly, similar to time_limit it tells the engine how long to search for. The difference is instead of specifying a time limit, the
depth tells the engine how many moves to search. In this example, depth is 2, so a proper move search algorithm (i.e. not random) would evaluate two moves ‘deep’. This means it will evaluate the position after simulating 2 moves, rather than just one. This will be more clear as we look at move search algorithms.
There are many ways to search for the best move, some better than others. Some faster. Some find a better move, but take much longer to get there. For this guide we’re going to focus on a single search algorithm called “negamax”, but I would recommend exploring chessprogramming.org to find better algorithms and get some ideas to make your own.
Negamax is a very simple algorithm, but it’s good enough for our chess engine.
from time import time
import random
import math
import chess
def evaluate(board: chess.Board) -> int:
return random.randint(-1000, 1000)
def negamax(board: chess.Board, end_time: int, depth: int) -> int:
# If depth is 0 or the time limit has expired, evaluate the position and return that value
if depth == 0 or time() >= end_time:
return evaluate(board)
best_score = -math.inf
# loop through every possible move
for move in board.legal_moves:
# simulate the move
board.push(move)
# evaluate one depth down
score = -negamax(board, end_time, depth - 1)
board.pop() # Un-simulate move
best_score = max(score, best_score)
return best_score
The first thing you’ll notice is that evaluate just returns a random number. Later this function will return a number based on the actual position, but we’ll work on that later. Let’s focus on the move search for
now.
As you can see, Negamax is a recursive algorithm, meaning it calls itself. This is how it evaluates at depths more than just 1. It’s pretty simple: it loops through every possible move, and then calls itself, passing the new position, and a depth of one less than it itself was set to search, and negates the value (hence the name). After that it compares the newly computed position’s score to the current best score, and if it’s higher it saves it. After looping through all the moves it returns the best score. Notably this function does not return the best move. The reason why will become clear later, but for now just be aware that this function only returns a score. You can write your version of the function to return the move as well, but this is slightly more complicated.
The reason we need to negate the value of negamax each cycle is because in chess you take turns playing, therefore the next iteration is actually the score for the opposite colour. Because of how the evaluation
function will work, the score for one side will be exactly the score for the other times -1.
The next step is to perform the evaluation of the position. There are many methods to do this, and many aspects we can use to score the position, and then still, even more methods we can use to represent that score.
For this guide, we’re going to use simple piece value scoring, and we will score all the pieces in numbers of pawns that they are worth. It is common to use ‘centipawns’ (1/100th of a pawn) as the base unit too, but the exact unit we use matters very little, as long as we’re consistent with it.
So, we’ll assign each piece a value. We’ll use the standard piece values, but you can adjust these if you find certain values make the engine better. A queen is worth 9 points, rooks are 5, bishops and knights are 3, and pawns, of course, are 1 point. What about the king? You can’t capture the king, so normally it doesn’t need a score. In fact, in our engine, it won’t need a score either, however, we will give it one anyway, as it is useful with other evaluation methods. It is very arbitrary what value the king gets, but for this example we’ll say it’s worth 10000 pawns. By convention, white pieces have positive values, and black pieces have negative values.
Here is our evaluation function:
import chess
def evaluate(board: chess.Board) -> int:
score = 0
# Loop through each square, and add to the score for each piece
for square in board.piece_map():
value = 0
piece = board.piece_at(square)
match piece.piece_type:
case chess.PAWN:
value = 1
case chess.KNIGHT:
value = 3
case chess.BISHOP:
value = 3
case chess.ROOK:
value = 5
case chess.QUEEN:
value = 9
case chess.KING:
value = 10000
if not piece.color:
value *= -1
score += value
return score
That’s all the basics of a chess engine, but we are missing one thing: we don’t actually call negamax; so let’s fix that. We’ll change our gen_best_move function.
from time import time
import math
import chess
def gen_best_move(board: chess.Board, time_limit: int, depth: int) -> chess.Move:
end_time = time() + (time_limit / 1000) # / 1000 to convert the ms to seconds
best_move = None
best_score = -math.inf
for move in legal_moves:
# Simulate move
board.push(move)
# Evaluate
score = -negamax(board, end_time, depth - 1)
# Un-simulate move
board.pop()
if score > best_score:
best_score = score
best_move = move
return best_move
This is basically the same as negamax, except that it returns a move instead of just an evaluation. In fact, if you change negamax to return both a move and an evaluation, you don’t even need the
extra gen_best_move function. I prefer to have the additional function, just because it tends to make the code easier to read when using more complex move search algorithms. Most importantly, you
can use the gen_best_move function to perform some setup functions for negamax (calculating end_time, in this example). As gen_best_move is not called recursively, it’s generally easier to do it here.
Putting that all together, we now have a very basic chess engine. It should make some moves that are not horrible more than it does. You can run it, and play against it, and probably win, if you’re any good at chess.
However, we’re still missing one thing. Up until now we’ve been inputting moves by just typing the move in to stdin. We still don’t have support for UCI. Technically UCI is still just passing moves in to
stdin, but in a way that allows for more things, such as reseting the board, and starting the game at arbitrary positions. It also lets us configure the depth and time limit of the move search at
runtime, instead of having to edit the numbers in the code. And, perhaps most importantly, it also allows us to use other chess clients.
I could teach you how to write your own UCI implementation, like I did for bitboards, but writing the UCI support is my least favourite part of writing a chess engine, and bitboards are one of the things I find most interesting about them, so I will not be showing you how to write a UCI implementation. It is pretty simple. You can probably figure it out yourself from the UCI specification explanation I linked to in the resources section, or probably from some guide on chessprogramming.org or one of many other websites, I’m sure.
Instead, I’ll be showing you how to implement the UCI implementation I’ve already written, and is available for free under the LGPG-3.0 on my Github. You can install it like you would any other Python module: using pip.
pip install git+https://github.com/Jacob-MacMillan-Software/python_chess_interface.git
If you use poetry you can also do
poetry add git+https://github.com/Jacob-MacMillan-Software/python_chess_interface.git
We’ll have to change our gen_best_move, and negamax functions slightly to support this.
from time import time
import math
from queue import Queue
import chess
def gen_best_move(position: str, time_limit: int, depth: int, send_queue: Queue, recv_queue: Queue):
end_time = time() + (time_limit / 1000) # / 1000 to convert the ms to seconds
board = chess.Board(position)
best_move = None
best_score = -math.inf
for move in legal_moves:
# Simulate move
board.push(move)
# Evaluate
result = negamax(board, end_time, depth - 1, recv_queue)
score = -result[0]
# Un-simulate move
board.pop()
if score > best_score:
best_score = score
best_move = move
# Check for stop command
if result[1]:
break
if not recv_queue.empty():
if recv_queue.get() == "stop":
break
print("bestmove", best_move)
def negamax(board: chess.Board, end_time: int, depth: int, recv_queue: Queue) -> (int, bool):
# If depth is 0 or the time limit has expired, evaluate the position and return that value
if depth == 0 or time() >= end_time:
return (evaluate(board), False)
best_score = -math.inf
forced_stop = False
# loop through every possible move
for move in board.legal_moves:
# simulate the move
board.push(move)
# evaluate one depth down
result = negamax(board, end_time, depth - 1, recv_queue)
score = -result[0]
board.pop() # Un-simulate move
best_score = max(score, best_score)
if result[1]:
forced_stop = True
break
if not recv_queue.empty():
if recv_queue.get() == "stop":
break
return (best_score, forced_stop)
There are a few differences, but the functions remain mostly unchanged. The gen_best_move function now takes two additional parameters: two Queues. The send_queue is unused in this example, but it can
be used to send messages to the UCI class. The recv_queue is used to receive messages from the UCI. In this example it only accepts one command, stop, but there are other commands,
such as setting options to change how the engine works, or possibly even change the type of chess that the engine plays, or whatever else you want to allow to be changed at runtime. The gen_best_move function also
no longer returns any value. Instead, it prints out ‘bestmove’ followed by the move in UCI format. This is what clients that interact with UCI expect. This tells them what move the engine wants to play.
You’ll also notice that the board parameter has been changed from a chess.Board to a str and renamed position. This is simply because UCI accepts the position as a string called a FEN followed by a list
of moves. The UCI class I wrote converts this to just a FEN (with the moves made), and passes that FEN string to the gen_best_move function.
negamax has also been changed. It now returns a tuple, and it takes an additional parameter: recv_queue. The queue, of course, is used in the same way as in gen_best_move as it is the same queue. The client
interacting via the UCI can send a stop command at any time, after which it expects the engine to stop searching for moves and return the best move it has found so far. The tuple is used to propagate this stop
command up through the recursive calls.
The last thing we need to do is actually start the UCI:
interface = UCI(gen_best_move)
while True:
try:
command = interface.read()
except KeyboardInterrupt:
exit(0)
That code will replace our previous move input code.
So, now we have the following:
from time import time
import math
from queue import Queue
import chess
def gen_best_move(position: str, time_limit: int, depth: int, send_queue: Queue, recv_queue: Queue):
end_time = time() + (time_limit / 1000) # / 1000 to convert the ms to seconds
board = chess.Board(position)
best_move = None
best_score = -math.inf
for move in legal_moves:
# Simulate move
board.push(move)
# Evaluate
result = negamax(board, end_time, depth - 1, recv_queue)
score = -result[0]
# Un-simulate move
board.pop()
if score > best_score:
best_score = score
best_move = move
# Check for stop command
if result[1]:
break
if not recv_queue.empty():
if recv_queue.get() == "stop":
break
print("bestmove", best_move)
def evaluate(board: chess.Board) -> int:
score = 0
# Loop through each square, and add to the score for each piece
for square in board.piece_map():
value = 0
piece = board.piece_at(square)
match piece.piece_type:
case chess.PAWN:
value = 1
case chess.KNIGHT:
value = 3
case chess.BISHOP:
value = 3
case chess.ROOK:
value = 5
case chess.QUEEN:
value = 9
case chess.KING:
value = 10000
if not piece.color:
value *= -1
score += value
return score
def negamax(board: chess.Board, end_time: int, depth: int, recv_queue: Queue) -> (int, bool):
# If depth is 0 or the time limit has expired, evaluate the position and return that value
if depth == 0 or time() >= end_time:
return (evaluate(board), False)
best_score = -math.inf
forced_stop = False
# loop through every possible move
for move in board.legal_moves:
# simulate the move
board.push(move)
# evaluate one depth down
result = negamax(board, end_time, depth - 1, recv_queue)
score = -result[0]
board.pop() # Un-simulate move
best_score = max(score, best_score)
if result[1]:
forced_stop = True
break
if not recv_queue.empty():
if recv_queue.get() == "stop":
break
return (best_score, forced_stop)
if __name__ == "__main__":
print("Chess Engine Example by Jacob MacMillan Software Inc.")
interface = UCI(gen_best_move)
while True:
try:
command = interface.read()
except KeyboardInterrupt:
exit(0)
And that’s it. You now have a chess engine. If you want, you can open up your preferred chess GUI (assuming it has UCI support) and use it to play against you’re newly created engine. Enjoy!
If you have any questions or comments, please feel free to email me at [email protected].
]]>Similar to how stable coins such as USDT are backed by USD, a Fungible Token (FT) can be backed by a share of a company. This would effectively allow zero non-network-fee transaction of stocks. With non-fungible tokens (NFT), this can be extended to derivatives as well.
Securities trading is highly subject to regulation in most, if not all, jurisdictions. I’m not familiar with the majority of these regulations, nor am I familiar with any relevant rules and regulations outside of Canada and The United States. Because of this, some or all of this proposal may be difficult or impossible to do legally. Because the tokens are technically derivatives and anyone (to the best of my knowledge) can sell options and futures, which are also derivatives, my assumption is that at least the core premise (selling tokens backed by a security) is legal in both Canada and The United States, but obviously I don’t know for sure.
Stocks of public companies can’t be traded without using a trusted third party (such as Wealth Simple Trade, RBC Direct Investing, etc.) and those third parties often take high fees on transactions. It also makes it difficult to transfer/sell a share to a specific person, although this isn’t something that’s particularly useful to do, apart from gifts, which is also uncommon.
This system would be effectively the same as current stable coins: a trust company/bank receives the backing asset (in this case shares of a company, or similar security), and, in exchange, will issue a FT to the party that deposited the backing asset. The trust will then hold the backing asset until the owner of the FT backed by the backing asset (whether it be the original depositor or anyone else) requests to exchange the FT for the real company stock. When this happens, the trust will burn the FT (or verify the user has burned it), and transfer the backing asset to the requesting party.
This has two obvious problems:
As both FTs and NFTs are required to achieve the full scope of this proposal, and multiple types of each are required, using a (very custom) ERC-1155 contract (or an equivalent on whichever network this system is being implemented on) will probably be ideal.
I believe the best implementation of this system for stocks is to simply receive money from users that is equal to the current price of the stock the user wants (optionally, plus a few), and then the user is issued an FT that they can return at any time to receive the money equal to the value of the stock at time of return (optionally, less a fee).
What is done with the users money between purchase and return is irrelevant to this section. Some options are described in the Potential Monetization section.
Bonds are effectively stocks with a par value that increases at a constant rate, or drops to zero instantaneously. For this reason, bonds can be implemented in exactly the same way as stocks.
NOTE: I will not explain what options are or how they work in this section, nor will I explain the different types of options, as that is not what this proposal is about and that information can be easily found on the internet.
One possible way to implement options is through special NFTs, as stated above. The contract must support advanced functionality for these NFTs, which will differ depending on the type of option.
Similarly to how Enjin and Xenum allow NFTs to be backed by ENJ and ETH respectively, these option NFTs should have a backing of tokens representing the assets the options are based on, as well as the money (likely in a stable coin) with which to buy the asset. This is to enforce a payout to both parties in the event the option is exercised. This has the added benefit of ensuring both parties agree to the terms, because they must both send an asset in order to create the NFT.
It should be noted that naked calls are impossible with this system, as it requires the assets to be held in escrow before the NFT can be created.
For the rest of this section “option” and “NFT” will be used interchangeably.
All NFTs will require a method to exercise them. What this does exactly will vary based on the type of NFT. For most options this function will only be callable by the buyer, however, in some cases, such as when the option specifies that it must be exercised (ex. oil futures), any party, including parties not involved in the transaction, may execute the method.
Any option that needs to expire (I’m unaware of any types of options that don’t fit this category) will also need to have a method to expire them. This method will burn the token and release the assets that the token was “holding” to the parties who have the rights to such tokens, based on the terms of the option. In most, if not all, cases it makes sense to let any party call the expire function, including parties not involved with the transaction. Of course, the expire function should check if the option has actually expired before doing anything.
In some cases the expire and exercise methods will do the exact same thing. This is fine. They should both still exist in order to maintain consistency within the smart contract.
The most obvious way to monetize this system is to sell the tokens for slightly more than the backing asset is worth, and sell them for slightly less. Technically this would be the same monetization method as current investment platforms (taking a fee), but in this case the fee is only paid when the token is exchanged for the underlying asset. In the case where this system is popular, most people will not do this. It will be much easier to sell the token to another user rather than go through the process of exchanging it for the underlying asset, which would likely require some KYC process.
I am not familiar with the way trust companies work, so I’m not sure this is possible/legal, but another potential to monetize is to sell the underlying asset in the open market, and simply buy it back when a user wants to exchange a token representing the asset for the asset itself. This would effectively be a short position, so this should not be done with every asset.
If the implementation of this system is such that the users never receive actual stock, just the money in exchange for the stock, one possible monetization method would be to never actually buy the stock, and simply pay out when requested. However, this would be a short position on every supported security, so it is also possible to lose money this way.
This concept in itself is very similar to base cryptocurrency: people can trade an asset without involvement of any third party. However, to simplify the process, and increase liquidity, a number of cryptocurrency exchanges have been created. These exchanges necessarily take a fee. They typically take the fee on withdraw, and not when purchasing the tokens, but they still take a fee. Decentralized Exchanges (DEX) do exist, but they are uncommon.
This proposed system, if successful, will also inevitably lead to the creation of exchanges, which would take a fee, thus not actually solving the problem. This would instead just create the same problem, but now it’s more difficult to get to the point where you have that problem. So people will just use regular investment platforms so they don’t have to take extra steps to then have the exact same problem.
Because of this, it will be necessary to create and popularize a DEX. It must also be very simple to use, otherwise people will just not use this solution for the reasons described in the above paragraph.
]]>This optionally fully decentralized messaging system allows messaging from a cryptocurrency address to any other cryptocurrency address. By asymmetrically encrypting messages, the system ensures that only the intended recipient is able to read the message, in most cases1.
By hosting both the messages, and the client on IPFS, and using an ENS name as the domain name to access said client, most liability from the party running the system is removed, as there is no way to know any information about users2, including their names, email addresses, and even their IP address. There is also no way to control who can and cannot access the client3.
The optional centralization comes in with how the user’s “inboxes” are stored, and how the messages are stored. For the purposes of this document, “inbox” and “inboxes” refers to a database containing either:
It is probably prohibitively expensive to store inboxes in a decentralized manner, therefore to make this system practical, it may be necessary to store inboxes in a centralized database such as Redis or PostgreSQL. However, the inbox itself contains only enough information to find encrypted messages sent to users, so there is no significant downside for both the users and the party hosting the system if the inboxes are stored in a centralized way. In fact, it is likely more of a benefit to the users, as it reduces costs to send messages by 100%4.
Hosting the client can also be done in either a centralized or decentralized way. The messaging client can be a static website hosted on IPFS, which is assigned an ENS domain name. This would be fully decentralized. The alternative is to host the client on a regular server like any other website and have a regular DNS name point to the server. Optionally, an ENS name can be set to redirect to this DNS name.
Weirdly, it would likely be less expensive to host the client in a fully decentralized way, as it means no recurring costs for a server, and ENS names are typically less expensive than DNS names. Apart from the cost of the ENS name, the only fees from the decentralized hosting would be a small fee to update the ENS name when the client is updated. This fee would likely be less than a dollar, and updates can be restricted in order to reduce costs.
However, there seems to be a few problems with hosting a website on IPFS, namely that the entire website must be a single file. The obvious work-around being to link to external JavaScript and CSS files, but that would then no longer be fully decentralized, and those files would need to be hosted somewhere, likely at a (small) cost. In that case, since you need to store files in a centralized manner anyway, it seems like a better solution may be to host the entire website in an AWS S3 bucket. The cost to hosting a website in an S3 bucket is very minimal, if not completely free. It still requires a DNS name to point to the bucket, but it removes any possible server costs (apart from the aforementioned cost of the S3 bucket).
The client must at least allow a user to verify they own the address they wish to send the message from (‘from address’), allow the user to choose an address to send the message to (‘to address’), allow the user to write a plaintext message (‘the message’), and allow the user to send ‘the message’ from the specified ‘from address’ and to the specified ‘to address’. The client must also allow users to view the encrypted versions of any messages they have been sent. These features are the absolute minimum requirements for a client; obviously additional features can be added, so long as the base features remain.
Messages are sent to and received by cryptocurrency wallet addresses.
Sending the messages is a multi-step process:
The message is considered sent after these steps have been processed.
Receiving messages is also a multi-step process.
If the message is stored on IPFS:
If the message is stored in a database:
As well as all of the above, which is required for any functional implementation of this system, a more complex—and probably more user friendly—implementation would include the following:
Messages may contain an optional metadata string. It is recommended that the metadata not be encrypted, or if it is encrypted, that it be encrypted separately, if it will be used by the client for anything other than display purposes. This way it is impossible for the message sender to insert malicious code into the message that resembles a metadata string, and therefore may be accidentally executed as though it were a metadata string.
If the implementation does support metadata strings all messages MUST include a metadata string, however, it may be empty (ie. “{}”).
Metadata strings MUST be JSON encoded and be the absolute first thing in the message file, including whitespace.
The information required to be included in the metadata string (if any) is determined by the implementation, but ideally it would contain at least the time the message was sent, and who it was sent from. If the from address is included in the metadata it SHOULD also contain a short message signed by the from address, in order to verify the message was actually sent from that address.
1. Messages may be read by an unintended party in cases where the recipient's private key has been compromised, or other user error.
2. Depending on the exact details of the system's implementation.
3. Depending on the exact details of the implementation of the client.
4. Assuming the implementation doesn't require the user to pay to send messages.