Chris' Blog.

My occasional thoughts on iOS development, developers careers, trying to make an income from the App Store, and updates on life in general.

Rust

Hi all, I’d like to share some adventures I’ve recently had with interacting with Interactive Brokers’ TWS API directly using Rust, without an SDK.

You’ll need TWS (or IB Gateway) running on your machine. Note that their API host is TWS - you don’t talk directly to their servers. You’ll also need a real account, but not a paid data subscription.

The code can be found here in completed form: main.rs gist

And without further ado…

Go here and download the latest TWS: https://www.interactivebrokers.com.au/en/trading/tws.php
You must login with a real live or paper account. If you use the ‘try the demo’ mode the API does not respond.
You will not require a paid data subscription for the symbols we’re watching here.
Go File > Global Configuration > Api > Settings > check ‘Enable ActiveX and Socket Clients’
!!!! ENSURE “Read-Only API” IS CHECKED !!!!
Also take note of ‘Socket port’ on that screen (7496 is what I see).
Select Apply and Ok
Run nc -z localhost 7496 to test that TWS API is listening.

Ok lets start talking to this with Rust.

First lets make a new rust project:

Install rust if you need to!
cargo new my_tws_api_talker
cd my_tws_api_talker
cargo run

You should see a Hello message.

Lets make it connect. Open up src/main.rs:

use std::net::TcpStream;

fn main() {
    let mut stream = TcpStream::connect("127.0.0.1:7497").expect("connect");
    stream.set_nodelay(true).expect("nodelay"); // Because we're wannabe HFT traders.
    println!("Able to connect")
}

Perform cargo run, it should say it was able to connect. Now lets send the greeting. Most messages to TWS need a big-endian 4-byte length prefix. So lets implement that as a prerequisite:

fn add_length_prefix(bytes: &[u8]) -> Vec<u8> {
    let len = bytes.len() as u32;
    let mut data: Vec<u8> = Vec::new();
    data.push(((len >> 24) & 0xff) as u8);
    data.push(((len >> 16) & 0xff) as u8);
    data.push(((len >> 8) & 0xff) as u8);
    data.push((len & 0xff) as u8);
    data.extend(bytes);
    data
}

And one small helper. Not sure why this isn’t part of the stdlib, unless I’m missing something:

fn concat(a: &[u8], b: &[u8]) -> Vec<u8> {
    let mut both = a.to_owned();
    both.extend(b);
    both
}

So now we can use these helpers to send the greeting:

use std::io::{Read, Write}; // Add this up top.

const MIN_SERVER_VER_BOND_ISSUERID: u32 = 176; // From server_versions.py.
const DESIRED_VERSION: u32 = MIN_SERVER_VER_BOND_ISSUERID;

fn send_greeting(stream: &mut TcpStream) {
    let prefix = "API\0";
    let version = format!("v{}..{}", DESIRED_VERSION, DESIRED_VERSION);
    let version_msg = add_length_prefix(version.as_bytes());
    let both = concat(prefix.as_bytes(), &version_msg);
    stream.write_all(&both).expect("Greeting");
}

Now to actually call that greeting, add this to the bottom of the main function:

send_greeting(&mut stream);
println!("Greeting sent");

Then cargo run and it should say greeting sent. Great!

Next we need to read a message from TWS. These messages all look like this:

4 bytes big endian message length
Message

A message is a list of fields. Each field is a string, with a zero after it. Numbers are strings, and bools are “0” (false) or “1” (true). Eg a simple 3-field message would be “Field A\0Field B\0Field C\0”.

First we need to read a length:

fn read_length(reader: &mut TcpStream) -> u32 {
    let mut len_buf: [u8; 4] = [0; 4];
    reader.read_exact(&mut len_buf).expect("Read length");
    let length: u32 = ((len_buf[0] as u32) << 24)
        | ((len_buf[1] as u32) << 16)
        | ((len_buf[2] as u32) << 8)
        | len_buf[3] as u32;
    length
}

Next we need something to split a read message into its fields:

fn split_message(message: &[u8]) -> Vec<String> {
    // Split into an array of buffers:
    let mut components = Vec::<Vec<u8>>::new();
    let mut current_component = Vec::<u8>::new();
    for byte in message {
        if *byte == 0 {
            if !current_component.is_empty() {
                components.push(current_component.clone());
                current_component.clear();
            }
        } else {
            current_component.push(*byte);
        }
    }
    if !current_component.is_empty() {
        components.push(current_component);
    }

    // Convert the buffers into strings:
    components
        .into_iter()
        .map(|v| String::from_utf8_lossy(&v).to_string())
        .collect()
}

Next lets bring all that together. Read the length, then read the message, then split it:

fn read_message(reader: &mut TcpStream) -> Vec<String> {
    let length = read_length(reader);
    let mut buffer: Vec<u8> = vec![0; length as usize];
    reader.read_exact(&mut buffer).expect("Read message");
    split_message(&buffer)
}

To use this, add the following to the bottom of main.rs:

let greeting_response = read_message(&mut stream);
println!("Greeting response: {:?}", greeting_response);

cargo run and you should see TWS respond with something like ["176", "20230826 22:07:41 AEST"] These fields are the api version then datetime. The version should match DESIRED_VERSION from our code. You should verify that it does, in production code.

Right after the greeting, we have to send the ‘Start API’ message. If you download IB’s python api and look in message.py, you can see all the request type codes. Lets write an enum with only a few request types we’ll use:

enum OutgoingRequestType {
    ReqMktData = 1,
    StartApi   = 71,
}

Now we’ll need a function to compose a message:

// Join and delimit the fields, prefixing the length.
fn make_message(fields: &[String]) -> Vec<u8> {
    let mut delimited_fields = Vec::<u8>::new();
    for field in fields {
        delimited_fields.extend(field.as_bytes());
        delimited_fields.push(0); // Even goes after the last field.
    }
    add_length_prefix(&delimited_fields)
}

Then we need our function to compose and send the ‘start api’ message:

fn send_start_api(stream: &mut TcpStream, client_id: u32) {
    let fields: Vec<String> = vec![
        (OutgoingRequestType::StartApi as u32).to_string(),
        "2".to_string(), // Version of the Start API message.
        client_id.to_string(),
        "".to_string(), // Optional capabilities.
    ];
    let message = make_message(&fields);
    stream.write_all(&message).expect("Start api");
}

And let’s call it. Add this to the end of main():

send_start_api(&mut stream, 123456); // Client ID 123456.

If you cargo run now it should work.

At this stage, the API will want to send us a bunch of messages. So we’d better loop for those. Firstly lets identify some of the incoming message types:

enum IncomingRequestType {
    TickPrice = 1,
    ErrMsg    = 4,
} // From message.py.

Using the above message types, lets add a message handler:

fn message_received(fields: &[String]) {
    if fields.is_empty() { return }
    let Ok(t) = fields[0].parse::<u32>() else { return };
    if t == IncomingRequestType::ErrMsg as u32 {
        let _request_id = fields.get(2);
        let _code = fields.get(3);
        let Some(text) = fields.get(4) else { return };
        println!("Message: {}", text);
    } else if t == IncomingRequestType::TickPrice as u32 {
        let Some(tick_type) = fields.get(3) else { return };
        if tick_type != "4" { return } // We only want 4 = 'Last' from ticktype.py.
        let Some(price) = fields.get(4) else { return };
        println!("Price: {}", price);
    } else {
        println!("Received: {:?}", fields); 
    }
}

Now, lets listen for messages. Add this at the bottom of main(), but keep in mind we’ll replace it with a threaded version soon:

loop {
    let message = read_message(&mut stream);
    message_received(&message);
}

Run it with cargo run and you should see a bunch of messages printed. You’ll need to do ctrl-c to quit.

But you won’t see any prices come in, because we haven’t requested any. This led me to some thinking around how to receive and send messages. I personally like the idea of making a reading thread, a writing thread, and a control (main) thread, to spread the load. So firstly, in main(), remove the loop above, and add this in its place, to ‘split the socket’ into a reader and writer:

// !! Ensure you remove the previous loop above !!
let mut writer = stream.try_clone().expect("Clone");
let mut reader = stream;

Next we want a mechanism for the main thread to enqueue messages for the writer thread to send:

// use std::sync::mpsc::channel;
let (writer_tx, writer_rx) = channel::<Vec<String>>();

Next lets spawn our reader thread for incoming messages:

// use std::thread;
let reader_handle = thread::Builder::new().name("Reader".into()).spawn(move || {
    loop {
        let message = read_message(&mut reader);
        message_received(&message);
    }
}).expect("Spawn");

Next lets spawn the writer thread that listens to the channel, then writes the messages to the socket:

let writer_handle = thread::Builder::new().name("Writer".into()).spawn(move || {
    loop {
        let fields = writer_rx.recv().expect("Writer queue receive");
        let message = make_message(&fields);
        println!("Sending {:?} aka {:?} aka {:?}", 
            fields, String::from_utf8_lossy(&message), message);
        writer.write_all(&message).expect("Write all");
    }
}).expect("Spawn");

Now we need to get the main thread to wait for the new threads:

writer_handle.join().unwrap();
reader_handle.join().unwrap();

You can run it now, and it should receive messages and run until you perform ctrl-c.

It still isn’t showing price data however, as we need to ask TWS for that. We’ll do that now by sending a message from the main thread to the writer thread. Add the following code just above the previously-added calls to ‘join()’:

// Post-handshake delay or TWS bugs out.
thread::sleep(std::time::Duration::from_millis(500));

let request_market_data: Vec<String> = vec![
    (OutgoingRequestType::ReqMktData as u32).to_string(),
    "11".to_string(),     // Version
    "999".to_string(),    // Request id aka Ticker id
    "".to_string(),       // Contract id
    "BTC".to_string(),    // Symbol
    "CRYPTO".to_string(), // Security type
    "".to_string(),       // LastTradeDateOrContractMonth
    "".to_string(),       // strike
    "".to_string(),       // right
    "".to_string(),       // multiplier
    "PAXOS".to_string(),  // exchange
    "".to_string(),       // primaryExchange
    "USD".to_string(),    // currency
    "".to_string(),       // localSymbol
    "".to_string(),       // tradingClass
    "0".to_string(),      // Delta neutral contract? 0=false, 1=true
    "".to_string(),       // genericTickList
    "0".to_string(),      // is a snapshot?
    "0".to_string(),      // regulatorySnapshot - costs 1c ea.
    "".to_string(),       // mktDataOptions - unsupported.
];
writer_tx.send(request_market_data).expect("Writer queue send");

// join calls go here...

Then run it, and you’ll get (among other messages) the real-time price of Bitcoin in USD! You can take it from here and build the trading bot of your dreams.

This code carries no warranty, use it at your own risk, and is MIT licensed!

For reference for the other messages, you’ll have to read through IB’s Python SDK. Thanks for reading, God bless, and have a nice week :)

Rust

Hi! Here is a PNG writer I wrote as a utility as part of a hobby project I’ve been working on, where I preferred to minimise dependencies and keep things simple. It hope it serves as a good way of understanding the PNG format! Note that this doesn’t compress, and it only supports RGBA; but it can be useful for tiny pixel art images.

Let’s start with the prerequisites:

RFC 1951 Deflate stream

The first thing needed is a function to convert an arbitrary buffer into a ‘deflate stream’. A deflate stream is made up of blocks of up to 64kb. The simplest possible compliant deflate block looks like this:

1 byte: 1 = this is the final block; 0 = this is not the final block.
2 bytes: data length, lowest-significant-byte first.
2 bytes: length 1’s complement (all bits flipped).
N bytes: data

In Rust, this looks like:

fn to_deflate_stream(input: &[u8]) -> Vec<u8> {
    if input.is_empty() {
        return vec![1, 0, 0, 0xff, 0xff]; // 1 block with no content.
    }
    let mut output = Vec::<u8>::new();
    let chunks = input.chunks(0xffff);
    let final_index = chunks.len() - 1;
    for (index, chunk) in chunks.enumerate() {
        let is_final = index == final_index;
        output.push(if is_final { 1 } else { 0 });
        let len = chunk.len();
        let len_lsb = (len & 0xff) as u8;
        let len_msb = (len >> 8) as u8;
        output.push(len_lsb); // Max len.
        output.push(len_msb);
        output.push(!len_lsb); // 1's complement of len.
        output.push(!len_msb);
        output.extend_from_slice(chunk);
    }
    output
}

RFC 1950 Zlib stream

The next building block necessary is to convert a data buffer into a Zlib stream. This consists of adding a header and footer to a Deflate stream. The simplest possible header for the uncompressed case is 0x7801, meaning:

A standard window size of 7 (meaningless when not compressed).
A method of 8 (meaning deflate).
A 1 in the second byte for a checksum, no dictionary bit set, and 0 for the compression level bits.

The footer is an ADLER32 checksum.

In Rust, this looks like:

fn to_zlib_stream(input: &[u8]) -> Vec<u8> {
    // Header.
    let mut output = Vec::<u8>::new();
    output.push(0x78); // CMF byte. Bits 0-3=method, 4-7=info/window size. Method=8, Window size=7.
    output.push(1); // FLG byte. Bits 0-4=fcheck, 5=fdict which we dont want so 0, 6-7=flevel where 0 means fastest.

    // Body.
    let deflated = to_deflate_stream(input);
    output.extend(deflated);

    // Checksum.
    // See: https://en.wikipedia.org/wiki/Adler-32#Example_implementation
    let mut a: u32 = 1;
    let mut b: u32 = 0;
    for data in input {
        a = (a + (*data as u32)) % 65521;
        b = (b + a) % 65521;
    }
    output.push((b >> 8) as u8);
    output.push((b & 0xff) as u8);
    output.push((a >> 8) as u8);
    output.push((a & 0xff) as u8);

    output
}

CRC32

Another important piece of the puzzle is the ability to calculate CRCs for a buffer.

I can’t claim to understand how these work. If interested, you could read more here: http://libpng.org/pub/png/spec/1.0/PNG-CRCAppendix.html

In Rust, it looks like:

fn crc(data: &[u8]) -> u32 {
    // Make the CRC table first.
    // An optimised implementation would only do this once.
    let mut crc_table: [u32; 256] = [0; 256];
    for n in 0..256 {
        let mut c: u32 = n as u32;
        for _k in 0..8 {
            if c & 1 == 1 {
                c = 0xedb88320u32 ^ (c >> 1);
            } else {
                c = c >> 1;
            }
        }
        crc_table[n] = c;
    }

    // Calculate the CRC.
    let mut crc: u32 = 0xffffffff;
    for b in data {
        let index = ((crc ^ (*b as u32)) & 0xff) as usize;
        crc = crc_table[index] ^ (crc >> 8);
    }
    !crc
}

PNG

And now we have the prerequisites for generating the PNG. PNG files consist of a header then at least 3 blocks. The following describes the RGBA case, which has no palette:

The header looks like: 0x89 “PNG” CR LF EOF LF.

Each block looks like:

Data length: 4 bytes, most significant byte first.
Type: 4 ascii chars.
Data
4 bytes CRC of the type and data.

The compulsory 3 blocks are: IHDR, IDAT, and IEND.

IHDR

The header block’s data consists of:

Width and height: 4 bytes each, MSB first.
Bits per pixel: 1 byte.
Type: 6 means RGBA.
Compression: always 0 which means Zlib.
Filter method: always 0.
Interlacing flag.

IDAT

The data block consists of RGBA samples, left to right, top to bottom, in a Zlib stream. At the start of each row, a 0 is prefixed which means ‘no filter applies to this row’.

IEND

The end block has empty data. I’d be willing to bet you could omit this block and no parser would care.

The PNG generator code looks like the following in Rust:

fn png(width: u32, height: u32, rgba: &[u32]) -> Vec<u8> {
    let mut output = Vec::<u8>::new();

    // Header.
    output.push(0x89);
    output.push(b'P');
    output.push(b'N');
    output.push(b'G');
    output.push(0x0d); // Cr
    output.push(0x0a); // Lf
    output.push(0x1a); // Eof
    output.push(0x0a); // Lf

    // Build IHDR.
    let mut ihdr_type_and_data = Vec::<u8>::new();
    ihdr_type_and_data.push(b'I');
    ihdr_type_and_data.push(b'H');
    ihdr_type_and_data.push(b'D');
    ihdr_type_and_data.push(b'R');
    append_msb(&mut ihdr_type_and_data, width);
    append_msb(&mut ihdr_type_and_data, height);
    ihdr_type_and_data.push(8); // 8bpp.
    ihdr_type_and_data.push(6); // RGBA.
    ihdr_type_and_data.push(0); // Compression method: zlib.
    ihdr_type_and_data.push(0); // Filter method.
    ihdr_type_and_data.push(0); // No interlace.
    let ihdr_len = ihdr_type_and_data.len() - 4; // Minus the type.
    let ihdr_crc = crc(&ihdr_type_and_data);
    // Append IHDR to output.
    append_msb(&mut output, ihdr_len as u32);
    output.extend_from_slice(&ihdr_type_and_data);
    append_msb(&mut output, ihdr_crc);

    // Build image data.
    let mut idat_data = Vec::<u8>::new();
    let mut rgba_iter = rgba.iter();
    for _y in 0..height {
        idat_data.push(0); // Each line is prepended a filter type byte (0).
        for _x in 0..width {
            let rgba = rgba_iter.next().unwrap_or(&0);
            idat_data.push((rgba >> 24) as u8);
            idat_data.push((rgba >> 16) as u8); // 'as u8' truncates upper bits for us.
            idat_data.push((rgba >> 8) as u8);
            idat_data.push((*rgba) as u8);
        }
    }
    let compressed_idat_data = to_zlib_stream(&idat_data);

    // Build IDAT.
    let mut idat_type_and_data = Vec::<u8>::new();
    idat_type_and_data.push(b'I');
    idat_type_and_data.push(b'D');
    idat_type_and_data.push(b'A');
    idat_type_and_data.push(b'T');
    idat_type_and_data.extend_from_slice(&compressed_idat_data);
    let idat_len = idat_type_and_data.len() - 4; // Minus the type.
    let idat_crc = crc(&idat_type_and_data);
    // Append IDAT to output.
    append_msb(&mut output, idat_len as u32);
    output.extend_from_slice(&idat_type_and_data);
    append_msb(&mut output, idat_crc);

    // IEND (no data).
    append_msb(&mut output, 0); // Length.
    output.push(b'I'); // Type.
    output.push(b'E');
    output.push(b'N');
    output.push(b'D');
    let iend_crc = crc(b"IEND");
    append_msb(&mut output, iend_crc);

    output
}

fn append_msb(vec: &mut Vec<u8>, value: u32) {
    vec.push((value >> 24) as u8);
    vec.push((value >> 16) as u8);
    vec.push((value >> 8) as u8);
    vec.push((*value) as u8);
}

And that’s it! That’s the basics of writing PNGs in around 150LOC.

The full code can be seen here: github.com/chrishulbert/rust_png_write_sans_dependencies.

Feel free to use it in your projects by copying the file in; I haven’t published this as crate, I think it’s too simple to justify that.

Thanks for reading, I hope this is helpful, God bless :)

Photo by Tengyart on Unsplash

Stack

In practice, I usually find that using a UITableView makes most sense for displaying scrollable content in an app, however every now and again it is handy to use a UIScrollView + UIStackView. The problem is: it occurs infrequently enough that I cannot recall how to set up the constraints! So here’s my notes for my future reference, and hopefully yours too :)

Here’s a screenshot of how to set it up in Xcode: (click to zoom)

Theory

The frame layout guide and content layout guides are only to be used for child views of the scroll, and the child views are to be (usually) only constrained to these guides.
The frame layout guide is for setting child view’s sizes to match the scroll’s visual size.
The content layout guide is what defines the scroll view’s content size.

Vertical scroll setup

Make a UIScrollView and constrain it normally to define its size (eg not using its layout guides).
Put a vertical UIStackView inside the UIScrollView.
Constrain the stack’s leading/trailing/top/bottom to the scroll’s content layout guide.
Make the stack’s width equal to the scroll’s frame layout guide.
Put views in the stack. These views don’t need any constraints put on them at all.

Horizontal scroll setup

Make a UIScrollView and constrain it normally to define its size (eg not using its layout guides).
Put a horizontal UIStackView inside the UIScrollView.
Constrain the stack’s leading/trailing/top/bottom to the scroll’s content layout guide.
Make the stack’s height equal to the scroll’s frame layout guide.
Put views in the stack. These views don’t need any constraints put on them at all.

Paged scroll setup

If you want each child of the stack to be a whole ‘page’:

Horizontal scrolling: Constrain the stack’s childrens’ width to equal the scroll’s frame layout guide.
Vertical scrolling: Constrain the stack’s children’ height to equal the scroll’s frame layout guide.
Set ‘isPagingEnabled’ to true on the scroll.

Thanks for reading, I hope this is helpful, God bless :)

Photo by Fernando Andrade on Unsplash

You can see older posts in the right panel, under 'archive'.