Human-robot interface

We can interact with our robot via any laptop or desktop computer (although the latter isn't very practical), using either a Wireless or Ethernet (our choice) connection.

There are two main programs for interacting with Kókánybot: kokanyctl and kokanyrecognize. The former is used for controlling the robot's actuators and performing QR code recognition, while the latter uses our custom image-recognition model to recognize hazmat signs. The details of the programs and the connection are elaborated in the Communication section.

kokanyctl takes keyboard input and sends it to a daemon running on Kókánybot which processes the raw input into usable events.

Simplifying our software with device trees

Last year we used a library called libi2c to control some of our stepper motors using an external MCP23017 GPIO expander. Since then we have switched to using servos for our robotic arm. We also wanted to make our software more flexible, and the best way to do this was ripping out the useless code and using the right tools for the job.

While programs are allowed to just use libi2c, it adds (often redundant) extra code to a project, not to mention a lot of extra complexity since low-level driver code is not trivial. The Kernel actually has drivers for a lot of common ICs such as GPIO expanders—like the MCP23017 we used—or PWM controllers like the PCA9685.

One improvement we've made was deleting the I2C parts from our code, and instead using the kernel driver instead. This is great because we can just interact with the userspace PWM API via something like libhwpwm (more on that later).

To achieve this, we wrote a device tree overlay. Device tree overlays are kind of like patch files for device trees. This file allows us to tell Linux what chips are available on certain I2C addresses.

Switching from TCP to UDP

Robotics taught us just how fragile computers are when subjected to harsher environments. Short circuits may occur, components might get knocked against a robot's frame, boards can overheat, all of which can result in a robot rebooting or shutting down entirely.

In our previous competitions, we have always had faults like these occur. If things were going too well, then our UTP cable slipped out of our control station's Ethernet port. In these cases, we almost always had to reboot our robot and do the software side of the setup again and then reconnect, wasting precious time.

Mitigating these issues while sticking with TCP sockets would mean having to handle potential connection issues every time we send/receive data. Instead of doing that, we've switched to UDP, which is a different communications protocol. As of now, none of our programs contain any any TCP related code making all networking stateless (the multimedia bits might be stateful but FFmpeg deals with all of that).

The main advantage of UDP compared to TCP is its statelessness: networked programs do not need to connect(), listen() nor accept(), which also means no need to track clients; you create a socket, and send/receive data over it using sendto() and recvfrom(). This eliminates the need for handling reconnections. This is great because our custom network code sends only a stream of octets (detailed in the next section), which always correspond to a keypress or key release.

The main downsides of UDP are that it does not guarantee data packets arriving in the correct order nor does it guarantee that they arrive at all. This is not really an issue for us, since the entire network is point-to-point with no more than two hosts, and we have yet to experience any problems.

Network arcitecture

Kókánybot has to transport two kinds of data: keyboard input and video data. There are two ways to go about transporting them: we could have chosen to multiplex the input and the video and data streams (similar to this SO answer implemented using the API). We chose the path of least resistance though, and went with separate connections for each video feed and input. Currently there are two camera feeds and a connection for transporting keycodes . This allows us to avoid a lot of headaches and keep the networking as stateless as possible.

We don't use any complicated protocols for data transmission, because we'd liked to keep the processing needs as low as possible.

Efficiently encoding keyboard input for low overhead

Keys on a keyboard usually correspond to scancodes in software. These values do not necessarily correspond to characters, since they're the raw values of keys, and do not take the user's layout into account. SDL2 allows us to easily convert scancodes to keycodes which do correspond to actual characters a user may type in. The macro used to this conversion is SDL_SCANCODE_TO_KEYCODE().

Sending keycodes over the wire is part of the solution, but we also need to send the state of the key (whether it's pressed or not), ideally in a single byte.

A key can have two states meaning, two possible values, so it can be represented in log₂2=1 bits. This leaves 7 bits to represent keycodes, but there are more than 2⁷ possible values. However, we only handle WASD and numeric keys. Anything else is ignored on the server side.

We use some bit manipulation to encode and decode the actual data: net_encode_scancode() in kokanyctl encodes the state into the most significant bit of the octet by shifting a boolean (false and true correspond to 0 and 1 in C) 7 bits to the left (pressed << 7) and then we bitwise OR it with the keycode.

On the receiving end, kokanybot processes the incoming data in input_process_key_event . This calls an inline function, is_key_pressed which uses a boolean trick (in C, everything aside from 0 is true, so two boolean NOTs result in 0 or 1 if the value is anything but zero). 0x80 corresponds to 128 in hexadecimal, which corresponds to the most significant bit (10000000₍₂₎), where we stored the state of the key. bitwise ANDing the keycode with 128 produces either 0 or 128, and this is what we convert to either 0 or 1 with the previous boolean trick. Afterwards we discard the most significant bit, because the eight bit represents the key state and isn't part of the keycode itself. We then look up the correct function.

Removing gas sensing code

The rulebook's latest draft does not mention CO₂ sensing, therefore we've removed all code related to it. If a later draft brings it back, we can just reuse a previous commit, as we've often done during development.

Training an object detection model

Last time, one of the most challenging aspects of the competition was the object detection feature robots needed. Back in Bordeaux, we failed to detect anything and got 0 points for object detection. We definitely needed to improve on that.

Firstly, we found a large enough dataset on the internet (1k+ photos), which was a godsend because it spared us from having to manually take pictures and tag them.

Secondly, we switched from the outdated YoloV5 to the up-to-date YoloV8 which is supposedly better in every single way. We chose the small version, because the larger the model, the slower inference is, and while performance doesn’t really matter when training, one can simply leave their computer running while they are not home, it is still vital during inference, because our laptops are not on par with our workstations at home.

To use our model, we wrote a script named kokanyrecognize at the last minute. It wasn’t very performant nor really clean, so we spent a significant amount of time working on it. So far it was rewritten to use the new model, however it still has a long way to go.

Here is the result so far:

Communication

We interact with our robot using a few custom programs, named KókányControl (kokanyctl) and KókányRecognize (kokanyrecognize).

KókányControl has a graphical interface for displaying the video and the sensor data it receives from Kókánybot. It takes keyboard input, and sends commands to the Raspberry Pi. It also recognizes QR codes that appear on Kókánybot’s cameras.

KókányRecognize was written to reduce the complexity of KókányControl, since image recognition functionality is only needed in a few runs, and we can just launch KókányRecognize whenever we need it. This also enabled us to build KókányControl in pure C, since we would have needed to use C++ to build the image recognition bits (which uses OpenCV).

Video and audio streaming

During tests, operators are only allowed to see the arenas from their robot’s point of view. This meant we needed a way to find a way to display the video data from the robot’s cameras.

We've learnt from our mistakes last year, and have opted for using multiple cameras so that we have better peripheral vision while controlling Kókánybot.

Multimedia related tasks are surprisingly computation heavy when one is working with embedded systems. The CPU in the Raspberry Pi 4B+ is fairly capable, however we also had to consider power draw and thermal related problems. We considered several video formats: H.265, AV1 and H.264, but in the end we settled on using raw frames from our cameras to minimize latency as much as possible, since at the 2023 RoboCup, our camera's high latency caused a lot of trouble.

Linux assigns /dev/videoN to every camera. The kernel does not guarantee the order of the camera devices, however we must be able to tell them apart in a consistent way. Linux provides a way to do this using udev rules. We gathered the attributes of the cameras using udevadm—a standard udev utility—and wrote rules to assign the /dev/front-camera and /dev/rear-camera names to our cameras.

On the client side, we implemented the decoding of the video data using FFmpeg’s libavformat and libavcodec libraries. Rendering the video frames was tricky to figure out because pretty much all video encoders store pixels in YCbCr colour space, which SDL isn’t the best for.

The APIs of the libav* libraries are huge. Thankfully we only really needed the high level decoding API. The Learn FFmpeg libav the Hard Way tutorial combined with the examples in the project's documentation also made things much easier.

Human-robot interface

Still thinkking