Communication speed improvements
In the [Ultimaker support group], people reported "hickups" in movements and "blobs" in their printed models. They where reported to disappear when another G-code generator is used ([netfabb] instead of [skeinforge]).
A debate is going on if this is purely related to the Gcode, or a side effect of skeinforge gcode interacting with the firmware, motion, communication or process/profile settings.
In an attempt to find the root cause, and distinguish fact from fiction, some measurements are proposed.
It is observed that buffer underruns are frequent with some profiles and settings. These cause problems, especially on the contours. Buffer underruns occur if there are too many small segments at high speeds. There are a few factors that influence this:
- Movement Speed
- Segment length (accuracy)
- Baudrate (communication time)
- Caluculation time in the firmware
- Response time of the application (latency)
- Buffer size.
- Use an application to send the GCODE to the Ultimaker with a low latency (that is, NOT replicatorG)
- Optimize the profile to go slower on the contour (high dimentional accuracy), or make larger segments (lower accuracy)
- Use a gcode generator that has an optimized profile that generates contours with optimal speed/accuracy (this is where netfabb comes in)
- Disable the "send temperature option" in replicatorG, This will definitely cause underruns at high speeds!
- Build from SD card, that has a higher bandwidth (added bonus: your build won't stop if your screensaver kicks in) This is the solutions recommended by various people, and has known to work. Unfortunately the Ultimaker ships without SD card support. This can be solved!
- Further improvements in the firmware that include: larger buffers, delayed ACKs or a Binary Protocol
Communication latency from the application to the hardware
This is one of the potential sources. This measurement tries to quantify actual real-world communication times.
The communication is measured by using a "dummy" echo program in the Arduino. This measures the interval between two commands, and stores it in a histogram.
- Download this special firmware in your Ultimaker (using arduino0022 software)
- Open a model, generate gcode and "build"
- No actual model is build (no movements), but you will see that all data is sent to the Arduino.
- If you check the "show communication" flag in the Real-time control window, you can see the communication interval [in msec] after each "ok" message in the log window.
- After the build is done, the histogram and statistics are printed in the log, so you can "cut and paste" the results from the replicatorG window in a text editor and plot it using libreOffice, Excel or gnuplot.
- Reprogram your Ultimaker with the orginal firmware, and you are good to go...
We would like to see your results on other platforms and with other files!
The dummy echo firmware:' File:Latency-measurement1.zip
The majority of the new commands are send 10msec after the previous command, (this may be related to the rxtx timeout and/or task scheduler). There are other groups at 20 and 30msec (10 msec intervals). (This may be related to the 10msec windows task scheduling interval). There is only one sample larger than the 40 msec(at dt=3059 msec). This may be due to the start or end of the measurement. It needs to be checked when it occurs.
The consequence of this latency, and the 4 element buffer in the firmware, is that for an uninterrupted build (no buffer under-runs, no hickups) the average movement time per segment over a 4 segment period needs to higher than the worst case latency (4 x 35 msec = 140 msec).
This value is 7 mm at v=50 mm/sec, and 14 mm at v=100 mm/sec.
Any shorter and under-runs can occur. This is where the g-code generator comes in: depending on the exact segmentation and ordering of commands, this may pose problems.
Note: this is not including any overhead (communication/calculation) and retransmissions that may be present when using the actual Ultimaker firmware.
Note: The nett communication time for the 44 characters/line at 57600 bps is approx 7.6msec
- Decrease latency (other means of communication)
- make sure the minimal segment length rule is respected during gcode generation
- increase buffer length, in order to average out over more segments, so it is easier to stay within the communication limits
Alternative control program
As you can see, latency is less, and more predictable. Latency is consistently around 15 msec, a factor 2 better than ReplicatorG!
A similar test is done with Ubuntu linux and ReplicatorG-025
This shows, linux latency is 25msec average. Not too good. A few under runs where observed.
The same measurements are repeated with 115200 bps, instead of 57600 bps. I'm not printing the complete graphs, but with PrintRun the shape was similar, the center just shifted a little to the left.
PrintRun: 57600 bps: average 12.5 msec/line 115200 bps: average 8.2 msec/line
ReplicatorG: 57600 bps: average 14.6 msec/line 115200 bps: average 10.2 msec/line
A special C# program is written to allow gcode files to be sent. C# has a few advantages over Java (RecplicatorG) and Python (PrintRun): it has native serial port support (
SerialPort object) with event based communication (Java needs external libraries like rxtx), and it is a fast and efficient Object Oriented compiled language (Python can be slower, and problematic to run).
C# uses the .NET runtime, that is available for windows (native) and Linux/Mac (Mono). The binaries are portable between the platforms.
Latency is dominant, and at 10 msec (Note: Linux systems with a 1000HZ kernel scheduler frequency might display lower latencies!) What if we would assume the arduino can receive at maximum speed (so the sender does not have to wait for the "ACK") ? This is much faster! But not practical, only a arduino that has nothing else to do can receive data at the maximum speed (the test program can do just that). We observed no data corruption at maximum speed.
The program and source code can be downloaded SendG.
As latency is a problem, and every line needs to be ACKed, this will always give a maximum transfer rate of 1/latency lines/sec. This is typically 50 to 100 lines/sec, more or less independent of the line length and baudrate. Significant improvement can only be made, when not every line needs to be ACKed, but more than one line can be ACKed per read. To test this, we use our SendG C# program that allows a variable number of lines to be send without an ACK.
This shows more lines can be sent, when we allow 3 or 4 lines to be buffered without an ACK. This approaches the theoretical bandwidth of the serial port. (at 300 lines per second, of 30 chars each, this is 9000 bytes/sec, where the maximum at 115kpbs is approx 10000 lines/sec). Note: the arduino serial buffer is 128 chars, so 5 lines of 30 characters exceeds the capacity of the buffer, it only works because the special timing firmware reads continuous
Hickups in the firmware
Is something unexpected happening? Some tests to be defined. I think I see that catching up from buffer under-runs / communication errors is slow... see if that can be visualized.
- IRQ issues?
- Motion/Communication interaction
- floating point overhead?
- digital I/O overhead
- Analogue I/O overhead
A special ("cleaned") firmware is made in order to investigate these issues. It contains all functions for normal operation, but all "dead" code is removed from the source to make development easier. All "old" hardware configurations are removed.
It reports the number of buffer under runs after the "ok" message. Enable the "show communication" flag in ReplicatorG realtime control window.
There is a port of the Sprinter firmware for the Ultimaker this handles (almost) everything in the foreground job (not IRQ's used) and buffers movements in the serial buffer.
There are some pro's and con's in this approach. Because everything is in the foreground, hickups are reported when the LCD is enabled.
We came up with a firmware performance test to judge the smoothness and (speed) accuracy of the movements. It involves moving with large (100mm) and small (0.25mm) segments, and measuring the time delay between these two movements.
Sprinter firmware came up with a ratio of 1.43 Using optimized command sending (sendg.exe). This is not actually not better than the original firmware.
There are many different firmware implementations and there is a lot of development. In order to improve speed and accuracy, we study the use of other firmware on the Ultimaker:
- Sprinter (see: Sprinter firmware for the Ultimaker): Less pausing
- Marlin (see Marlin firmware for the Ultimaker): True acceleration and look-ahead!
Optimized communication protocol
Given the limitations of the current ASCII based protocol and the required higher speeds, drastic measures are required to overcome this problem.
This page Ultimate Binary Protocol describes a proof of concept to define a binary encoding that is
- "GCODE compatible"
It is not a "quick-fix" and requires significant work to implement in firmware and PC driver software.
- Are extruder or process issues affecting the print?
- Some tests to be defined.
- Unwanted moves: co-linear segments!