libsixel is a SIXEL encoder/decoder implementation derived from kmiya's sixel. From to 1.8.7-r1, signed integer overflow in sixel_encode_highcolor's allocation size calculation can lead to a heap buffer overflow. The public sixel_encode entry point validates only that width and height are greater than zero, with no upper bound. width and height are multiplied as plain int when computing the allocation size for paletted_pixels and normalized_pixels. Any caller that asks libsixel to encode a pixel buffer with width times height greater than INT_MAX (about 2.15 billion) will hit a wrapped allocation size.

under the right wrap, the malloc succeeds with a buffer much smaller than the encoder expects, and the encoder writes past the end of the heap allocation. This vulnerability is fixed in 1.8.7-r2.